Cannot use node.spec.podCIDR

[dnugmanov]

Issue: Kilo currently relies on node.spec.podCIDR for subnet announcement. However, when Cilium is in cluster-mode, it doesn't respect node.spec.podCIDR and doesn't populates subnet from CiliumNode to Node. This behavior leads to inconsistencies in subnet allocation.

Suggestion: I propose introducing the annotation kilo.squat.ai/force-subnet as a means to enforce subnet allocation. This annotation would allow users to specify the desired subnet directly, ensuring consistency between Kilo and your environment.

Expected Benefits:

• Flexibility: Provides users with the flexibility to enforce specific subnets where necessary, accommodating diverse network configurations.

[dnugmanov]

@squat Hi, please take a look, i am ready to bring small PR with following implementation if you don't mind the idea with new annotation.