Proposed Changes

Adds Cilium CNI initialization option
Disabled by default unless cilium_iface is defined, overrides flannel and calico
Default settings assume nodes on same subnet and leverage native routing with auto direct routes, kube-proxy replacement, BPF masquerade, hybrid DSR, and maglev LB algorithm
Routed mode available for environments requiring it
Provides BGP control plane option with LB IPAM, replacing MetalLB when enabled
BGP control plane advertises LB services and pod CIDRs by default, allowing one to enable native routing mode in environments with cross-subnet cluster node traffic
Hubble observability platform enabled by default
Adds molecule scenario
Adds new scenario to CI test workflow
Tested with MetalLB in native L2 mode
Tested with Cilium in BGP control plane mode
Tested molecule scenario
Maintains integrity of interface contract and is non-breaking

Why Cilium?

Performance: Cilium is built from the ground up on eBPF, ensuring leading edge performance and scalability.
Maturity and Stability: Cilium is a CNCF "Graduated" maturity level CNI implementation and is widely adopted for production workloads by many of the largest cloud providers in the industry.
Advanced Policies: Cilium provides advanced layer 4, layer 3, and layer 7 policy capabilities.
Visibility and Monitoring: Cilium features Hubble observability platform, built on eBPF, providing live visibility into your cluster network activity.
Customization: Cilium is highly customizable for advanced use-cases post-initialization.

[x] Tested locally
[x] Ran site.yml playbook
[x] Ran reset.yml playbook
[x] Did not add any unnecessary changes
[x] Ran pre-commit install at least once before committing (actually did this time)
[x] 🚀