spike: Set worker networkings secondary

[qinqon]

What this PR does / why we need it: Hammer the infra to have one controller one worker tenant cluster using secondary interfaces + bridge cni + nmstate

TODO:

• [x] Basic scenario of one worker harcoding one ip for worker and one for node.
• [x] Basic secenario for two workers with harcoded IPs
• [x] Test live migration during a TCP connection
• [x] Generate IPs for workers and infra nodes at capk controller
• [x] Set ip from the tenant range on the nodes with NNCP at capk controller
• [x] Use netmask from adquiered IP (right now is harcoded to 24)
• [x] Use different network per tenant
• [x] Do infra node network teardown (delete vlan and bridge).
• [ ] Set iptables masquerade at infra node primary interface from the capk controller.
• [ ] Check services with custom endpoints.
• [ ] Scale up infra cluster (new nncp need to be created)
• [ ] Use secondary networks at control-plane too (kubeinit is called at cloud-init)
• [ ] Check integration at hypershift.
• [ ] Check if we can use proxy ARP so we have the same default gw on tenant nodes
  • https://chandanduttachowdhury.wordpress.com/2017/06/29/using-a-router-for-switching/
  • https://www.sobyte.net/post/2022-06/proxy-arp-calico/

FIXME:

• [ ] Don't modify guest VM networking for live migration (default gw need to be changed to node's IP)
  • Use virt-launcher sidecar pod to do the routing with one IP only accessible to the VM and repated at all the enodes

Release notes:

NONE

[coveralls]

Pull Request Test Coverage Report for Build 3204954945

• 14 of 16 (87.5%) changed or added relevant lines in 1 file are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage increased (+0.3%) to 51.743%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
pkg/kubevirt/utils.go	14	16	87.5%
<!--	Total:	14	16	87.5%	-->

Totals
Change from base Build 3154490540:	0.3%
Covered Lines:	965
Relevant Lines:	1865

---

💛 - Coveralls

[qinqon]

There is a chicken egg issue, we need to create one NNCP per tenant cluster + node since it will contain the specific ip for that tenant cluster and for that we need to know where the VMI machine is going to run but the VMI depends on the bridge to be created or the virt-launcher pod will be on "Scheduling" forever.

We have to configure all the infra workers nodes where VMIs will run with the bridge and an IPs from the ipamer, that should be done at KubevirtCluster reconcile.

[qinqon]

/ok-to-test

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: qinqon Once this PR has been reviewed and has the lgtm label, please assign agradouski for approval by writing /assign @agradouski in a comment. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files: - **[OWNERS](https://github.com/kubernetes-sigs/cluster-api-provider-kubevirt/blob/main/OWNERS)** Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment

[k8s-ci-robot]

@qinqon: PR needs rebase.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

• Mark this PR as fresh with /remove-lifecycle stale
• Close this PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

• Mark this PR as fresh with /remove-lifecycle rotten
• Close this PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten