spike: Workers primary multus and secondary pod

[qinqon]

What this PR does / why we need it: Different spike to play to use multus at primary interface at the VMI but keep the pod + masquerade at the second.

Special notes for your reviewer: FIXME:

• kubejoin error: error execution phase preflight: couldn't validate the identity of the API Server: could not find a JWS signature in the cluster-info ConfigMap for token ID "4pm9wv"
  • FIX: Manual works re-creating the token kubeadm token create --print-join-command
• calico error: calico/node is not ready: BIRD is not ready: BGP not established with 10.244.196.159
  • FIXES: Use same networking for control plane and ./kubevirtci kubectl-tenant set env daemonset/calico-node -n kube-system IP_AUTODETECTION_METHOD=cidr=192.168.4.0/24
• calico error: startup/startup.go 417: Hit error connecting to datastore - retry error=Get "https://10.95.0.1:443/api/v1/nodes/foo": dial tcp 10.95.0.1:443: connect: connection refused
  • control-plane VM can reach 10.95.0.1 but worker cannot.
  • woker cannot access the kubernetes api service default kubernetes ClusterIP 10.95.0.1 <none> 443/TCP 46m
  • The svc is pointing to the masquerade IP and that is duplicated between the VMs, apiserver has to bind and advertise the primary interface IP address.

NONE

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: qinqon Once this PR has been reviewed and has the lgtm label, please assign justinsb for approval by writing /assign @justinsb in a comment. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files: - **[OWNERS](https://github.com/kubernetes-sigs/cluster-api-provider-kubevirt/blob/main/OWNERS)** Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment

[k8s-ci-robot]

@qinqon: PR needs rebase.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

• Mark this PR as fresh with /remove-lifecycle rotten
• Close this PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[davidvossel]

/close

[k8s-ci-robot]

@davidvossel: Closed this PR.

In response to [this](https://github.com/kubernetes-sigs/cluster-api-provider-kubevirt/pull/199#issuecomment-1478065182): >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.