Support Ignition Bootstrap for flatcar

[aamoyel]

At our company, we've recently tested flatcar images build with image-builder project but we've noticed that the cluster-api-provider-kubevirt doesn't support ignition.

What happened in our case:

The default implementation (https://github.com/flatcar/coreos-cloudinit) doesn't support specific keys used by the kubevirt-provider. Here you can find cloud-init logs on our flatcar :

Jun 02 16:08:27 localhost coreos-cloudinit[1229]: 2023/06/02 16:08:27 line 277: warning: unrecognized key "runcmd"
Jun 02 16:08:27 localhost coreos-cloudinit[1229]: 2023/06/02 16:08:27 line 286: warning: unrecognized key "sudo"
Jun 02 16:08:27 localhost coreos-cloudinit[1229]: 2023/06/02 16:08:27 line 287: warning: incorrect type for "groups" (want []string)

We tried to use ignition format in the Cluster API ControlPlane Provider with this configuration:

apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: KubeadmControlPlane
metadata:
  name: xxxxxx
  namespace: xxx
spec:
  kubeadmConfigSpec:
    format: ignition

The secret bootstraped by the Cluster API was correctly generated but when the kubevirt provider try to update the ignition config to add the capk user in the secret data, he failed but there is no errors of warnings in the pod logs (cf this func).

If we want to solve the CAPK user creation in the ignition config, we need to modify the kubevirtMachine controller to add the ignition compatibility.

We saw that Kubevirt is currently in "experimental" mode with the ignition support. An issue is actually opened to use "cloudInitConfigDrive" like cloud-init: https://github.com/kubevirt/kubevirt/pull/9212#issuecomment-1434436265

Possible anwsers could be:

1. Wait for the Kubevirt ignition "cloudInitConfigDrive" to be merged, and warn ignition users that ssh bootstrap check is not supported at all in favor or "none" strategy (cf #230 ).

2. Add the ignition compatibility to the machine controller for the CAPK (works for the ssh bootstrap check mode).

Environment

• Cluster-api version: 1.4.2
• Cluster-api-provider-kubevirt version: 0.1.7
• Kubernetes version: (use kubectl version): 1.27.1
• KubeVirt version: 0.59.0
• OS (e.g. from /etc/os-release): Flatcar 3510.2.1

/kind bug

[mh4ck-Thales]

Kubevirt already support cloudInitConfigDrive to be used with ignition, but this usage is yet to be documented. I've made a PR https://github.com/kubevirt/user-guide/pull/669 documenting this use which is yet to be merged, but you may be able to use what I wrote to solve your problem. Regarding the experimental Ignition support mentioned here, it's no longer supported and supposed to be removed with https://github.com/kubevirt/kubevirt/pull/9212 as you may have seen.

[davidvossel]

Kubevirt already support cloudInitConfigDrive to be used with ignition, but this usage is yet to be documented.

yep, i'm using this for coreos variants and it works.

[davidvossel]

The secret bootstraped by the Cluster API was correctly generated but when the kubevirt provider try to update the ignition config to add the capk user in the secret data, he failed but there is no errors of warnings in the pod logs (cf this func).

We're trying to move away from having our controllers manipulate the bootstrap scripts (whether it's cloud init or ignition). The existing logic that injects the capk user is being supported for backwards compatibility reasons. Some users of capk still depend on that logic, but we don't want to keep expanding on that if possible.

[aamoyel]

Documentation was added, thx @mh4ck-Thales : https://github.com/kubevirt/user-guide/pull/669