Env vs config file error when I tried to use both of them

[ksemele]

What steps did you take and what happened?

I followed this manual from capv for simple test vSphere cluster.

So I skipped Set required environment variables: part because of I have an existing config file here: ~/.cluster-api/clusterctl.yaml

I added 2 extra envs to my env and checked that they are exist:

env | grep EXP_
EXP_CLUSTER_RESOURCE_SET=true
EXP_KUBEADM_BOOTSTRAP_FORMAT_IGNITION=true

When I tried to apply a cluster.yaml file I got an validation error:

kubectl apply -f cluster.yaml
cluster.cluster.x-k8s.io/my-cluster created
vspherecluster.infrastructure.cluster.x-k8s.io/my-cluster created
vspheremachinetemplate.infrastructure.cluster.x-k8s.io/my-cluster created
machinedeployment.cluster.x-k8s.io/my-cluster-md-0 created
clusterresourceset.addons.cluster.x-k8s.io/my-cluster-crs-0 created
secret/my-cluster created
secret/vsphere-config-secret created
configmap/csi-manifests created
secret/cloud-provider-vsphere-credentials created
configmap/cpi-manifests created
Error from server (Invalid): error when creating "cluster.yaml": admission webhook "validation.kubeadmcontrolplane.controlplane.cluster.x-k8s.io" denied the request: KubeadmControlPlane.cluster.x-k8s.io "my-cluster" is invalid: [spec.kubeadmConfigSpec.format: Forbidden: can be set only if the KubeadmBootstrapFormatIgnition feature gate is enabled, spec.kubeadmConfigSpec.ignition: Forbidden: can be set only if the KubeadmBootstrapFormatIgnition feature gate is enabled]
Error from server (Invalid): error when creating "cluster.yaml": admission webhook "validation.kubeadmconfigtemplate.bootstrap.cluster.x-k8s.io" denied the request: KubeadmConfigTemplate.bootstrap.cluster.x-k8s.io "my-cluster-md-0" is invalid: [spec.template.spec.format: Forbidden: can be set only if the KubeadmBootstrapFormatIgnition feature gate is enabled, spec.template.spec.ignition: Forbidden: can be set only if the KubeadmBootstrapFormatIgnition feature gate is enabled]

What did you expect to happen?

Cluster is created.

Cluster API version

1.8.4

Kubernetes version

1.31.1

Anything else you would like to add?

I think there are some errors with envs vs configfile... Because when I added EXP_KUBEADM_BOOTSTRAP_FORMAT_IGNITION: true to my existing ~/.cluster-api/clusterctl.yaml This error is gone.

I also created the task first in the capv repo, but they said that this is probably a problem in clusterctl: https://github.com/kubernetes-sigs/cluster-api-provider-vsphere/issues/3245

Label(s) to be applied

/kind bug One or more /area label. See https://github.com/kubernetes-sigs/cluster-api/labels?q=area for the list of labels.

[k8s-ci-robot]

This issue is currently awaiting triage.

If CAPI contributors determine this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.

[neolit123]

clusterctl uses the viper library by default https://github.com/kubernetes-sigs/cluster-api/blob/f8105895c4bc8431f207eed0348eadbfcb0f67bd/cmd/clusterctl/client/config/reader_viper.go#L78

i tried adding a new line in this unit test: https://github.com/kubernetes-sigs/cluster-api/blob/f8105895c4bc8431f207eed0348eadbfcb0f67bd/cmd/clusterctl/client/config/reader_viper_test.go#L44

t.Setenv("EXP_CLUSTER_RESOURCE_SET", "true")

and reading it later:

t.Log(v.Get("EXP_CLUSTER_RESOURCE_SET"))

it works as expected, i.e. both config from file and env vars are read.

the error comes from https://github.com/kubernetes-sigs/cluster-api/blob/f8105895c4bc8431f207eed0348eadbfcb0f67bd/bootstrap/kubeadm/api/v1beta1/kubeadmconfig_types.go#L287

so seems the feature gate is not read correctly.

but checking from where the FG comes from in this case, it is applied as a flag value by using kustomize: https://github.com/kubernetes-sigs/cluster-api/blob/f8105895c4bc8431f207eed0348eadbfcb0f67bd/bootstrap/kubeadm/config/manager/manager.yaml#L25

and that only works from env vars i.e. viper and config is not in the picture, i would think.

are you sure adding these EXP_ vars in the clusterctl config works? because i cannot confirm any GO code reading them i.e. clusterctl doens't.

[sbueringer]

@ksemele Can you please share at which time you ran which clusterctl commands with which content of the clusterctl.yaml file?