clusterctl init: Failed to get provider components for the "cluster-api:v1.4.99" provider

[josecastillolema]

What steps did you take and what happened?

$ git branch
  main
* release-1.4

$ cat clusterctl-settings.json 
{
    "providers": ["cluster-api", "bootstrap-kubeadm", "control-plane-kubeadm", "infrastructure-docker", "infrastructure-kubemark"],
    "provider_repos": ["../cluster-api-provider-kubemark"]
}

$ python3 cmd/clusterctl/hack/create-local-repository.py 
/var/home/jose/dev/kubemark/cluster-api/cmd/clusterctl/hack/create-local-repository.py:46: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
  from distutils.dir_util import copy_tree
clusterctl local overrides generated from local repositories for the cluster-api, bootstrap-kubeadm, control-plane-kubeadm, infrastructure-docker, infrastructure-kubemark providers.
in order to use them, please run:
clusterctl init \
   --core cluster-api:v1.4.99 \
   --bootstrap kubeadm:v1.4.99 \
   --control-plane kubeadm:v1.4.99 \
   --infrastructure docker:v1.4.99 \
   --infrastructure kubemark:v0.5.99 \
   --config ~/.cluster-api/dev-repository/config.yaml
please check the documentation for additional steps required for using the docker provider

$ tree .cluster-api/
.cluster-api/
└── dev-repository
    ├── bootstrap-kubeadm
    │   └── v1.4.99
    │       ├── bootstrap-components.yaml
    │       └── metadata.yaml
    ├── cluster-api
    │   └── v1.4.99
    │       ├── core-components.yaml
    │       └── metadata.yaml
    ├── config.yaml
    ├── control-plane-kubeadm
    │   └── v1.4.99
    │       ├── control-plane-components.yaml
    │       └── metadata.yaml
    ├── infrastructure-docker
    │   └── v1.4.99
    │       ├── cluster-template-development.yaml
    │       ├── clusterclass-quick-start.yaml
    │       ├── infrastructure-components.yaml
    │       └── metadata.yaml
    └── infrastructure-kubemark
        └── v0.5.99
            ├── infrastructure-components.yaml
            └── metadata.yaml
11 directories, 13 files

$ cat <<EOF | kind create cluster --config -
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: mgmt
networking:
  apiServerAddress: "127.0.0.1"
nodes:
- role: control-plane
  image: docker.io/kindest/node:v1.25.3@sha256:f52781bc0d7a19fb6c405c2af83abfeb311f130707a0e219175677e366cc45d1
  extraMounts:
    - hostPath: /var/run/docker.sock
      containerPath: /var/run/docker.sock
containerdConfigPatches:
- |-
  [plugins."io.containerd.grpc.v1.cri".registry.mirrors."localhost:5000"]
    endpoint = ["http://kind-registry:5000"]
EOF

$ clusterctl init \
   --core cluster-api:v1.4.99 \
   --bootstrap kubeadm:v1.4.99 \
   --control-plane kubeadm:v1.4.99 \
   --infrastructure docker:v1.4.99 \
   --infrastructure kubemark:v0.5.99 \
   --config ~/.cluster-api/dev-repository/config.yaml
Fetching providers
Error: failed to get provider components for the "cluster-api:v1.4.99" provider: failed to set the TargetNamespace on the components: unable to convert unstructured object to apiextensions.k8s.io/v1, Kind=CustomResourceDefinition: parsing time "null" as "2006-01-02T15:04:05Z07:00": cannot parse "null" as "2006"

What did you expect to happen?

For the clusterctl init command to succeed

Cluster API version

$ git branch
  main
* release-1.4

Kubernetes version

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.2", GitCommit:"b05f7d40f9a2dac30771be620e9e9148d26ffd07", GitTreeState:"clean", BuildDate:"2023-02-28T00:59:02Z", GoVersion:"go1.19.4", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.3", GitCommit:"434bfd82814af038ad94d62ebe59b133fcb50506", GitTreeState:"clean", BuildDate:"2022-10-25T19:35:11Z", GoVersion:"go1.19.2", Compiler:"gc", Platform:"linux/amd64"}

Anything else you would like to add?

Could it be related to failed applied CustomResourceDefinitions containing integer validation rules using server-side apply? Something similar to https://cloud.google.com/kubernetes-engine/docs/release-notes#June_25_2021

Label(s) to be applied

/kind bug One or more /area label. See https://github.com/kubernetes-sigs/cluster-api/labels?q=area for the list of labels.

[killianmuldoon]

/triage accepted

I tested this out on the 1.3 branch and the same error exists there. It seems to be related to the marshalling of the creationTimestamp. Manually changing it from "null" to null (without quotations) in the manifests fixed the issue.

I'll try to find a fix for this when generating the CRDs.

/assign

[killianmuldoon]

Looks like: https://github.com/kubernetes-sigs/kustomize/issues/5031

Looks like an issue between kubebuilder - which adds CreationTimeStamp: null - and kustomize - which transforms it to CreationTimeStamp: "null".

Let me try to figure out if we can get an upstream fix for this, or if we can remediate in CAPI.

[fabriziopandini]

@killianmuldoon I think we should modify cmd/clusterctl/hack/create-local-repository.py to use kustomize from hack/bin directory, so we use a well know version

[chrischdi]

I thin kustomize changed to render null instead of dropping it recently. Maybe that's related.

[killianmuldoon]

Pinning to the local kustomize version - 4.5.2 - works to solve this. I'll open an issue about updating to kustomize 5. It looks like there might be a few issues to tackle.

https://github.com/kubernetes-sigs/cluster-api/pull/8414