CronJob tutorial results in error at step 1.9

[pauldotyu]

What broke? What's expected?

Packages that were included when initializing the project may be out of date. After running the command go get -u I can run the make install command successfully.

Reproducing this issue

Follow the CronJob tutorial on book.kubebuilder.io. On Step 1.9, run make manifests then make install and you will see the following error:

make install                                                                                                                                                         zsh  ﮫ 798ms  18:44:53 
test -s /home/paul/repos/pauldotyu/kubebuilder-book-tutorial/bin/controller-gen && /home/paul/repos/pauldotyu/kubebuilder-book-tutorial/bin/controller-gen --version | grep -q v0.11.1 || \
GOBIN=/home/paul/repos/pauldotyu/kubebuilder-book-tutorial/bin go install sigs.k8s.io/controller-tools/cmd/controller-gen@v0.11.1
/home/paul/repos/pauldotyu/kubebuilder-book-tutorial/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
test -s /home/paul/repos/pauldotyu/kubebuilder-book-tutorial/bin/kustomize || { curl -Ss "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh" | bash -s -- 3.8.7 /home/paul/repos/pauldotyu/kubebuilder-book-tutorial/bin; }
{Version:kustomize/v3.8.7 GitCommit:ad092cc7a91c07fdf63a2e4b7f13fa588a39af4f BuildDate:2020-11-11T23:14:14Z GoOs:linux GoArch:amd64}
kustomize installed to /home/paul/repos/pauldotyu/kubebuilder-book-tutorial/bin/kustomize
/home/paul/repos/pauldotyu/kubebuilder-book-tutorial/bin/kustomize build config/crd | kubectl apply -f -
The CustomResourceDefinition "cronjobs.batch.tutorial.kubebuilder.io" is invalid: 
* spec.validation.openAPIV3Schema.properties[spec].properties[jobTemplate].properties[spec].properties[template].properties[spec].properties[containers].items.properties[resources].properties[claims].items.x-kubernetes-map-type: Invalid value: "null": must be atomic as item of a list with x-kubernetes-list-type=set
* spec.validation.openAPIV3Schema.properties[spec].properties[jobTemplate].properties[spec].properties[template].properties[spec].properties[initContainers].items.properties[resources].properties[claims].items.x-kubernetes-map-type: Invalid value: "null": must be atomic as item of a list with x-kubernetes-list-type=set
* spec.validation.openAPIV3Schema.properties[spec].properties[jobTemplate].properties[spec].properties[template].properties[spec].properties[ephemeralContainers].items.properties[resources].properties[claims].items.x-kubernetes-map-type: Invalid value: "null": must be atomic as item of a list with x-kubernetes-list-type=set
* spec.validation.openAPIV3Schema.properties[spec].properties[jobTemplate].properties[spec].properties[template].properties[spec].properties[volumes].items.properties[ephemeral].properties[volumeClaimTemplate].properties[spec].properties[resources].properties[claims].items.x-kubernetes-map-type: Invalid value: "null": must be atomic as item of a list with x-kubernetes-list-type=set
make: *** [Makefile:107: install] Error 1```

### KubeBuilder (CLI) Version

Version: main.version{KubeBuilderVersion:"3.9.1", KubernetesVendor:"1.26.0", GitCommit:"cbccafa75d58bf6ac84c2f5d34ad045980f551be", BuildDate:"2023-03-08T21:23:07Z", GoOs:"linux", GoArch:"amd64"}

### PROJECT version

3

### Plugin versions

```yaml
go.kubebuilder.io/v3

Other versions

go version go1.20.2 linux/amd64

Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.3", GitCommit:"9e644106593f3f4aa98f8a84b23db5fa378900bd", GitTreeState:"clean", BuildDate:"2023-03-15T13:40:17Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"} Kustomize Version: v4.5.7 Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.3", GitCommit:"434bfd82814af038ad94d62ebe59b133fcb50506", GitTreeState:"clean", BuildDate:"2022-10-25T19:35:11Z", GoVersion:"go1.19.2", Compiler:"gc", Platform:"linux/amd64"}

Extra Labels

No response

[xiao-jay]

I think i can try to solve /assign

[xiao-jay]

@pauldotyu maybe you can list your cronjobs.batch.tutorial.kubebuilder.io file, in config/crd/base/cronjobs.batch.tutorial.kubebuilder.io. Better is you can push your code to your github ,that i can make install your code.

[xiao-jay]

@pauldotyu but your Kustomize Version: version looks like 3.8.7 instead of v4.5.7. Second,your golang version is go1.20.2 or GoVersion:"go1.19.2", Compiler:"gc", Platform:"linux/amd64"}?

[camilamacedo86]

Hi @xiao-jay,

We updated the tutorial last week. Could you please verify if it is not sorted out now?

[xiao-jay]

my computer is mac m1 ，Can't reproduce his mistake, I'm normal.

➜  test2 kubebuilder version
Version: main.version{KubeBuilderVersion:"v3.9.0-48-g73e25fad-dirty", KubernetesVendor:"unknown", GitCommit:"73e25fadb93daa1e240b4dda4df7c9016ef4d3ce", BuildDate:"2023-03-04T05:50:09Z", GoOs:"darwin", GoArch:"arm64"}

[pauldotyu]

@pauldotyu maybe you can list your cronjobs.batch.tutorial.kubebuilder.io file, in config/crd/base/cronjobs.batch.tutorial.kubebuilder.io. Better is you can push your code to your github ,that i can make install your code.

I've just created a new project and only added code to /api/v1/cronjob_types.go following steps 1.4 and 1.5. The controller logic is not implemented but running a make install command results in the same error listed above

You can duplicate the issue here 👉 https://github.com/pauldotyu/cronjob-operator

[pauldotyu]

@pauldotyu but your Kustomize Version: version looks like 3.8.7 instead of v4.5.7. Second,your golang version is go1.20.2 or GoVersion:"go1.19.2", Compiler:"gc", Platform:"linux/amd64"}?

I don't have kustomize installed locally so it looks like the make install command installed version 3.8.7 for me.

Here is the log.

$ make install
/home/paul/repos/pauldotyu/cronjob-operator/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
test -s /home/paul/repos/pauldotyu/cronjob-operator/bin/kustomize || { curl -Ss "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh" | bash -s -- 3.8.7 /home/paul/repos/pauldotyu/cronjob-operator/bin; }
{Version:kustomize/v3.8.7 GitCommit:ad092cc7a91c07fdf63a2e4b7f13fa588a39af4f BuildDate:2020-11-11T23:14:14Z GoOs:linux GoArch:amd64}
kustomize installed to /home/paul/repos/pauldotyu/cronjob-operator/bin/kustomize

Also here is my go version.

$ go version
go version go1.20.2 linux/amd64

And here is my kubebuilder version.

$ kubebuilder version
Version: main.version{KubeBuilderVersion:"3.9.1", KubernetesVendor:"1.26.0", GitCommit:"cbccafa75d58bf6ac84c2f5d34ad045980f551be", BuildDate:"2023-03-08T21:23:07Z", GoOs:"linux", GoArch:"amd64"}

[xiao-jay]

I found the error too, just appears in kubebuilder v3.9.1，and next code lead to error.

// CronJobSpec defines the desired state of CronJob
type CronJobSpec struct {

    // Specifies the job that will be created when executing a CronJob.
    JobTemplate batchv1.JobTemplateSpec `json:"jobTemplate"`
}

You just exec make manifests && make install

error info
* spec.validation.openAPIV3Schema.properties[spec].properties[jobTemplate].properties[spec].properties[template].properties[spec].properties[containers].items.properties[resources].properties[claims].items.x-kubernetes-map-type: Invalid value: "null": must be atomic as item of a list with x-kubernetes-list-type=set
* spec.validation.openAPIV3Schema.properties[spec].properties[jobTemplate].properties[spec].properties[template].properties[spec].properties[volumes].items.properties[ephemeral].properties[volumeClaimTemplate].properties[spec].properties[resources].properties[claims].items.x-kubernetes-map-type: Invalid value: "null": must be atomic as item of a list with x-kubernetes-list-type=set
* spec.validation.openAPIV3Schema.properties[spec].properties[jobTemplate].properties[spec].properties[template].properties[spec].properties[initContainers].items.properties[resources].properties[claims].items.x-kubernetes-map-type: Invalid value: "null": must be atomic as item of a list with x-kubernetes-list-type=set
* spec.validation.openAPIV3Schema.properties[spec].properties[jobTemplate].properties[spec].properties[template].properties[spec].properties[ephemeralContainers].items.properties[resources].properties[claims].items.x-kubernetes-map-type: Invalid value: "null": must be atomic as item of a list with x-kubernetes-list-type=set

[xiao-jay]

problem solving

change k8s.io/api v0.26.0 => k8s.io/api v0.26.1 in go.mod

[pauldotyu]

problem solving change k8s.io/api v0.26.0 => k8s.io/api v0.26.1 in go.mod

I noticed running go get -u resolves the issue as well.

[xiao-jay]

@camilamacedo86 what's the mean of+listType=set and +listType=map k8s.io/api v0.26.0/core/v1/types.go:2328line

    // Claims lists the names of resources, defined in spec.resourceClaims,
    // that are used by this container.
    //
    // This is an alpha field and requires enabling the
    // DynamicResourceAllocation feature gate.
    //
    // This field is immutable.
    //
    // +listType=set
    // +featureGate=DynamicResourceAllocation
    // +optional
    Claims []ResourceClaim `json:"claims,omitempty" protobuf:"bytes,3,opt,name=claims"`

k8s.io/api v0.26.1/core/v1/types.go:2329line

    // Claims lists the names of resources, defined in spec.resourceClaims,
    // that are used by this container.
    //
    // This is an alpha field and requires enabling the
    // DynamicResourceAllocation feature gate.
    //
    // This field is immutable.
    //
    // +listType=map
    // +listMapKey=name
    // +featureGate=DynamicResourceAllocation
    // +optional
    Claims []ResourceClaim `json:"claims,omitempty" protobuf:"bytes,3,opt,name=claims"`

[camilamacedo86]

Hi @xiao-jay,

I think you created the project using some KubeBuilder release which scaffolds by default the layout of go/v3 with the previous dependencies versions and not the master branch (see https://github.com/kubernetes-sigs/kubebuilder/blob/master/CONTRIBUTING.md#how-to-build-kubebuilder-locally).

If you look at https://github.com/kubernetes-sigs/kubebuilder/tree/master/docs/book/src/cronjob-tutorial/testdata/project it is build with go/v4. (kubebuilder init --plugins=go/v4) which is using:

k8s.io/apimachinery v0.26.1 k8s.io/client-go v0.26.1 sigs.k8s.io/controller-runtime v0.14.4

https://github.com/kubernetes-sigs/kubebuilder/blob/master/docs/book/src/cronjob-tutorial/testdata/project/go.mod#L9-L11

And controller-tools version (v0.11.3) https://github.com/kubernetes-sigs/kubebuilder/blob/master/docs/book/src/cronjob-tutorial/testdata/project/Makefile#L136.

Therefore, I think this issue is sorted out by using the upper versions and for the next Kubebuilder release. Could you please check it with the master branch and let us know?

[camilamacedo86]

Just for we have a summary:

This problem occurs because you update the go dependencies incompatible with the controller-runtime version used.

You can fix it by ensuring that your go.mod has the following:

k8s.io/apimachinery v0.26.1 k8s.io/client-go v0.26.1 sigs.k8s.io/controller-runtime v0.14.4

And you are using controller-tools version (v0.11.3) https://github.com/kubernetes-sigs/kubebuilder/blob/master/docs/book/src/cronjob-tutorial/testdata/project/Makefile#L136.

Just to highlight that if users change the go deps manually or the versions used by the artefacts without checking what versions are scaffolded and tested within each Kubebuilder release, then they might end up in scenarios like that where the versions are not compatible, see https://book.kubebuilder.io/quick-start.html#versions-and-supportability

To avoid it, I recommend checking the samples in the test data directory to know what versions are used for each tagged release and ensure that you upgrade the project with compatible ones.

In this way, I think we can close this one as sorted out. @xiao-jay if you see that it is still a problem, please raise a new issue.

[xiao-jay]

Hi @xiao-jay,

I think you created the project using some KubeBuilder release which scaffolds by default the layout of go/v3 with the previous dependencies versions and not the master branch (see https://github.com/kubernetes-sigs/kubebuilder/blob/master/CONTRIBUTING.md#how-to-build-kubebuilder-locally).

If you look at https://github.com/kubernetes-sigs/kubebuilder/tree/master/docs/book/src/cronjob-tutorial/testdata/project it is build with go/v4. (kubebuilder init --plugins=go/v4) which is using:

k8s.io/apimachinery v0.26.1 k8s.io/client-go v0.26.1 sigs.k8s.io/controller-runtime v0.14.4

https://github.com/kubernetes-sigs/kubebuilder/blob/master/docs/book/src/cronjob-tutorial/testdata/project/go.mod#L9-L11

And controller-tools version (v0.11.3) https://github.com/kubernetes-sigs/kubebuilder/blob/master/docs/book/src/cronjob-tutorial/testdata/project/Makefile#L136.

Therefore, I think this issue is sorted out by using the upper versions and for the next Kubebuilder release. Could you please check it with the master branch and let us know?

I have test master kubebuilder,Don't existed this error.

[RafalSkolasinski]

This problem occurs because you update the go dependencies incompatible with the controller-runtime version used.

I have not upgraded any go dependencies and I still got the same error. I was following the kubebuilder book using

Version: main.version{KubeBuilderVersion:"3.9.1", KubernetesVendor:"1.26.0", GitCommit:"cbccafa75d58bf6ac84c2f5d34ad045980f551be", BuildDate:"2023-03-08T21:23:07Z", GoOs:"linux", GoArch:"amd64"}

and had initially in my go.mod

       k8s.io/api v0.26.0
       k8s.io/apimachinery v0.26.0
       k8s.io/client-go v0.26.0
       sigs.k8s.io/controller-runtime v0.14.1

which lead to same errors on make install.

Once upgraded to

    k8s.io/api v0.26.1
    k8s.io/apimachinery v0.26.1
    k8s.io/client-go v0.26.1
    sigs.k8s.io/controller-runtime v0.14.4

it works but I have the feeling that Kubebuilder should scaffold go.mod with right versions to start with?

[mostafahussein]

I had the same go.mod file as @RafalSkolasinski, and updating it as he describes solved the issue

[jumping]

kubebuilder 3.10.0 works.

[RafalSkolasinski]

yes, it seems it has been resolved

[Adembc]

problem solving

change k8s.io/api v0.26.0 => k8s.io/api v0.26.1 in go.mod

Thank you @xiao-jay solved