jx-admin: Incorrect CustomResourceDefinition api version for for kubernetes 1.22+.

[bhomrich]

During an install onto a new 1.22.10 kubernetes on-prem, jx3 boot pod fails with CRD api version errors.

After following the on-prem install instructions (https://jenkins-x.io/v3/admin/platforms/on-premises/), and getting to the operator install step (jx admin.....)

Logged output is the same as the previously reported issue #10:

saved docs/releases.yaml saved docs/README.md jx gitops scheduler jx gitops hash --pod-spec --kind Deployment -s config-root/namespaces/jx/lighthouse-config/config-cm.yaml -s config-root/namespaces/jx/lighthouse-config/plugins-cm.yaml -d config-root/namespaces/jx/lighthouse jx gitops label --dir config-root/cluster gitops.jenkins-x.io/pipeline=cluster jx gitops label --dir config-root/customresourcedefinitions gitops.jenkins-x.io/pipeline=customresourcedefinitions jx gitops label --dir config-root/namespaces gitops.jenkins-x.io/pipeline=namespaces jx gitops annotate --dir config-root --selector app=pusher-wave kapp.k14s.io/change-group=apps.jenkins-x.io/pusher-wave jx gitops annotate --dir config-root --selector app.kubernetes.io/name=ingress-nginx kapp.k14s.io/change-group=apps.jenkins-x.io/ingress-nginx jx gitops label --dir config-root/cluster --kind=Namespace team=jx jx gitops annotate --dir config-root/namespaces --kind Deployment --selector app=pusher-wave --invert-selector wave.pusher.com/update-on-config-change=true using kubectl to apply resources kubectl apply --force --prune -l=gitops.jenkins-x.io/pipeline=customresourcedefinitions -R -f config-root/customresourcedefinitions customresourcedefinition.apiextensions.k8s.io/environments.jenkins.io configured customresourcedefinition.apiextensions.k8s.io/pipelineactivities.jenkins.io configured customresourcedefinition.apiextensions.k8s.io/releases.jenkins.io configured customresourcedefinition.apiextensions.k8s.io/sourcerepositories.jenkins.io configured customresourcedefinition.apiextensions.k8s.io/previews.preview.jenkins.io unchanged customresourcedefinition.apiextensions.k8s.io/lighthousebreakpoints.lighthouse.jenkins.io unchanged customresourcedefinition.apiextensions.k8s.io/lighthousejobs.lighthouse.jenkins.io unchanged customresourcedefinition.apiextensions.k8s.io/externalsecrets.kubernetes-client.io unchanged customresourcedefinition.apiextensions.k8s.io/clustertasks.tekton.dev configured customresourcedefinition.apiextensions.k8s.io/conditions.tekton.dev unchanged customresourcedefinition.apiextensions.k8s.io/pipelineresources.tekton.dev unchanged customresourcedefinition.apiextensions.k8s.io/pipelineruns.tekton.dev configured customresourcedefinition.apiextensions.k8s.io/pipelines.tekton.dev configured customresourcedefinition.apiextensions.k8s.io/runs.tekton.dev configured customresourcedefinition.apiextensions.k8s.io/taskruns.tekton.dev configured customresourcedefinition.apiextensions.k8s.io/tasks.tekton.dev configured unable to recognize "config-root/customresourcedefinitions/kuberhealthy/kuberhealthy/khchecks.comcast.github.io-crd.yaml": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1" unable to recognize "config-root/customresourcedefinitions/kuberhealthy/kuberhealthy/khjobs.comcast.github.io-crd.yaml": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1" unable to recognize "config-root/customresourcedefinitions/kuberhealthy/kuberhealthy/khstates.comcast.github.io-crd.yaml": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1" make[1]: [versionStream/src/Makefile.mk:289: kubectl-apply] Error 1 make[1]: Leaving directory '/workspace/source' error: failed to regenerate: failed to regenerate phase 1: failed to run 'make regen-phase-1' command in directory '.', output: '' make: [versionStream/src/Makefile.mk:243: regen-check] Error 1 pod jx-boot-8268e686-0799-42b6-b7ab-5a264e2062ae-96z9b has status Ready

I am a novice in understanding how to get the environment to deploy with changed dependencies (my guess is the current specification of kuberhealthy @ version "64" is playing a role in the issue, but I can't seem to make my install change versions at deployment). If there is a primer on how the install/build process works (where it takes inputs to finally build/deploy resources), I'll happily take a pointer and try to help resolve this issue.

[fabian-kramer]

Hi @bhomrich ,

could you solve the issue? If so, could you please share the solution? I'm having the same issue on Kubernetes 1.23.x on Ubuntu 20.04

[babadofar]

To fix this you may remove the line that includes kuberhealthy in helmfile.yaml here: https://github.com/jx3-gitops-repositories/jx3-kubernetes/blob/main/helmfile.yaml#L4

[babadofar]

Here is an overview of how it works https://jenkins-x.io/v3/about/how-it-works/

[CPinhoK]

Issue still occurs in K8s 1.24.10 with the line that includes kuberhealthy in helmfile.yaml removed

unable to recognize "config-root/namespaces/jx/jx-kh-check-health-checks-jx/jx-bot-token-kuberhealthycheck.yaml": no matches for kind "KuberhealthyCheck" in version "comcast.github.io/v1"
unable to recognize "config-root/namespaces/jx/jx-kh-check-health-checks-jx/jx-webhook-events-kuberhealthycheck.yaml": no matches for kind "KuberhealthyCheck" in version "comcast.github.io/v1"
unable to recognize "config-root/namespaces/jx/jx-kh-check-health-checks-jx/jx-webhook-kuberhealthycheck.yaml": no matches for kind "KuberhealthyCheck" in version "comcast.github.io/v1"

[kyleli666]

I tried with newer version kuberhealthy charts, then this problem seems solved, but I still have another issue that tekton cannot strart...(Solved according to https://github.com/tektoncd/pipeline/issues/3367#issuecomment-706988789) However lighthouse webhook still not working ... :(

[stewartshea]

I opened this PR to help resolve the CRD apiVersion issue https://github.com/jx3-gitops-repositories/jx3-kubernetes/pull/14

[zwpaper]

if anyone is trying to disable kuberhealthy and meet errors, should look into the below, which also has a kuberhealthy check, and you should also disable those lines.

https://github.com/jx3-gitops-repositories/jx3-kubernetes/blob/a08f795877bb407b535a14206bece1806563ade0/helmfiles/jx/helmfile.yaml#L57-L62