search

jenkinsci / kubernetes-operator

Kubernetes native Jenkins Operator
https://jenkinsci.github.io/kubernetes-operator
Other
598 stars 235 forks source link

Boot loop with kubernetes version 1.21 #570

Closed robinMcA closed 3 years ago

robinMcA commented 3 years ago

Describe the bug When deploying into a cluster with api version 1.21 the jenkins runner pod is never created. The pod loops from ContainerCreating to Terminating to Pending every 20 secons.

To Reproduce snap install microk8s --classic --channel=1.21 (or the latest version of kind) add Jenkins operator add Jenkins definition from getting started

Additional information

Kubernetes version: Jenkins Operator version: v1alpha2

Add error logs about the problem here (operator logs and Kubernetes events). kubectl version | grep Server

Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"5e58841cce77d4bc13713ad2b91fa0d961e69192", GitTreeState:"clean", BuildDate:"2021-05-21T23:01:33Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"}

kubectl -n jenkins logs jenkins-operator-XXXX

2021-06-04T05:46:58.167Z    INFO    controller-jenkins  base/pod.go:159 Creating a new Jenkins Master Pod jenkins/jenkins-example   {"cr": "example"}
2021-06-04T05:46:58.345Z    INFO    controller-jenkins  base/pod.go:210 Jenkins master pod restarted by operator: Jenkins pod volumes have changed, actual '[{jenkins-home {nil &EmptyDirVolumeSource{Medium:,SizeLimit:<nil>,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}} {scripts {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:jenkins-operator-scripts-example,},Items:[]KeyToPath{},DefaultMode:*511,Optional:nil,} nil nil nil nil nil nil nil nil nil}} {init-configuration {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:jenkins-operator-init-configuration-example,},Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} nil nil nil nil nil nil nil nil nil}} {operator-credentials {nil nil nil nil nil &SecretVolumeSource{SecretName:jenkins-operator-credentials-example,Items:[]KeyToPath{},DefaultMode:*420,Optional:nil,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}} {kube-api-access-c7f7f {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ProjectedVolumeSource{Sources:[]VolumeProjection{VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:nil,ServiceAccountToken:&ServiceAccountTokenProjection{Audience:,ExpirationSeconds:*3607,Path:token,},},VolumeProjection{Secret:nil,DownwardAPI:nil,ConfigMap:&ConfigMapProjection{LocalObjectReference:LocalObjectReference{Name:kube-root-ca.crt,},Items:[]KeyToPath{KeyToPath{Key:ca.crt,Path:ca.crt,Mode:nil,},},Optional:nil,},ServiceAccountToken:nil,},VolumeProjection{Secret:nil,DownwardAPI:&DownwardAPIProjection{Items:[]DownwardAPIVolumeFile{DownwardAPIVolumeFile{Path:namespace,FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,Mode:nil,},},},ConfigMap:nil,ServiceAccountToken:nil,},},DefaultMode:*420,} nil nil nil nil}}]' required '[]'    {"cr": "example"}
SfietKonstantin commented 3 years ago

I'm facing the same issue here. For what I see, the issue seems to be related to volumes that are changing every 20s. More precisely, The kube-api-access volume name changes, making the operator think that there is a change, thus restarting Jenkins.

This seems to be related to the feature BoundServiceAccountTokenVolume, that reached beta in 1.21. Cf https://kubernetes.io/docs/reference/access-authn-authz/_print/.

prryb commented 3 years ago

This issue is already fixed with https://github.com/jenkinsci/kubernetes-operator/pull/550.

Not yet released though.

prryb commented 3 years ago

You could test the nightly build to see if the problem persists on the most recent version.

andlaz commented 3 years ago

i rolled a new image from 38fcdf5d ( master HEAD currently ) and used the chart also from38fcdf5d to deploy the operator.

# k8s version
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1+k3s1", GitCommit:"75dba57f9b1de3ec0403b148c52c348e1dee2a5e", GitTreeState:"clean", BuildDate:"2021-05-21T16:12:33Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/arm"}

here's the operator's values.yaml as deployed

jenkins:
  enabled: false
  namespace: jenkins # apparently this sets the operator 
                     # deployment's WATCH_NAMESPACE envvar
operator:
  image: jenkins-operator:38fcdf5d-dirty
  resources:
    requests:
      memory: "512Mi"
      cpu: "500m"

this is my jenkins CR

apiVersion: jenkins.io/v1alpha2
kind: Jenkins
metadata:
  name: andlaz 
spec:
  master:
    containers:
    - name: jenkins-master
      resources:
        limits:
          cpu: "1500m"
          memory: "1Gi"
        requests:
          cpu: "500m"
          memory: "500Mi"
      livenessProbe:
        failureThreshold: 12
        httpGet:
          path: /login
          port: http
          scheme: HTTP
        initialDelaySeconds: 120
        periodSeconds: 10
        successThreshold: 1
        timeoutSeconds: 5
      readinessProbe:
        failureThreshold: 12
        httpGet:
          path: /login
          port: http
          scheme: HTTP
        initialDelaySeconds: 120
        periodSeconds: 10
        successThreshold: 1
        timeoutSeconds: 5

as you can see i increased initialDelay as suggested in similar issues.

here's what the operator logs after the above jenkins CR hits the API

2021-06-07T20:33:33.053Z    INFO    controller-jenkins  *v1alpha2.Jenkins/andlaz was created    {"cr": "andlaz"}
2021-06-07T20:33:33.053Z    DEBUG   controller-jenkins  Reconciling Jenkins {"cr": "andlaz"}
2021-06-07T20:33:33.053Z    INFO    controller-jenkins  Setting default Jenkins container command   {"cr": "andlaz"}
2021-06-07T20:33:33.053Z    INFO    controller-jenkins  Setting default Jenkins container JAVA_OPTS environment variable    {"cr": "andlaz"}
2021-06-07T20:33:33.053Z    INFO    controller-jenkins  Setting default operator plugins    {"cr": "andlaz"}
2021-06-07T20:33:33.053Z    INFO    controller-jenkins  Setting default Jenkins master service  {"cr": "andlaz"}
2021-06-07T20:33:33.053Z    INFO    controller-jenkins  Setting default Jenkins slave service   {"cr": "andlaz"}
2021-06-07T20:33:33.053Z    INFO    controller-jenkins  Setting default Jenkins API settings    {"cr": "andlaz"}
2021-06-07T20:33:33.095Z    INFO    controller-jenkins  *v1alpha2.Jenkins/andlaz has been updated   {"cr": "andlaz"}
2021-06-07T20:33:33.097Z    DEBUG   controller-jenkins  Reconciling Jenkins {"cr": "andlaz"}
2021-06-07T20:33:33.123Z    DEBUG   controller-jenkins  Operator credentials secret is present  {"cr": "andlaz"}
2021-06-07T20:33:33.144Z    DEBUG   controller-jenkins  Scripts config map is present   {"cr": "andlaz"}
2021-06-07T20:33:33.157Z    DEBUG   controller-jenkins  Init configuration config map is present    {"cr": "andlaz"}
2021-06-07T20:33:33.236Z    DEBUG   controller-jenkins  Base configuration config map is present    {"cr": "andlaz"}
2021-06-07T20:33:33.236Z    DEBUG   controller-jenkins  GroovyScripts Secret and ConfigMap added watched labels {"cr": "andlaz"}
2021-06-07T20:33:33.236Z    DEBUG   controller-jenkins  ConfigurationAsCode Secret and ConfigMap added watched labels   {"cr": "andlaz"}
2021-06-07T20:33:33.236Z    DEBUG   controller-jenkins  createServiceAccount with annotations map[] {"cr": "andlaz"}
2021-06-07T20:33:33.384Z    DEBUG   controller-jenkins  Service account, role and role binding are present  {"cr": "andlaz"}
2021-06-07T20:33:33.384Z    DEBUG   controller-jenkins  Extra role bindings are present {"cr": "andlaz"}
2021-06-07T20:33:33.542Z    DEBUG   controller-jenkins  Jenkins HTTP Service is present {"cr": "andlaz"}
2021-06-07T20:33:33.698Z    DEBUG   controller-jenkins  Jenkins slave Service is present    {"cr": "andlaz"}
2021-06-07T20:33:33.698Z    DEBUG   controller-jenkins  Kubernetes resources are present    {"cr": "andlaz"}
2021-06-07T20:33:33.698Z    INFO    controller-jenkins  Creating a new Jenkins Master Pod jenkins/jenkins-andlaz    {"cr": "andlaz"}
2021-06-07T20:33:33.784Z    WARN    controller-jenkins  Reconcile loop failed: jenkins.jenkins.io "andlaz" not found    {"cr": "andlaz"}
github.com/go-logr/zapr.(*zapLogger).Info
    /go/pkg/mod/github.com/go-logr/zapr@v0.2.0/zapr.go:126
github.com/jenkinsci/kubernetes-operator/controllers.(*JenkinsReconciler).Reconcile
    /workspace/controllers/jenkins_controller.go:175
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:235
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.1
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:198
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
    /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
    /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
    /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
    /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
    /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.UntilWithContext
    /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:99
2021-06-07T20:33:33.784Z    DEBUG   controller-jenkins  Reconciling Jenkins {"cr": "andlaz"}
2021-06-07T20:33:33.785Z    DEBUG   controller-jenkins  Operator credentials secret is present  {"cr": "andlaz"}
2021-06-07T20:33:34.357Z    DEBUG   controller-jenkins  Scripts config map is present   {"cr": "andlaz"}
2021-06-07T20:33:34.836Z    DEBUG   controller-jenkins  Init configuration config map is present    {"cr": "andlaz"}
2021-06-07T20:33:35.152Z    DEBUG   controller-jenkins  Base configuration config map is present    {"cr": "andlaz"}
2021-06-07T20:33:35.152Z    DEBUG   controller-jenkins  GroovyScripts Secret and ConfigMap added watched labels {"cr": "andlaz"}
2021-06-07T20:33:35.152Z    DEBUG   controller-jenkins  ConfigurationAsCode Secret and ConfigMap added watched labels   {"cr": "andlaz"}
2021-06-07T20:33:35.152Z    DEBUG   controller-jenkins  createServiceAccount with annotations map[] {"cr": "andlaz"}
2021-06-07T20:33:36.452Z    DEBUG   controller-jenkins  Service account, role and role binding are present  {"cr": "andlaz"}
2021-06-07T20:33:36.452Z    DEBUG   controller-jenkins  Extra role bindings are present {"cr": "andlaz"}
2021-06-07T20:33:36.590Z    DEBUG   controller-jenkins  Jenkins HTTP Service is present {"cr": "andlaz"}
2021-06-07T20:33:36.761Z    DEBUG   controller-jenkins  Jenkins slave Service is present    {"cr": "andlaz"}
2021-06-07T20:33:36.761Z    DEBUG   controller-jenkins  Kubernetes resources are present    {"cr": "andlaz"}
2021-06-07T20:33:36.761Z    INFO    controller-jenkins  Jenkins master pod restarted by operator:   {"cr": "andlaz"}
2021-06-07T20:33:36.761Z    INFO    controller-jenkins  Jenkins Operator version has changed, actual '' new 'v0.5.0'    {"cr": "andlaz"}
2021-06-07T20:33:36.761Z    INFO    controller-jenkins  Jenkins CR has been replaced    {"cr": "andlaz"}
2021-06-07T20:33:37.681Z    DEBUG   controller-jenkins  Reconciling Jenkins {"cr": "andlaz"}
2021-06-07T20:33:37.681Z    DEBUG   controller-jenkins  Operator credentials secret is present  {"cr": "andlaz"}
2021-06-07T20:33:38.235Z    DEBUG   controller-jenkins  Scripts config map is present   {"cr": "andlaz"}
2021-06-07T20:33:38.510Z    DEBUG   controller-jenkins  Init configuration config map is present    {"cr": "andlaz"}
2021-06-07T20:33:38.783Z    DEBUG   controller-jenkins  Base configuration config map is present    {"cr": "andlaz"}
2021-06-07T20:33:38.783Z    DEBUG   controller-jenkins  GroovyScripts Secret and ConfigMap added watched labels {"cr": "andlaz"}
2021-06-07T20:33:38.783Z    DEBUG   controller-jenkins  ConfigurationAsCode Secret and ConfigMap added watched labels   {"cr": "andlaz"}
2021-06-07T20:33:38.783Z    DEBUG   controller-jenkins  createServiceAccount with annotations map[] {"cr": "andlaz"}
2021-06-07T20:33:39.994Z    DEBUG   controller-jenkins  Service account, role and role binding are present  {"cr": "andlaz"}
2021-06-07T20:33:39.994Z    DEBUG   controller-jenkins  Extra role bindings are present {"cr": "andlaz"}
2021-06-07T20:33:40.021Z    DEBUG   controller-jenkins  Jenkins HTTP Service is present {"cr": "andlaz"}
2021-06-07T20:33:40.242Z    DEBUG   controller-jenkins  Jenkins slave Service is present    {"cr": "andlaz"}
2021-06-07T20:33:40.242Z    DEBUG   controller-jenkins  Kubernetes resources are present    {"cr": "andlaz"}
2021-06-07T20:33:40.242Z    DEBUG   controller-jenkins  Jenkins master pod is present   {"cr": "andlaz"}
2021-06-07T20:33:40.242Z    DEBUG   controller-jenkins  Reconciling Jenkins {"cr": "andlaz"}
2021-06-07T20:33:40.242Z    DEBUG   controller-jenkins  Operator credentials secret is present  {"cr": "andlaz"}
2021-06-07T20:33:40.531Z    DEBUG   controller-jenkins  Scripts config map is present   {"cr": "andlaz"}
2021-06-07T20:33:40.597Z    DEBUG   controller-jenkins  Init configuration config map is present    {"cr": "andlaz"}
2021-06-07T20:33:41.107Z    DEBUG   controller-jenkins  Base configuration config map is present    {"cr": "andlaz"}
2021-06-07T20:33:41.107Z    DEBUG   controller-jenkins  GroovyScripts Secret and ConfigMap added watched labels {"cr": "andlaz"}
2021-06-07T20:33:41.107Z    DEBUG   controller-jenkins  ConfigurationAsCode Secret and ConfigMap added watched labels   {"cr": "andlaz"}
2021-06-07T20:33:41.107Z    DEBUG   controller-jenkins  createServiceAccount with annotations map[] {"cr": "andlaz"}
2021-06-07T20:33:42.276Z    DEBUG   controller-jenkins  Service account, role and role binding are present  {"cr": "andlaz"}
2021-06-07T20:33:42.276Z    DEBUG   controller-jenkins  Extra role bindings are present {"cr": "andlaz"}
2021-06-07T20:33:42.314Z    DEBUG   controller-jenkins  Jenkins HTTP Service is present {"cr": "andlaz"}
2021-06-07T20:33:42.488Z    DEBUG   controller-jenkins  Jenkins slave Service is present    {"cr": "andlaz"}
2021-06-07T20:33:42.488Z    DEBUG   controller-jenkins  Kubernetes resources are present    {"cr": "andlaz"}
2021-06-07T20:33:42.488Z    DEBUG   controller-jenkins  Jenkins master pod is present   {"cr": "andlaz"}

and the jenkins pod itself immediately goes from ContainerCreating to Terminating without anything suspicious logged in events ( seems like it's being shut down externally )

any ideas, please?

andlaz commented 3 years ago

i find the following operator log lines interesting

2021-06-07T20:33:33.784Z    WARN    controller-jenkins  Reconcile loop failed: jenkins.jenkins.io "andlaz" not found    {"cr": "andlaz"}

it's definitely there in the namespace, as set in WATCH_NAMESPACE

then, there's this one:

2021-06-07T20:33:36.761Z    INFO    controller-jenkins  Jenkins master pod restarted by operator:   {"cr": "andlaz"}

Seems like some context is missing from this?

andlaz commented 3 years ago

i will also say that in master, the crd in the chart may be different from what the operator is using to validate at run-time. I can't e.g. spec.configurationAsCode as required at https://github.com/jenkinsci/kubernetes-operator/blob/master/chart/jenkins-operator/crds/jenkins-crd.yaml ( and the API validates them - i was sure to delete the CRD so it can be re-applied from master when i deploy ) , while the operator barks at the CR:

2021-06-07T21:05:43.860Z    WARN    controller-jenkins  Reconcile loop failed: Jenkins.jenkins.io "andlaz" is invalid: [spec.groovyScripts.configurations: Required value, spec.configurationAsCode.configurations: Required value]
github.com/jenkinsci/kubernetes-operator/controllers.(*JenkinsReconciler).setDefaults
    /workspace/controllers/jenkins_controller.go:459
github.com/jenkinsci/kubernetes-operator/controllers.(*JenkinsReconciler).reconcile
    /workspace/controllers/jenkins_controller.go:218
github.com/jenkinsci/kubernetes-operator/controllers.(*JenkinsReconciler).Reconcile
    /workspace/controllers/jenkins_controller.go:136
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:235
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.1
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:198
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
    /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
    /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
    /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
    /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
    /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.UntilWithContext
    /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:99
runtime.goexit
cammoraton commented 3 years ago

I also tried the latest and after fighting the docs being completely wrong(jenkins-operator being gone for controller) and the spec format changing got to where others have.

Ultimately I worked past it by patching out the podvolume check, make container-runtime-build and then started running on v0.5.0-dirty (lines 80 in pkg/configuration/base/pod.go):

if !r.compareVolumes(currentJenkinsMasterPod) {
               messages = append(messages, "Jenkins pod volumes have changed")
               verbose = append(verbose, fmt.Sprintf("Jenkins pod volumes have changed, actual '%v' required '%v'",
                       currentJenkinsMasterPod.Spec.Volumes, r.Configuration.Jenkins.Spec.Master.Volumes))
}

the podVolumes don't change but the secret does. Adjusting the feature does nothing.

For latest - it looks like some major refactoring happened? I added some additional logging (I'm terrible at golang debugging sorry) - what appears to be happening is when the configuration is applied at pkg/configuration/base/pod.go the return value of r.Client.Status().Update(context.TODO(), r.Configuration.Jenkins) is non-nil which causes the pod to requeue. When detectJenkinsMasterPodStartingIssues() runs on subsequent requeues for reconciliation it then never sees r.Configuration.Jenkins.Status.ProvisionStartTime as anything but nil and so fails and requeues forever until the pod terminates, at which point it loops again.

I expect the next release will have to fix this.

prryb commented 3 years ago

@andlaz CRDs are slightly different since 0.5.0 and after I removed the old CRDs, combo nightly operator build + helm chart from master branch worked.

@cammoraton 0.5.0 is the last release that uses the older Operator SDK version. The current master is fully migrated to a newer (and incompatible with the old Operator SDK) version. We are working on releasing new operator and helm chart versions.

KorusMateusz commented 3 years ago

Fixed in https://github.com/jenkinsci/kubernetes-operator/releases/tag/v0.6.0

andlaz commented 3 years ago

i have nuked my 1.20 cluster and tried on a fresh install of 1.21 for v0.6.0 -- same effect. here is the jenkins.jenkins.io CRD, as created by the chart in chart/jenkins-operator. There doesn't seem to be schema/validation though ( NonStructuralSchema ).. am i missing something?

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  creationTimestamp: "2021-06-11T21:33:47Z"
  generation: 1
  name: jenkins.jenkins.io
  resourceVersion: "4545"
  uid: 08043c8b-1ee4-4bb1-87f3-bb65e2d2fd9f
spec:
  conversion:
    strategy: None
  group: jenkins.io
  names:
    kind: Jenkins
    listKind: JenkinsList
    plural: jenkins
    singular: jenkins
  preserveUnknownFields: true
  scope: Namespaced
  versions:
  - name: v1alpha2
    served: true
    storage: true
  - name: v1alpha1
    served: true
    storage: false
status:
  acceptedNames:
    kind: Jenkins
    listKind: JenkinsList
    plural: jenkins
    singular: jenkins
  conditions:
  - lastTransitionTime: "2021-06-11T21:33:47Z"
    message: no conflicts found
    reason: NoConflicts
    status: "True"
    type: NamesAccepted
  - lastTransitionTime: "2021-06-11T21:33:47Z"
    message: the initial names have been accepted
    reason: InitialNamesAccepted
    status: "True"
    type: Established
  - lastTransitionTime: "2021-06-11T21:33:47Z"
    message: 'spec.preserveUnknownFields: Invalid value: true: must be false'
    reason: Violations
    status: "True"
    type: NonStructuralSchema
  storedVersions:
  - v1alpha2

here's the jenkins CRD instance i'm trying to run:

apiVersion: jenkins.io/v1alpha2
kind: Jenkins
metadata:
  name: dind-tests 
spec:
  seedJobs:
  - id: dind-tests
    credentialType: basicSSHUserPrivateKey
    credentialID: repo-reader
    targets: "seed/Jenkinsfile"
    description: "DinDTests"
    repositoryBranch: "develop"
    repositoryUrl: ssh://nas.andlaz/volume1/dind-kube-tests
  master:
    nodeSelector:
      beta.kubernetes.io/arch: amd64
    plugins:
    - name: timestamper
      version: "1.13"
    basePlugins: # don't extend this. you can change versions
    - name: kubernetes
      version: "1.29.2"
    - name: workflow-job
      version: "2.40"
    - name: workflow-aggregator
      version: "2.6"
    - name: git
      version: "4.5.0"
    - name: job-dsl
      version: "1.77"
    - name: configuration-as-code
      version: "1.51"
    - name: kubernetes-credentials-provider
      version: "0.15"
    containers:
    - name: jenkins-master
      image: jenkins/jenkins:lts
      imagePullPolicy: Always
      resources:
        limits:
          cpu: "1500m"
          memory: "1500Mi"
        requests:
          cpu: "500m"
          memory: "500Mi"
      livenessProbe:
        failureThreshold: 12
        httpGet:
          path: /login
          port: http
          scheme: HTTP
        initialDelaySeconds: 120 # extended, just in case plugins take a while
        periodSeconds: 10
        successThreshold: 1
        timeoutSeconds: 5
      readinessProbe:
        failureThreshold: 12
        httpGet:
          path: /login
          port: http
          scheme: HTTP
        initialDelaySeconds: 120
        periodSeconds: 10
        successThreshold: 1

here's how i started the operator:

helm upgrade jenkins-operator ../../workspace/kubernetes-operator/chart/jenkins-operator --install --values operator/values.yaml --namespace jenkins

here's my values.yaml for the chart

jenkins:
  enabled: false
  namespace: jenkins # apparently this sets the operator 
                     # deployment's WATCH_NAMESPACE envvar
operator:
  image: registry.andlaz:15000/jenkins-operator:v0.6.0-arm
  resources:
    requests:
      memory: "512Mi"
      cpu: "500m"  
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: kubernetes.io/arch 
            operator: In
            values:
            - arm

As you can see i'm using a custom image, which is a branch with 2-3 lines of Makefile and Dockerfile changes to produce an arm32 binary and container image

Here's what gets continuously logged in the operator, basically the exact same thing as before.

2021-06-11T21:57:54.198Z    DEBUG   controller-jenkins  Reconciling Jenkins {"cr": "dind-tests"}
2021-06-11T21:57:54.200Z    DEBUG   controller-jenkins  Operator credentials secret is present  {"cr": "dind-tests"}
2021-06-11T21:57:54.531Z    DEBUG   controller-jenkins  Scripts config map is present   {"cr": "dind-tests"}
2021-06-11T21:57:54.638Z    DEBUG   controller-jenkins  Init configuration config map is present    {"cr": "dind-tests"}
2021-06-11T21:57:54.776Z    DEBUG   controller-jenkins  Base configuration config map is present    {"cr": "dind-tests"}
2021-06-11T21:57:54.777Z    DEBUG   controller-jenkins  GroovyScripts Secret and ConfigMap added watched labels {"cr": "dind-tests"}
2021-06-11T21:57:54.778Z    DEBUG   controller-jenkins  ConfigurationAsCode Secret and ConfigMap added watched labels   {"cr": "dind-tests"}
2021-06-11T21:57:54.780Z    DEBUG   controller-jenkins  createServiceAccount with annotations map[] {"cr": "dind-tests"}
2021-06-11T21:57:55.292Z    DEBUG   controller-jenkins  Service account, role and role binding are present  {"cr": "dind-tests"}
2021-06-11T21:57:55.293Z    DEBUG   controller-jenkins  Extra role bindings are present {"cr": "dind-tests"}
2021-06-11T21:57:55.330Z    DEBUG   controller-jenkins  Jenkins HTTP Service is present {"cr": "dind-tests"}
2021-06-11T21:57:55.356Z    DEBUG   controller-jenkins  Jenkins slave Service is present    {"cr": "dind-tests"}
2021-06-11T21:57:55.357Z    DEBUG   controller-jenkins  Kubernetes resources are present    {"cr": "dind-tests"}
2021-06-11T21:57:55.362Z    INFO    controller-jenkins  Creating a new Jenkins Master Pod jenkins/jenkins-dind-tests    {"cr": "dind-tests"}
2021-06-11T21:57:55.470Z    WARN    controller-jenkins  Reconcile loop failed 10 times with the same errors, giving up: jenkins.jenkins.io "dind-tests" not found   {"cr": "dind-tests"}
github.com/go-logr/zapr.(*zapLogger).Info
    /go/pkg/mod/github.com/go-logr/zapr@v0.2.0/zapr.go:126
github.com/jenkinsci/kubernetes-operator/controllers.(*JenkinsReconciler).Reconcile
    /workspace/controllers/jenkins_controller.go:157
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:235
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.1
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:198
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
    /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
    /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
    /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
    /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
    /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.UntilWithContext
    /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:99
2021-06-11T21:57:55.475Z    DEBUG   controller-jenkins  Reconciling Jenkins {"cr": "dind-tests"}
2021-06-11T21:57:55.479Z    DEBUG   controller-jenkins  Operator credentials secret is present  {"cr": "dind-tests"}
2021-06-11T21:57:55.653Z    DEBUG   controller-jenkins  Scripts config map is present   {"cr": "dind-tests"}
2021-06-11T21:57:55.748Z    DEBUG   controller-jenkins  Init configuration config map is present    {"cr": "dind-tests"}
2021-06-11T21:57:55.818Z    DEBUG   controller-jenkins  Base configuration config map is present    {"cr": "dind-tests"}
2021-06-11T21:57:55.819Z    DEBUG   controller-jenkins  GroovyScripts Secret and ConfigMap added watched labels {"cr": "dind-tests"}
2021-06-11T21:57:55.819Z    DEBUG   controller-jenkins  ConfigurationAsCode Secret and ConfigMap added watched labels   {"cr": "dind-tests"}
2021-06-11T21:57:55.819Z    DEBUG   controller-jenkins  createServiceAccount with annotations map[] {"cr": "dind-tests"}
2021-06-11T21:57:56.290Z    DEBUG   controller-jenkins  Service account, role and role binding are present  {"cr": "dind-tests"}
2021-06-11T21:57:56.291Z    DEBUG   controller-jenkins  Extra role bindings are present {"cr": "dind-tests"}
2021-06-11T21:57:56.360Z    DEBUG   controller-jenkins  Jenkins HTTP Service is present {"cr": "dind-tests"}
2021-06-11T21:57:56.392Z    DEBUG   controller-jenkins  Jenkins slave Service is present    {"cr": "dind-tests"}
2021-06-11T21:57:56.393Z    DEBUG   controller-jenkins  Kubernetes resources are present    {"cr": "dind-tests"}
2021-06-11T21:57:56.398Z    INFO    controller-jenkins  Jenkins master pod restarted by operator:   {"cr": "dind-tests"}
2021-06-11T21:57:56.398Z    INFO    controller-jenkins  Jenkins Operator version has changed, actual '' new 'v0.6.0'    {"cr": "dind-tests"}
2021-06-11T21:57:56.399Z    INFO    controller-jenkins  Jenkins CR has been replaced    {"cr": "dind-tests"}
2021-06-11T21:57:56.399Z    INFO    controller-jenkins  Readiness probe has changed to '&Probe{Handler:Handler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/login,Port:{1 0 http},Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,},InitialDelaySeconds:120,TimeoutSeconds:0,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:12,}' in container 'jenkins-master' {"cr": "dind-tests"}
2021-06-11T21:57:56.431Z    DEBUG   controller-jenkins  Reconciling Jenkins {"cr": "dind-tests"}
2021-06-11T21:57:56.433Z    DEBUG   controller-jenkins  Operator credentials secret is present  {"cr": "dind-tests"}
2021-06-11T21:57:56.586Z    DEBUG   controller-jenkins  Scripts config map is present   {"cr": "dind-tests"}
2021-06-11T21:57:56.659Z    DEBUG   controller-jenkins  Init configuration config map is present    {"cr": "dind-tests"}
2021-06-11T21:57:56.796Z    DEBUG   controller-jenkins  Base configuration config map is present    {"cr": "dind-tests"}
2021-06-11T21:57:56.797Z    DEBUG   controller-jenkins  GroovyScripts Secret and ConfigMap added watched labels {"cr": "dind-tests"}
2021-06-11T21:57:56.798Z    DEBUG   controller-jenkins  ConfigurationAsCode Secret and ConfigMap added watched labels   {"cr": "dind-tests"}
2021-06-11T21:57:56.799Z    DEBUG   controller-jenkins  createServiceAccount with annotations map[] {"cr": "dind-tests"}
2021-06-11T21:57:57.369Z    DEBUG   controller-jenkins  Service account, role and role binding are present  {"cr": "dind-tests"}
2021-06-11T21:57:57.371Z    DEBUG   controller-jenkins  Extra role bindings are present {"cr": "dind-tests"}
2021-06-11T21:57:57.395Z    DEBUG   controller-jenkins  Jenkins HTTP Service is present {"cr": "dind-tests"}
2021-06-11T21:57:57.462Z    DEBUG   controller-jenkins  Jenkins slave Service is present    {"cr": "dind-tests"}
2021-06-11T21:57:57.462Z    DEBUG   controller-jenkins  Kubernetes resources are present    {"cr": "dind-tests"}
2021-06-11T21:57:57.464Z    DEBUG   controller-jenkins  Jenkins master pod is present   {"cr": "dind-tests"}
2021-06-11T21:57:57.467Z    DEBUG   controller-jenkins  Reconciling Jenkins {"cr": "dind-tests"}
2021-06-11T21:57:57.470Z    DEBUG   controller-jenkins  Operator credentials secret is present  {"cr": "dind-tests"}
2021-06-11T21:57:57.516Z    DEBUG   controller-jenkins  Scripts config map is present   {"cr": "dind-tests"}
2021-06-11T21:57:57.601Z    DEBUG   controller-jenkins  Init configuration config map is present    {"cr": "dind-tests"}
2021-06-11T21:57:57.741Z    DEBUG   controller-jenkins  Base configuration config map is present    {"cr": "dind-tests"}
2021-06-11T21:57:57.741Z    DEBUG   controller-jenkins  GroovyScripts Secret and ConfigMap added watched labels {"cr": "dind-tests"}
2021-06-11T21:57:57.742Z    DEBUG   controller-jenkins  ConfigurationAsCode Secret and ConfigMap added watched labels   {"cr": "dind-tests"}
2021-06-11T21:57:57.743Z    DEBUG   controller-jenkins  createServiceAccount with annotations map[] {"cr": "dind-tests"}
2021-06-11T21:57:58.269Z    DEBUG   controller-jenkins  Service account, role and role binding are present  {"cr": "dind-tests"}
2021-06-11T21:57:58.270Z    DEBUG   controller-jenkins  Extra role bindings are present {"cr": "dind-tests"}
2021-06-11T21:57:58.304Z    DEBUG   controller-jenkins  Jenkins HTTP Service is present {"cr": "dind-tests"}
2021-06-11T21:57:58.334Z    DEBUG   controller-jenkins  Jenkins slave Service is present    {"cr": "dind-tests"}
2021-06-11T21:57:58.336Z    DEBUG   controller-jenkins  Kubernetes resources are present    {"cr": "dind-tests"}
2021-06-11T21:57:58.336Z    DEBUG   controller-jenkins  Jenkins master pod is present   {"cr": "dind-tests"}

the jenkins master pod itself is just bouncing from Pending to Terminating like before.

I'll be happy to provide any further info or perform any debugging i can.

thanks,

Harguer commented 3 years ago

I'm having the same issue, trying jenkins-operator with arm64, i have configured exactly the same as the documentation says. My k8s cluster is 1.21 too and opertor 0.6 version. wonder if there is a fix for it.

andlaz commented 3 years ago

for me the fix was removing the CRD then letting the 0.6 chart recreate it. Running fine right now. I think the previous minor version installed a NonStructuralSchema under the same CRD version.

rarecrumb commented 2 years ago

Still boot looping on Kubernetes version 1.21, not sure why... This is a fresh install.