search

jenkinsci / kubernetes-operator

Kubernetes native Jenkins Operator
https://jenkinsci.github.io/kubernetes-operator
Other
597 stars 233 forks source link

couldn't poll data from Jenkins API, invalid status code returned: 401 #996

Closed Rory-Z closed 3 months ago

Rory-Z commented 6 months ago

Describe the bug I deploy Jenkins operator and Jenkins instance by helm, and I got this error log in Jenkins operator pod:

2024-04-02T12:57:08.365Z        DEBUG   controller-jenkins      Reconciling Jenkins     {"cr": "jenkins"}
2024-04-02T12:57:08.367Z        DEBUG   controller-jenkins      Operator credentials secret is present  {"cr": "jenkins"}
2024-04-02T12:57:08.381Z        DEBUG   controller-jenkins      Scripts config map is present   {"cr": "jenkins"}
2024-04-02T12:57:08.394Z        DEBUG   controller-jenkins      Init configuration config map is present        {"cr": "jenkins"}
2024-04-02T12:57:08.413Z        DEBUG   controller-jenkins      Base configuration config map is present        {"cr": "jenkins"}
2024-04-02T12:57:08.413Z        DEBUG   controller-jenkins      GroovyScripts Secret and ConfigMap added watched labels {"cr": "jenkins"}
2024-04-02T12:57:08.413Z        DEBUG   controller-jenkins      ConfigurationAsCode Secret and ConfigMap added watched labels     {"cr": "jenkins"}
2024-04-02T12:57:08.413Z        DEBUG   controller-jenkins      createServiceAccount with annotations map[]     {"cr": "jenkins"}
2024-04-02T12:57:08.507Z        DEBUG   controller-jenkins      Service account, role and role binding are present      {"cr": "jenkins"}
2024-04-02T12:57:08.507Z        DEBUG   controller-jenkins      Extra role bindings are present {"cr": "jenkins"}
2024-04-02T12:57:08.512Z        DEBUG   controller-jenkins      Jenkins HTTP Service is present {"cr": "jenkins"}
2024-04-02T12:57:08.516Z        DEBUG   controller-jenkins      Jenkins slave Service is present        {"cr": "jenkins"}
2024-04-02T12:57:08.516Z        DEBUG   controller-jenkins      Kubernetes resources are present        {"cr": "jenkins"}
2024-04-02T12:57:08.517Z        DEBUG   controller-jenkins      Jenkins master pod is present   {"cr": "jenkins"}
2024-04-02T12:57:08.518Z        DEBUG   controller-jenkins      Jenkins master pod is ready     {"cr": "jenkins"}
2024-04-02T12:57:08.741Z        WARN    controller-jenkins      Reconcile loop failed 10 times with the same errors, giving up: couldn't poll data from Jenkins API, invalid status code returned: 401
github.com/jenkinsci/kubernetes-operator/pkg/client.newClient
        /workspace/pkg/client/jenkins.go:186
github.com/jenkinsci/kubernetes-operator/pkg/client.NewUserAndPasswordAuthorization
        /workspace/pkg/client/jenkins.go:138
github.com/jenkinsci/kubernetes-operator/pkg/configuration.(*Configuration).GetJenkinsClientFromSecret
        /workspace/pkg/configuration/configuration.go:275
github.com/jenkinsci/kubernetes-operator/pkg/configuration.(*Configuration).GetJenkinsClient
        /workspace/pkg/configuration/configuration.go:186
github.com/jenkinsci/kubernetes-operator/pkg/configuration/base.(*JenkinsBaseConfigurationReconciler).Reconcile
        /workspace/pkg/configuration/base/reconciler.go:101
github.com/jenkinsci/kubernetes-operator/controllers.(*JenkinsReconciler).reconcile
        /workspace/controllers/jenkins_controller.go:252
github.com/jenkinsci/kubernetes-operator/controllers.(*JenkinsReconciler).Reconcile
        /workspace/controllers/jenkins_controller.go:136
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:235
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.1
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:198
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
        /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
        /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
        /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
        /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
        /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.UntilWithContext
        /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:99
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1374      {"cr": "jenkins"}
github.com/go-logr/zapr.(*zapLogger).Info
        /go/pkg/mod/github.com/go-logr/zapr@v0.2.0/zapr.go:126
github.com/jenkinsci/kubernetes-operator/controllers.(*JenkinsReconciler).Reconcile
        /workspace/controllers/jenkins_controller.go:157
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:235
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.1
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:198
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
        /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
        /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
        /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
        /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
        /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.UntilWithContext
        /go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:99

To Reproduce Deploy Jenkins operator by Jenkins Helm chart 0.8.0, this is my values.yaml

jenkins:
  enabled: true
  name: jenkins
  namespace: jenkins
  # image is the name (and tag) of the Jenkins instance
  # It's recommended to use LTS (tag: "lts") version
  image: jenkins/jenkins:2.440.2-lts
  # basePlugins are plugins installed and required by the operator
  # Shouldn't contain plugins defined by user
  # You can change their versions here
  basePlugins:
    - name: kubernetes
      version: 4203.v1dd44f5b_1cf9
    - name: workflow-job
      version: 1400.v7fd111b_ec82f
    - name: workflow-aggregator
      version: 596.v8c21c963d92d
    - name: git
      version: 5.2.1
    - name: job-dsl
      version: "1.87"
    - name: configuration-as-code
      version: 1775.v810dc950b_514
    - name: kubernetes-credentials-provider
      version: 1.262.v2670ef7ea_0c5
  # plugins are plugins required by the user
  # You can define plugins here
  plugins:
    - name: pipeline-utility-steps
      version: 2.16.2
    - name: role-strategy
      version: 713.vb_3837801b_8cc
  configuration:
    configurationAsCode:
      - configMapName: jenkins-casc
        content:
          1-node.yaml: |
            jenkins:
              numExecutors: 1
              mode: NORMAL
          2-auth.yaml: |
            jenkins:
              authorizationStrategy:
                roleBased:
                  roles:
                    global:
                    - entries:
                      - user: "jenkins-operator"
                      name: "admin"
                      pattern: ".*"
                      permissions:
                      - "Overall/Administer"
                    - entries:
                      - user: "fake"
                      - group: "authenticated"
                      name: "fake"
                      pattern: ".*"
                      permissions:
                      - "Overall/Read"
                    items:
                    - entries:
                      - user: "fake"
                      - group: "authenticated"
                      name: "fake"
                      pattern: "deploy-serverless"
                      permissions:
                      - "Job/Cancel"
                      - "Run/Delete"
                      - "Job/Build"
                      - "Run/Update"
                      - "Job/Create"
                      - "View/Create"
                      - "View/Delete"
                      - "Job/Read"
                      - "View/Read"
                      - "View/Configure"
                      - "Run/Replay"
              securityRealm:
                local:
                  allowsSignup: false
                  enableCaptcha: false
                  users:
                  - id: "jenkins-operator"
                    name: "jenkins-operator"
                    properties:
                    - "apiToken"
                    - "mailer"
                    - "myView"
                    - preferredProvider:
                        providerId: "default"
                    - "timezone"
                    - "experimentalFlags"
                  - id: "fake"
                    name: "fake"
                    password: "fake"
                    properties:
                    - "apiToken"
                    - mailer:
                        emailAddress: "fake@fake.com"
                    - preferredProvider:
                        providerId: "default"
                    - "experimentalFlags"
          3-seed-jobs.yaml: |
            jobs:
            - script: >
                pipelineJob('deploy-serverless') {
                  definition {
                    cpsScm {
                      scm {
                        git {
                          remote {
                            url('fake.git')
                            credentials('fake')
                          }
                          branch('*/main')
                        }
                      }
                      scriptPath('Jenkinsfile')
                      lightweight()
                    }
                  }
                }

Additional information

Kubernetes version: 

Client Version: v1.27.4
Kustomize Version: v5.0.1
Server Version: v1.29.1-eks-b9c9ed7

Jenkins Operator version:

quay.io/jenkins-kubernetes-operator/operator:v0.8.0

Add error logs about the problem here (operator logs and Kubernetes events).

$ k get events -n jenkins                                                                                
LAST SEEN   TYPE      REASON              OBJECT                                           MESSAGE
14m         Normal    LeaderElection      configmap/c674355f.jenkins.io                    jenkins-jenkins-operator-665d5d579-x6h72_627942be-908a-438a-b9cd-6a4baafd3c1e became leader
14m         Normal    LeaderElection      lease/c674355f.jenkins.io                        jenkins-jenkins-operator-665d5d579-x6h72_627942be-908a-438a-b9cd-6a4baafd3c1e became leader
15m         Normal    Scheduled           pod/jenkins-jenkins-operator-665d5d579-x6h72     Successfully assigned jenkins/jenkins-jenkins-operator-665d5d579-x6h72 to ip-10-0-69-14.us-west-2.compute.internal
15m         Normal    Pulled              pod/jenkins-jenkins-operator-665d5d579-x6h72     Container image "quay.io/jenkins-kubernetes-operator/operator:v0.8.0" already present on machine
15m         Normal    Created             pod/jenkins-jenkins-operator-665d5d579-x6h72     Created container jenkins-operator
15m         Normal    Started             pod/jenkins-jenkins-operator-665d5d579-x6h72     Started container jenkins-operator
15m         Normal    SuccessfulCreate    replicaset/jenkins-jenkins-operator-665d5d579    Created pod: jenkins-jenkins-operator-665d5d579-x6h72
15m         Normal    Killing             pod/jenkins-jenkins-operator-6769fc87df-9l8bb    Stopping container jenkins-operator
15m         Normal    SuccessfulDelete    replicaset/jenkins-jenkins-operator-6769fc87df   Deleted pod: jenkins-jenkins-operator-6769fc87df-9l8bb
15m         Normal    ScalingReplicaSet   deployment/jenkins-jenkins-operator              Scaled up replica set jenkins-jenkins-operator-665d5d579 to 1
15m         Normal    ScalingReplicaSet   deployment/jenkins-jenkins-operator              Scaled down replica set jenkins-jenkins-operator-6769fc87df to 0 from 1
15m         Warning                       jenkins/jenkins                                  Reconcile loop failed 10 times with the same errors, giving up: couldn't poll data from Jenkins API, invalid status code returned: 401
4m37s       Warning                       jenkins/jenkins                                  Reconcile loop failed 10 times with the same errors, giving up: couldn't poll data from Jenkins API, invalid status code returned: 401
Rory-Z commented 6 months ago

And I can use username and password from secret jenkins-operator-credentials-jenkins login Jenkins dashboard.