ks-jenkins containers failed to deploy

[sokuyucu]

I have enabled the devops plugin for my kubesphere cluster with 3 master 3 worker nodes, after the installation I have tried to create a DevOps project. Although the project was created, I was not able to access the pipelines tab inside the DevOps project but i could access the Project Management tab.

After a few tries I decided to check the DevOps plugin and I saw that both the ks-jenkins and ks-jenkins-agent components were in warning status, so i proceeded to observe the workloads of kubesphere-devops-system namespace and ks-jenkins workload was in updating status with MinimumReplicasUnavaiable warning.

There was only one pod that had 2 containers with waiting status.

• The Init container 'copy-default-config' was restarting with CrashLoopBackOff warning
• the 'ks-jenkins' container was in PodInitializing warning

Init container's log: cp: cannot stat '/var/jenkins_config/*.groovy': No such file or directory

kubernetes varsion: 1.18.6 kubesphere platform version: 3.0.0 clusterRole: host 3 master 3 worker each has 4 CPU and 16GB ram storage: ceph 30GB/150GB avaiable

enabled plugins: -service mesh -openpitrix -notification -networkpolicy -logging -events -devops -alerting -auditing

ks-jenkins workload's YAML

  kind: Deployment
  apiVersion: apps/v1
  metadata:
    name: ks-jenkins
    namespace: kubesphere-devops-system
    labels:
      app.kubernetes.io/managed-by: Helm
      chart: jenkins-0.19.0
      component: ks-jenkins-jenkins-master
      heritage: Helm
      release: ks-jenkins
    annotations:
      deployment.kubernetes.io/revision: '23'
      meta.helm.sh/release-name: ks-jenkins
      meta.helm.sh/release-namespace: kubesphere-devops-system
  spec:
    replicas: 1
    selector:
      matchLabels:
        component: ks-jenkins-jenkins-master
    template:
      metadata:
        creationTimestamp: null
        labels:
          app: ks-jenkins
          chart: jenkins-0.19.0
          component: ks-jenkins-jenkins-master
          heritage: Helm
          release: ks-jenkins
        annotations:
          checksum/config: 650abeeb17f701de473fd9c7a16b46920febad327ab5adf2fd2e04bf5a4782b2
          kubesphere.io/restartedAt: '2020-11-24T13:40:16.241Z'
      spec:
        volumes:
          - name: casc-config
            configMap:
              name: jenkins-casc-config
              defaultMode: 420
          - name: jenkins-config
            configMap:
              name: ks-jenkins
              defaultMode: 420
          - name: secrets-dir
            emptyDir: {}
          - name: jenkins-home
            persistentVolumeClaim:
              claimName: ks-jenkins
        initContainers:
          - name: copy-default-config
            image: 'kubespheredev/ks-jenkins:2.176.2'
            command:
              - sh
              - /var/jenkins_config/apply_config.sh
            env:
              - name: JENKINS_UC_DOWNLOAD
                value: 'http://uc-jenkins-update-center'
            resources:
              limits:
                cpu: '1'
                memory: 2Gi
              requests:
                cpu: 100m
                memory: 1500Mi
            volumeMounts:
              - name: jenkins-home
                mountPath: /var/jenkins_home
              - name: jenkins-config
                mountPath: /var/jenkins_config
              - name: secrets-dir
                mountPath: /usr/share/jenkins/ref/secrets/
            terminationMessagePath: /dev/termination-log
            terminationMessagePolicy: File
            imagePullPolicy: IfNotPresent
        containers:
          - name: ks-jenkins
            image: 'kubespheredev/ks-jenkins:2.176.2'
            args:
              - '--argumentsRealm.passwd.$(ADMIN_USER)=$(ADMIN_PASSWORD)'
              - '--argumentsRealm.roles.$(ADMIN_USER)=admin'
            ports:
              - name: http
                containerPort: 8080
                protocol: TCP
              - name: slavelistener
                containerPort: 50000
                protocol: TCP
            env:
              - name: JAVA_TOOL_OPTIONS
                value: >-
                  -Xms512m -Xmx512m -XX:MaxRAM=2g
                  -Dhudson.slaves.NodeProvisioner.initialDelay=20
                  -Dhudson.slaves.NodeProvisioner.MARGIN=50
                  -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85
                  -Dhudson.model.LoadStatistics.clock=5000
                  -Dhudson.model.LoadStatistics.decay=0.2
                  -Dhudson.slaves.NodeProvisioner.recurrencePeriod=5000
                  -Dio.jenkins.plugins.casc.ConfigurationAsCode.initialDelay=10000
                  -verbose:gc -Xloggc:/var/jenkins_home/gc-%t.log
                  -XX:NumberOfGCLogFiles=2 -XX:+UseGCLogFileRotation
                  -XX:GCLogFileSize=100m -XX:+PrintGC -XX:+PrintGCDateStamps
                  -XX:+PrintGCDetails -XX:+PrintHeapAtGC -XX:+PrintGCCause
                  -XX:+PrintTenuringDistribution -XX:+PrintReferenceGC
                  -XX:+PrintAdaptiveSizePolicy -XX:+UseG1GC
                  -XX:+UseStringDeduplication -XX:+ParallelRefProcEnabled
                  -XX:+DisableExplicitGC -XX:+UnlockDiagnosticVMOptions
                  -XX:+UnlockExperimentalVMOptions 
              - name: JENKINS_OPTS
              - name: ADMIN_PASSWORD
                valueFrom:
                  secretKeyRef:
                    name: ks-jenkins
                    key: jenkins-admin-password
              - name: ADMIN_USER
                valueFrom:
                  secretKeyRef:
                    name: ks-jenkins
                    key: jenkins-admin-user
              - name: CASC_JENKINS_CONFIG
                value: /var/jenkins_home/casc_configs/jenkins.yaml
              - name: com.sun.jndi.ldap.connect.timeout
                value: '15000'
              - name: com.sun.jndi.ldap.read.timeout
                value: '60000'
              - name: kubernetes.connection.timeout
                value: '60000'
              - name: kubernetes.request.timeout
                value: '60000'
              - name: EMAIL_SMTP_HOST
                value: mail.example.com
              - name: EMAIL_SMTP_PORT
                value: '465'
              - name: EMAIL_USE_SSL
                value: 'False'
              - name: EMAIL_FROM_NAME
                value: KubeSphere
              - name: EMAIL_FROM_ADDR
                value: admin@example.com
              - name: EMAIL_FROM_PASS
                value: P@ssw0rd
            resources:
              limits:
                cpu: '1'
                memory: 2Gi
              requests:
                cpu: 100m
                memory: 1500Mi
            volumeMounts:
              - name: casc-config
                readOnly: true
                mountPath: /var/jenkins_home/casc_configs
              - name: jenkins-home
                mountPath: /var/jenkins_home
              - name: jenkins-config
                readOnly: true
                mountPath: /var/jenkins_config
              - name: secrets-dir
                mountPath: /usr/share/jenkins/ref/secrets/
            terminationMessagePath: /dev/termination-log
            terminationMessagePolicy: File
            imagePullPolicy: IfNotPresent
        restartPolicy: Always
        terminationGracePeriodSeconds: 30
        dnsPolicy: ClusterFirst
        serviceAccountName: ks-jenkins
        serviceAccount: ks-jenkins
        securityContext:
          runAsUser: 0
        schedulerName: default-scheduler
    strategy:
      type: Recreate
    revisionHistoryLimit: 10
    progressDeadlineSeconds: 600

PS: sorry for so many edits, it is my first issue

[LinuxSuRen]

Probably we can just ignore the cp command errors. See also ks-installer/devops/jenkins.

[sokuyucu]

Probably we can just ignore the cp command errors. See also ks-installer/devops/jenkins.

that fixed the problem. thanks!

[LinuxSuRen]

Actually, this is a potential issue. I'll keep it open until it is confirmed.

[LinuxSuRen]

Fixed