search

jenkinsci / helm-charts

Jenkins helm charts
https://artifacthub.io/packages/helm/jenkinsci/jenkins
Apache License 2.0
562 stars 890 forks source link

installPlugins issues, git version behind other plugins, kubernets version unable to deploy agents #911

Closed SimonStiil closed 6 months ago

SimonStiil commented 1 year ago

Describe the bug

Multiple plugins have started requirering git:5.2.0. But the chart still selects git:5.1.0

There seems to be a bug in kubernetes:3937.vd7b_82dbe347b that preverts jenkins from starting agents in kubernetes with an error about empty namespace even though namespace for agents is defined in chart. This error is fixed in later versions of the plugin. Using kubernetes:4029.v5712230ccb_f8 it works without issues. Probably will work with earlier builds but have only tried the one.

Version of Helm and Kubernetes

- flux: v2.0.1
- Kubernetes: v1.25.9

Chart version

jenkins-4.6.1

What happened?

1. Upgraded helm chart to v1.25.9
2. Chart fetched new plugins
3. Chart unable to start due to pinned git:5.1.0
4. Tried setting git to git:5.2.0
5. Start a build that should start a agentpod.
6. agentpod would not start with a long log with fabric8 errors (dependency of kubernetes plugin)

What you expected to happen?

Upgrade chart. Jenkins starts. Jenkins able to build in agent pod.

How to reproduce it

controller:
      additionalPlugins:
      - ldap
      - http_request
      - pipeline-maven
      - pipeline-utility-steps
      - pipeline-stage-view
      - pipeline-github
      - github-api
      - github-branch-source
      - github-autostatus
      - job-dsl
      - basic-branch-build-strategies
      - email-ext
      - mailer
      - timestamper
      - kubernetes-credentials-provider
      - credentials-binding
      - generic-webhook-trigger
    persistence:
      storageClass: nfs-csi
    rbac:
      create: true
      readSecrets: true

Anything else we need to know?

no

jseiser commented 1 year ago

We have a prod/dev setup with jenkins, and they are deploying using the same helm chart + values except for the ingress name.

We randomly started getting this error, on a working system ( but only in dev )

2023-09-20 16:23:21.772+0000 [id=38]    INFO    hudson.slaves.NodeProvisioner#update: default-9tcrd provisioning successfully completed. We have now 2 computer(s)
2023-09-20 16:23:21.782+0000 [id=2317]  WARNING o.c.j.p.k.KubernetesLauncher#launch: Error in provisioning; agent=KubernetesSlave name: default-9tcrd, template=PodTemplate{id='a6e410bbc54d23e84ac3f58bd1608da0e09b1ef712ce19292fc4c1ee6c8add69', name='default', namespace='jenkins', slaveConnectTimeout=100, label='jenkins-jenkins-agent', serviceAccount='jenkins-worker', nodeUsageMode=NORMAL, podRetention='Never', containers=[ContainerTemplate{name='jnlp', image='jenkins/inbound-agent:3107.v665000b_51092-15', workingDir='/home/jenkins/agent', args='${computer.jnlpmac} ${computer.name}', resourceRequestCpu='512m', resourceRequestMemory='512Mi', resourceLimitCpu='512m', resourceLimitMemory='512Mi', envVars=[KeyValueEnvVar [getValue()=http://jenkins.jenkins.svc.cluster.local:8080/, getKey()=JENKINS_URL]]}]}
java.lang.IncompatibleClassChangeError: Found class io.fabric8.kubernetes.api.model.VolumeFluent$EmptyDirNested, but interface was expected
    at org.csanchez.jenkins.plugins.kubernetes.volumes.workspace.EmptyDirWorkspaceVolume.buildVolume(EmptyDirWorkspaceVolume.java:63)
    at org.csanchez.jenkins.plugins.kubernetes.PodTemplateBuilder.build(PodTemplateBuilder.java:211)
    at org.csanchez.jenkins.plugins.kubernetes.PodTemplate.build(PodTemplate.java:902)
    at org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher.launch(KubernetesLauncher.java:119)
    at hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:297)
    at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
    at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:80)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
2023-09-20 16:23:21.782+0000 [id=2317]  INFO    o.c.j.p.k.KubernetesSlave#_terminate: Terminating Kubernetes instance for agent default-9tcrd
2023-09-20 16:23:21.782+0000 [id=2317]  WARNING o.c.j.p.k.KubernetesSlave#deleteSlavePod: Failed to delete pod for agent null/default-9tcrd: namespace cannot be null
io.fabric8.kubernetes.client.KubernetesClientException: namespace cannot be null
    at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.inNamespace(BaseOperation.java:245)
    at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.inNamespace(BaseOperation.java:92)
    at org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave.deleteSlavePod(KubernetesSlave.java:357)
    at org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave._terminate(KubernetesSlave.java:343)
    at hudson.slaves.AbstractCloudSlave.terminate(AbstractCloudSlave.java:88)
    at org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher.launch(KubernetesLauncher.java:259)
    at hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:297)
    at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
    at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:80)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
2023-09-20 16:23:21.788+0000 [id=2317]  INFO    o.c.j.p.k.KubernetesSlave#_terminate: Disconnected computer default-9tcrd

I assume this is the name space issue you are referring too

ArtursTrubacs commented 6 months ago

Also have a problem with Jenkins, Jenkins not build a job, jenkins job stands with error - Still waiting to schedule task Waiting for next available executor


podName-65504-h41ws-6-d4z19 provisioning successfully completed. We have now 1 computer(s)
Apr 04, 2024 11:40:41 AM WARNING org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
Error in provisioning; agent=KubernetesSlave name: podName-65504-h41ws-6-39f42, template=PodTemplate{id='63c841c4-7ab5-4c5f-bb54-19bb775a2d81', name='CRONJOBS_JUPYTER_jupyter-trip-planning-data_65504-h41ws-6krfr', namespace='jenkins', label='CRONJOBS_JUPYTER_jupyter-trip-planning-data_65504-h41ws', annotations=[PodAnnotation{key='buildUrl', value='http://jenkins.jenkins.svc.cluster.local:8080/job/CRONJOBS/job/JUPYTER/job/jupyter-trip-planning-data/65504/'}, PodAnnotation{key='runUrl', value='job/CRONJOBS/job/JUPYTER/job/jupyter-trip-planning-data/65504/'}]}
java.lang.IncompatibleClassChangeError: Found class io.fabric8.kubernetes.api.model.VolumeFluent$EmptyDirNested, but interface was expected
    at org.csanchez.jenkins.plugins.kubernetes.volumes.workspace.EmptyDirWorkspaceVolume.buildVolume(EmptyDirWorkspaceVolume.java:63)
    at org.csanchez.jenkins.plugins.kubernetes.PodTemplateBuilder.build(PodTemplateBuilder.java:211)
    at org.csanchez.jenkins.plugins.kubernetes.PodTemplate.build(PodTemplate.java:902)
    at org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher.launch(KubernetesLauncher.java:119)
    at hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:298)
    at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
    at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:80)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)

Apr 04, 2024 11:40:41 AM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
Terminating Kubernetes instance for agent podName-65504-h41ws-6-39f42
Apr 04, 2024 11:40:41 AM WARNING org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave deleteSlavePod
Failed to delete pod for agent null/podName-65504-h41ws-6-39f42: namespace cannot be null
io.fabric8.kubernetes.client.KubernetesClientException: namespace cannot be null
    at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.inNamespace(BaseOperation.java:245)
    at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.inNamespace(BaseOperation.java:92)
    at org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave.deleteSlavePod(KubernetesSlave.java:357)
    at org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave._terminate(KubernetesSlave.java:343)
    at hudson.slaves.AbstractCloudSlave.terminate(AbstractCloudSlave.java:88)
    at org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher.launch(KubernetesLauncher.java:259)
    at hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:298)
    at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
    at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:80)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)

Apr 04, 2024 11:40:41 AM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
Disconnected computer podName-65504-h41ws-6-39f42
Apr 04, 2024 11:40:41 AM INFO hudson.slaves.NodeProvisioner update
podName-65504-h41ws-6-39f42 provisioning successfully completed. We have now 1 computer(s)
Apr 04, 2024 11:40:42 AM WARNING org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
kvanzuijlen commented 6 months ago

@ArtursTrubacs Can you check your plugin versions/see if they're the latest versions?

SimonStiil commented 6 months ago

From my point of view this can be resolved. Running chart version: 5.0.13 And all plugins are up to date.