Jenkins Master Keep restarting at regular intervals

[nukarajusundeep]

After deploying in GKE Jenkins started restarting by it self at regular intervals

[torstenwalter]

Hello, some context e.g. version of the chart used values, log files, kubectl status would help.

[nukarajusundeep]

Here is the values.yaml and I am using Jenkins/stable 1.17.0. And the restarts are not continuous they are random and once restarted it is working fine. But again suddenly restarts. Please let me know what logs I can provide

# Default values for jenkins.
# This is a YAML-formatted file.
# Declare name/value pairs to be passed into your templates.
# name: value

## Overrides for generated resource names
# See templates/_helpers.tpl
# nameOverride:
# fullnameOverride:
namespaceOverride: jenkins

# For FQDN resolving of the master service. Change this value to match your existing configuration.
# ref: https://github.com/kubernetes/dns/blob/master/docs/specification.md
clusterZone: "cluster.local"

master:
  # Used for label app.kubernetes.io/component
  componentName: "jenkins-master"
  image: "jenkins/jenkins"
  tag: "2.232"
  imagePullPolicy: "Always"
  imagePullSecretName:
  # Optionally configure lifetime for master-container
  lifecycle:
  #  postStart:
  #    exec:
  #      command:
  #      - "uname"
  #      - "-a"
  numExecutors: 0
  customJenkinsLabels: ["jenkins"]
  # configAutoReload requires UseSecurity is set to true:
  useSecurity: true

  # enables configuration done directly via XML files
  # People who want to configure Jenkins via https://github.com/jenkinsci/configuration-as-code-plugin only can set it to false
  enableXmlConfig: true
  # Allows to configure different SecurityRealm using Jenkins XML
  securityRealm: |-
    <securityRealm class="hudson.security.LegacySecurityRealm"/>
  # Allows to configure different AuthorizationStrategy using Jenkins XML
  authorizationStrategy: |-
     <authorizationStrategy class="hudson.security.FullControlOnceLoggedInAuthorizationStrategy">
       <denyAnonymousReadAccess>true</denyAnonymousReadAccess>
     </authorizationStrategy>
  hostNetworking: false
  # When enabling LDAP or another non-Jenkins identity source, the built-in admin account will no longer exist.
  # If you disable the non-Jenkins identity store and instead use the Jenkins internal one,
  # you should revert master.adminUser to your preferred admin user:
  adminUser: "admin"
  # adminPassword: <defaults to random>
  # This values should not be changed unless you use your custom image of jenkins or any devired from. If you want to use
  # Cloudbees Jenkins Distribution docker, you should set jenkinsHome: "/var/cloudbees-jenkins-distribution"
  jenkinsHome: "/var/jenkins_home"
  # This values should not be changed unless you use your custom image of jenkins or any devired from. If you want to use
  # Cloudbees Jenkins Distribution docker, you should set jenkinsRef: "/usr/share/cloudbees-jenkins-distribution/ref"
  jenkinsRef: "/usr/share/jenkins/ref"
  rollingUpdate: {}
  # Ignored if Persistence is enabled
  # maxSurge: 1
  # maxUnavailable: 25%
  resources:
    requests:
      cpu: "50m"
      memory: "256Mi"
    limits:
      cpu: "2000m"
      memory: "4096Mi"
  # Environment variables that get added to the init container (useful for e.g. http_proxy)
  # initContainerEnv:
  #   - name: http_proxy
  #     value: "http://192.168.64.1:3128"
  # containerEnv:
  #   - name: http_proxy
  #     value: "http://192.168.64.1:3128"
  # Set min/max heap here if needed with:
  javaOpts: "-Xms8500m -Xmx8500m -XX:NumberOfGCLogFiles=2 -XX:GCLogFileSize=100m -XX:+UseGCLogFileRotation -verbose:gc -XX:+UseG1GC -Xloggc:/var/jenkins_home/userContent/gc/gc-%t.log"
  # -XX:NumberOfGCLogFiles=2 -XX:GCLogFileSize=100m -XX:+UseGCLogFileRotation -verbose:gc -XX:+UseG1GC -Xloggc:/var/jenkins_home/userContent/gc/gc-%t.log
  # javaOpts: "-Xms512m -Xmx512m"
  # jenkinsOpts: ""
  # jenkinsUrl: ""
  # If you set this prefix and use ingress controller then you might want to set the ingress path below
  # jenkinsUriPrefix: "/jenkins"
  # Enable pod security context (must be `true` if runAsUser or fsGroup are set)
  usePodSecurityContext: true
  # Set runAsUser to 1000 to let Jenkins run as non-root user 'jenkins' which exists in 'jenkins/jenkins' docker image.
  # When setting runAsUser to a different value than 0 also set fsGroup to the same value:
  # runAsUser: <defaults to 0>
  # fsGroup: <will be omitted in deployment if runAsUser is 0>
  runAsUser: 1000
  fsGroup: 1000
  servicePort: 8080
  targetPort: 8081
  # For minikube, set this to NodePort, elsewhere use LoadBalancer
  # Use ClusterIP if your setup includes ingress controller
  serviceType: NodePort
  # Jenkins master service annotations
  serviceAnnotations: {}
  # Jenkins master custom labels
  deploymentLabels: {}
  #   foo: bar
  #   bar: foo
  # Jenkins master service labels
  serviceLabels: {}
  #   service.beta.kubernetes.io/aws-load-balancer-backend-protocol: https
  # Put labels on Jenkins master pod
  podLabels: {}
  # Used to create Ingress record (should used with ServiceType: ClusterIP)
  # nodePort: <to set explicitly, choose port between 30000-32767
  # Enable Kubernetes Liveness and Readiness Probes
  # ~ 2 minutes to allow Jenkins to restart when upgrading plugins. Set ReadinessTimeout to be shorter than LivenessTimeout.
  healthProbes: true
  healthProbesLivenessTimeout: 5
  healthProbesReadinessTimeout: 5
  healthProbeLivenessPeriodSeconds: 10
  healthProbeReadinessPeriodSeconds: 10
  healthProbeLivenessFailureThreshold: 5
  healthProbeReadinessFailureThreshold: 3
  healthProbeLivenessInitialDelay: 90
  healthProbeReadinessInitialDelay: 60
  slaveListenerPort: 31000
  slaveHostPort:
  disabledAgentProtocols:
    - JNLP-connect
    - JNLP2-connect
  csrf:
    defaultCrumbIssuer:
      enabled: true
      proxyCompatability: true
  cli: false
  # Kubernetes service type for the JNLP agent service
  # slaveListenerServiceType is the Kubernetes Service type for the JNLP agent service,
  # either 'LoadBalancer', 'NodePort', or 'ClusterIP'
  # Note if you set this to 'LoadBalancer', you *must* define annotations to secure it. By default
  # this will be an external load balancer and allowing inbound 0.0.0.0/0, a HUGE
  # security risk:  https://github.com/kubernetes/charts/issues/1341
#  loadBalancerIP: 35.230.72.55
  slaveListenerServiceType: NodePort
  # Optionally assign an IP to the LoadBalancer slaveListenerService LoadBalancer
  # GKE users: only regional static IPs will work for Service Load balancer.
#  slaveListenerLoadBalancerIP: 35.230.72.55
  slaveListenerServiceAnnotations: {}
  slaveKubernetesNamespace: jenkins

  # Example of 'LoadBalancer' type of agent listener with annotations securing it
  # slaveListenerServiceType: LoadBalancer
  # slaveListenerServiceAnnotations:
  #   service.beta.kubernetes.io/aws-load-balancer-internal: "True"
  #   service.beta.kubernetes.io/load-balancer-source-ranges: "172.0.0.0/8, 10.0.0.0/8"

  # LoadBalancerSourcesRange is a list of allowed CIDR values, which are combined with ServicePort to
  # set allowed inbound rules on the security group assigned to the master load balancer
  loadBalancerSourceRanges:
  - 0.0.0.0/0
  # Optionally assign a known public LB IP
#  loadBalancerIP: 35.230.72.55
  # Optionally configure a JMX port
  # requires additional javaOpts, ie
  # javaOpts: >
  #   -Dcom.sun.management.jmxremote.port=4000
  #   -Dcom.sun.management.jmxremote.authenticate=false
  #   -Dcom.sun.management.jmxremote.ssl=false
  # jmxPort: 4000
  # Optionally configure other ports to expose in the master container
  extraPorts: []
  # - name: BuildInfoProxy
  #   port: 9000

  # List of plugins to be install during Jenkins master start
  installPlugins:
    - kubernetes:1.25.3
    - artifactory:3.6.1
    - google-source-plugin:0.4
    - google-kubernetes-engine:0.8.1
    - google-storage-plugin:1.5.1
    - workflow-support:3.4
    - workflow-job:2.38
    - workflow-aggregator:2.6
    - credentials-binding:1.22
    - active-directory:2.16
    - matrix-auth:2.5
    - ansicolor:0.6.3
    - audit-trail:3.5
    - authorize-project:1.3.0
    - bitbucket:1.1.11
    - bitbucket-build-status-notifier:1.4.2
    - bitbucket-oauth:0.10
    - blueocean:1.23.0
    - branch-api:2.5.6
    - build-history-metrics-plugin:1.2
    - command-launcher:1.4
    - conditional-buildstep:1.3.6
    - config-file-provider:3.6.3
    - confluence-publisher:2.0.5
    - credentials:2.3.7
    - CustomHistory:1.6
    - dependency-check-jenkins-plugin:5.1.1
    - dependency-track:2.2.1
    - docker-build-step:2.4
    - email-ext:2.69
    - embeddable-build-status:2.0.3
    - external-monitor-job:1.7
    - favorite:2.3.2
    - gradle:1.36
    - jacoco:3.0.5
    - jira:3.0.15
    - job-import-plugin:latest
    - jobConfigHistory:latest
    - ldap:latest
    - lockable-resources:latest
    - mailer:latest
    - mapdb-api:latest
    - mercurial:latest
    - momentjs:latest
    - multiple-scms:latest
    - oauth-credentials:latest
    - parameterized-trigger:latest
    - pipeline-model-definition:1.6.0
    - pipeline-aggregator-view:latest
    - pipeline-aws:latest
    - pipeline-utility-steps:latest
    - publish-over:latest
    - publish-over-ssh:latest
    - purge-job-history:latest
    - rebuild:latest
    - role-strategy:latest
    - run-condition:latest
    - schedule-build:latest
    - scm-api:latest
    - script-security:latest
    - secondary-timestamper-plugin:latest
    - sonar:latest
    - sonar-quality-gates:1.3.1
    - ssh-agent:latest
    - ssh-slaves:latest
    - stashNotifier:latest
    - statusmonitor:latest
    - structs:latest
    - systemloadaverage-monitor:latest
    - trilead-api:latest
    - warnings:latest
    - warnings-ng:latest
    - websocket:latest
    - configuration-as-code:latest
    - prometheus:2.0.6
    - datadog:1.1.0

  # Enable to always override the installed plugins with the values of 'master.installPlugins' on upgrade or redeployment.
  overwritePlugins: true
  # Enable HTML parsing using OWASP Markup Formatter Plugin (antisamy-markup-formatter), useful with ghprb plugin.
  # The plugin is not installed by default, please update master.installPlugins.
  enableRawHtmlMarkupFormatter: false
  # Used to approve a list of groovy functions in pipelines used the script-security plugin. Can be viewed under /scriptApproval
  scriptApproval: []
  #  - "method groovy.json.JsonSlurperClassic parseText java.lang.String"
  #  - "new groovy.json.JsonSlurperClassic"
  # List of groovy init scripts to be executed during Jenkins master start
  initScripts: []
  #  - |
  #    print 'adding global pipeline libraries, register properties, bootstrap jobs...'
  # Kubernetes secret that contains a 'credentials.xml' for Jenkins
  # credentialsXmlSecret: jenkins-credentials
  # Kubernetes secret that contains files to be put in the Jenkins 'secrets' directory,
  # useful to manage encryption keys used for credentials.xml for instance (such as
  # master.key and hudson.util.Secret)
  # secretsFilesSecret: jenkins-secrets
  # Jenkins XML job configs to provision
  jobs: {}
  #  test: |-
  #    <<xml here>>

  # Below is the implementation of Jenkins Configuration as Code.  Add a key under configScripts for each configuration area,
  # where each corresponds to a plugin or section of the UI.  Each key (prior to | character) is just a label, and can be any value.
  # Keys are only used to give the section a meaningful name.  The only restriction is they may only contain RFC 1123 \ DNS label
  # characters: lowercase letters, numbers, and hyphens.  The keys become the name of a configuration yaml file on the master in
  # /var/jenkins_home/casc_configs (by default) and will be processed by the Configuration as Code Plugin.  The lines after each |
  # become the content of the configuration yaml file.  The first line after this is a JCasC root element, eg jenkins, credentials,
  # etc.  Best reference is https://<jenkins_url>/configuration-as-code/reference.  The example below creates a welcome message:
  JCasC:
    enabled: false
    defaultConfig: false
    pluginVersion: "1.36"
    configScripts: {}
    # welcome-message: |
    #   jenkins:
    #     systemMessage: Welcome to our CI\CD server.  This Jenkins is configured and managed 'as code'.
  # Optionally specify additional init-containers
  customInitContainers: []
  # - name: custom-init
  #   image: "alpine:3.7"
  #   imagePullPolicy: Always
  #   command: [ "uname", "-a" ]

  sidecars:
    configAutoReload:
      # If enabled: true, Jenkins Configuration as Code will be reloaded on-the-fly without a reboot.  If false or not-specified,
      # jcasc changes will cause a reboot and will only be applied at the subsequent start-up.  Auto-reload uses the
      # http://<jenkins_url>/reload-configuration-as-code endpoint to reapply config when changes to the configScripts are detected.
      enabled: false
      image: kiwigrid/k8s-sidecar:0.1.20
      imagePullPolicy: IfNotPresent
      resources: {}
        #   limits:
        #     cpu: 100m
        #     memory: 100Mi
        #   requests:
        #     cpu: 50m
        #     memory: 50Mi
      # folder in the pod that should hold the collected dashboards:
      folder: "/var/jenkins_home/casc_configs"
      # If specified, the sidecar will search for JCasC config-maps inside this namespace.
      # Otherwise the namespace in which the sidecar is running will be used.
      # It's also possible to specify ALL to search in all namespaces:
      # searchNamespace:

    # Allows you to inject additional/other sidecars
    other: []
    ## The example below runs the client for https://smee.io as sidecar container next to Jenkins,
    ## that allows to trigger build behind a secure firewall.
    ## https://jenkins.io/blog/2019/01/07/webhook-firewalls/#triggering-builds-with-webhooks-behind-a-secure-firewall
    ##
    ## Note: To use it you should go to https://smee.io/new and update the url to the generete one.
    # - name: smee
    #   image: docker.io/twalter/smee-client:1.0.2
    #   args: ["--port", "{{ .Values.master.servicePort }}", "--path", "/github-webhook/", "--url", "https://smee.io/new"]
    #   resources:
    #     limits:
    #       cpu: 50m
    #       memory: 128Mi
    #     requests:
    #       cpu: 10m
    #       memory: 32Mi
  # Node labels and tolerations for pod assignment
  # ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector
  # ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#taints-and-tolerations-beta-feature
  nodeSelector:
    node_pool: "ept-gke-node-pool1"

  terminationGracePeriodSeconds:

  tolerations: []
  # Leverage a priorityClass to ensure your pods survive resource shortages
  # ref: https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/
  # priorityClass: system-cluster-critical
  podAnnotations: {}
  # Add deployment annotation
  deploymentAnnotations: {}

  # The below two configuration-related values are deprecated and replaced by Jenkins Configuration as Code (see above
  # JCasC key).  They will be deleted in an upcoming version.
  customConfigMap: false
  # By default, the configMap is only used to set the initial config the first time
  # that the chart is installed.  Setting `overwriteConfig` to `true` will overwrite
  # the jenkins config with the contents of the configMap every time the pod starts.
  # This will also overwrite all init scripts
  overwriteConfig: false

  # By default, the Jobs Map is only used to set the initial jobs the first time
  # that the chart is installed.  Setting `overwriteJobs` to `true` will overwrite
  # the jenkins jobs configuration with the contents of Jobs every time the pod starts.
  overwriteJobs: false

# Expose Prometheus metrics
  prometheus:
    # If enabled, add the prometheus plugin to the list of plugins to install
    # https://plugins.jenkins.io/prometheus
    enabled: false
    # Additional labels to add to the ServiceMonitor object
    serviceMonitorAdditionalLabels:
    # Set a custom namespace where to deploy ServiceMonitor resource
    serviceMonitorNamespace: jenkins
    scrapeInterval: 120s
    # This is the default endpoint used by the prometheus plugin
    scrapeEndpoint: /prometheus
    # Additional labels to add to the PrometheusRule object
    alertingRulesAdditionalLabels: {}
    # An array of prometheus alerting rules
    # See here: https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/
    # The `groups` root object is added by default, simply add the rule entries
    alertingrules: []

  # Can be used to disable rendering master test resources when using helm template
  testEnabled: true

agent:
  enabled: true
  image: "jenkins/jnlp-slave"
  tag: "4.3-4"
  customJenkinsLabels: []
  # name of the secret to be used for image pulling
  persistence:
    size: 100Gi
  imagePullSecretName:
  componentName: "jenkins-slave"
  privileged: false
  resources:
    requests:
      cpu: "512m"
      memory: "512Mi"
    limits:
      cpu: "512m"
      memory: "512Mi"
  # You may want to change this to true while testing a new image
  alwaysPullImage: false
  # Controls how agent pods are retained after the Jenkins build completes
  # Possible values: Always, Never, OnFailure
  podRetention: "Never"
  # You can define the volumes that you want to mount for this container
  # Allowed types are: ConfigMap, EmptyDir, HostPath, Nfs, Pod, Secret
  # Configure the attributes as they appear in the corresponding Java class for that type
  # https://github.com/jenkinsci/kubernetes-plugin/tree/master/src/main/java/org/csanchez/jenkins/plugins/kubernetes/volumes
  # Pod-wide ennvironment, these vars are visible to any container in the agent pod
  envVars: []
  # - name: PATH
  #   value: /usr/local/bin
  volumes: []
  # - type: Secret
  #   secretName: mysecret
  #   mountPath: /var/myapp/mysecret
  # - type: EmptyDir
  #   mountPath: "/var/lib/containers"
  #   memory: false
  nodeSelector: {}
  # Key Value selectors. Ex:
  # jenkins-agent: v1

  # Executed command when side container gets started
  command:
  args:
  # Side container name
  sideContainerName: "jnlp"
  # Doesn't allocate pseudo TTY by default
  TTYEnabled: false
  # Max number of spawned agent
  containerCap: 10
  # Pod name
  podName: "default"
  # Allows the Pod to remain active for reuse until the configured number of
  # minutes has passed since the last step was executed on it.
  idleMinutes: 0
  # Raw yaml template for the Pod. For example this allows usage of toleration for agent pods.
  # https://github.com/jenkinsci/kubernetes-plugin#using-yaml-to-define-pod-templates
  # https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/
  yamlTemplate: ""
  # yamlTemplate: |-
  #   apiVersion: v1
  #   kind: Pod
  #   spec:
  #     tolerations:
  #     - key: "key"
  #       operator: "Equal"
  #       value: "value"
  # Timeout in seconds for an agent to be online
  slaveConnectTimeout: 100

persistence:
  enabled: true
  ## A manually managed Persistent Volume and Claim
  ## Requires persistence.enabled: true
  ## If defined, PVC must be created manually before volume will be bound
  existingClaim: jenkins-data-claim
  subPath: "jenkins-2.232"
  ## jenkins data Persistent Volume Storage Class
  ## If defined, storageClassName: <storageClass>
  ## If set to "-", storageClassName: "", which disables dynamic provisioning
  ## If undefined (the default) or set to null, no storageClassName spec is
  ##   set, choosing the default provisioner.  (gp2 on AWS, standard on
  ##   GKE, AWS & OpenStack)
  ##
  storageClass:
  annotations: {}
  accessMode: "ReadWriteOnce"
  size: "200Gi"
  volumes:
  #  - name: nothing
  #    emptyDir: {}
  mounts:
  #  - mountPath: /var/nothing
  #    name: nothing
  #    readOnly: true

networkPolicy:
  # Enable creation of NetworkPolicy resources.
  enabled: true
  # For Kubernetes v1.4, v1.5 and v1.6, use 'extensions/v1beta1'
  # For Kubernetes v1.7, use 'networking.k8s.io/v1'
  apiVersion: networking.k8s.io/v1

## Install Default RBAC roles and bindings
rbac:
  create: true
  readSecrets: false

serviceAccount:
  create: true
  # The name of the service account is autogenerated by default
  name: jenkins-rbac
  annotations: {}

serviceAccountAgent:
  # Specifies whether a ServiceAccount should be created
  create: false
  # The name of the ServiceAccount to use.
  # If not set and create is true, a name is generated using the fullname template
  name:
  annotations: {}

## Backup cronjob configuration
## Ref: https://github.com/maorfr/kube-tasks
backup:
  # Backup must use RBAC
  # So by enabling backup you are enabling RBAC specific for backup
  enabled: false
  # Used for label app.kubernetes.io/component
  componentName: "backup"
  # Schedule to run jobs. Must be in cron time format
  # Ref: https://crontab.guru/
  schedule: "0 2 * * *"
  labels: {}
  annotations: {}
    # Example for authorization to AWS S3 using kube2iam
    # Can also be done using environment variables
    # iam.amazonaws.com/role: "jenkins"
  image:
    repository: "maorfr/kube-tasks"
    tag: "0.2.0"
  # Additional arguments for kube-tasks
  # Ref: https://github.com/maorfr/kube-tasks#simple-backup
  extraArgs: []
  # Add existingSecret for AWS credentials
  existingSecret: {}
  ## Example for using an existing secret
   # jenkinsaws:
  ## Use this key for AWS access key ID
     # awsaccesskey: jenkins_aws_access_key
  ## Use this key for AWS secret access key
     # awssecretkey: jenkins_aws_secret_key
  # Add additional environment variables
   # jenkinsgcp:
  ## Use this key for GCP credentials
     # gcpcredentials: credentials.json
  env:
    # Example environment variable required for AWS credentials chain
    - name: "AWS_REGION"
      value: "us-east-1"
  resources:
    requests:
      memory: 1Gi
      cpu: 1
    limits:
      memory: 1Gi
      cpu: 1
  # Destination to store the backup artifacts
  # Supported cloud storage services: AWS S3, Minio S3, Azure Blob Storage
  # Additional support can added. Visit this repository for details
  # Ref: https://github.com/nuvo/skbn
  destination: "s3://nuvo-jenkins-data/backup"
checkDeprecation: true

[paranoidsp]

@nukarajusundeep If you've not checked already, does this happen when your nodes autoscale? Or is the jenkins master evicted when GKE decides to move your pod somewhere else? You can get around this by setting a PodDisruptionBudget.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[stale[bot]]

This issue is being automatically closed due to inactivity.