[stable/jenkins] Error: UPGRADE FAILED: Deployment.apps "jenkins" is invalid

[marcingluc-clearobject]

Request for help:

Version of Helm and Kubernetes: Kubernetes: 1.12.7-gke.10

Helm: Client: &version.Version{SemVer:"v2.12.0", GitCommit:"d325d2a9c179b33af1a024cdb5a4472b6288016a", GitTreeState:"clean"} Server: &version.Version{SemVer:"v2.12.0", GitCommit:"d325d2a9c179b33af1a024cdb5a4472b6288016a", GitTreeState:"clean"}

Which chart: stable/jenkins 1.0.0

What happened: I am trying to update from chart version 0.0.33 to 1.0.0. I have updated my values.yaml to reflect the new updates:

master:
  installPlugins:
    - kubernetes:1.14.5
    - workflow-job:2.31
    - workflow-aggregator:2.6
    - credentials-binding:1.17
    - git:3.9.3
    - github-branch-source:2.4.2
    - basic-branch-build-strategies:1.1.1
    - google-oauth-plugin:0.7
    - google-source-plugin:0.3
    - google-login:1.4
    - matrix-auth:2.3
    - slack:2.18
    - github:1.29.4
    - cobertura:1.13
    - timestamper:1.9
    - thinBackup:1.9
    - role-strategy:2.10
  javaOpts: "-Xms3500m -Xmx3500m"
  serviceType: ClusterIP
  podLabels:
    hostName: "redacted.com"
  ingress:
    enabled: true
    annotations:
      kubernetes.io/ingress.class: "nginx"
      kubernetes.io/ingress.allow-http: "false"
      kubernetes.io/tls-acme: "true"
      certmanager.k8s.io/cluster-issuer: letsencrypt
      #certmanager.k8s.io/cluster-issuer: letsencrypt-staging
      hostName: "redacted.com"
      #kubernetes.io/ingress.global-static-ip-name: static-ip-gcp can be used to assign static ip
    tls:
      - secretName: redacted
      #- secretName: redacted
        hosts:
            - "redacted.com"

persistence:
  size: 100Gi
networkPolicy:
  ApiVersion: networking.k8s.io/v1
#rbac:
#  create: true
#serviceAccount:
#        create: true
#        name: "jenkins"

When issuing the update command: ./helm upgrade jenkins stable/jenkins --version 1.0.0 --values values.yaml

I receive the following error: Error: UPGRADE FAILED: Deployment.apps "jenkins" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/instance":"jenkins", "app.kubernetes.io/component":"jenkins-master"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable

List of current deployments:

$ ./helm list -a
NAME            REVISION        UPDATED                         STATUS          CHART                   APP VERSION     NAMESPACE
cert-manager    2               Thu Apr 18 10:57:57 2019        DEPLOYED        cert-manager-v0.5.1     v0.5.1          kube-system
jenkins         50              Thu Apr 25 12:12:21 2019        DEPLOYED        jenkins-0.33.0          lts             default
nginx-ingress   1               Mon Feb 25 11:07:19 2019        DEPLOYED        nginx-ingress-1.3.1     0.22.0          default

What you expected to happen: Successful upgrade

How to reproduce it (as minimally and precisely as possible): ./helm upgrade jenkins stable/jenkins --version 1.0.0 --values values.yaml *using listed values.yaml

Anything else we need to know: I ran into a strange issue when issuing an upgrade where Helm reported: Error: UPGRADE FAILED: no x with the name "x" found

This was resolved by issuing:

  kubectl delete ServiceAccount jenkins
  kubectl delete RoleBinding jenkins-schedule-agents
  kubectl delete Role jenkins-schedule-agents
  kubectl delete Role jenkins-casc-reload
  kubectl delete RoleBinding jenkins-watch-configmaps

[torstenwalter]

Does this help? https://github.com/helm/charts/pull/13273

[marcingluc-clearobject]

Thanks - I'll give it a shot. I see that it requires the deletion of the deployment.

kubectl delete deploy jenkins

[torstenwalter]

Did that help? Do you have suggestions for the upgrade instructions?

[marcingluc-clearobject]

Thanks - Deleting the deployment and then running the upgrade resolved the issue.

 kubectl delete deploy jenkins
./helm upgrade jenkins stable/jenkins --version 1.1.5 --values values.yaml 

I ran into a problem with host not being set properly within my jenkins ingress configuration. The following gets deployed:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    certmanager.k8s.io/cluster-issuer: letsencrypt
    hostName: redacted.com
    kubernetes.io/ingress.allow-http: "false"
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: "true"
  creationTimestamp: 2019-04-18T16:13:54Z
  generation: 8
  labels:
    app.kubernetes.io/component: jenkins-master
    app.kubernetes.io/instance: jenkins
    app.kubernetes.io/managed-by: Tiller
    app.kubernetes.io/name: jenkins
    helm.sh/chart: jenkins-1.1.5
  name: jenkins
  namespace: default
  resourceVersion: "15338028"
  selfLink: /apis/extensions/v1beta1/namespaces/default/ingresses/jenkins
  uid: f637f8cf-61f4-11e9-bf9e-42010a80000c
spec:
  **rules:
  - http:**
      paths:
      - backend:
          serviceName: jenkins
          servicePort: 8080
  tls:
  - hosts:
    - redacted.com
    secretName: redacted
status:
  loadBalancer:
    ingress:
    - ip: redacted

The fix:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    certmanager.k8s.io/cluster-issuer: letsencrypt
    hostName: redacted.com
    kubernetes.io/ingress.allow-http: "false"
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: "true"
  creationTimestamp: 2019-04-18T16:13:54Z
  generation: 8
  labels:
    app.kubernetes.io/component: jenkins-master
    app.kubernetes.io/instance: jenkins
    app.kubernetes.io/managed-by: Tiller
    app.kubernetes.io/name: jenkins
    helm.sh/chart: jenkins-1.1.5
  name: jenkins
  namespace: default
  resourceVersion: "15338028"
  selfLink: /apis/extensions/v1beta1/namespaces/default/ingresses/jenkins
  uid: f637f8cf-61f4-11e9-bf9e-42010a80000c
spec:
  rules:
  **- host: redacted.com
    http:**
      paths:
      - backend:
          serviceName: jenkins
          servicePort: 8080
  tls:
  - hosts:
    - redacted.com
    secretName: redacted
status:
  loadBalancer:
    ingress:
    - ip: redacted

Manually modifying the ingress with -host: redacted.com under spec > rules allows the cert to properly load

The previous version of the chart 0.0.33 creates the following ingress:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    certmanager.k8s.io/cluster-issuer: letsencrypt
    kubernetes.io/ingress.allow-http: "false"
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: "true"
  creationTimestamp: 2019-02-25T15:07:48Z
  generation: 1
  name: jenkins
  namespace: default
  resourceVersion: "1007"
  selfLink: /apis/extensions/v1beta1/namespaces/default/ingresses/jenkins
  uid: 1d228f24-390f-11e9-bf2a-42010a800074
spec:
  rules:
  **- host: redacted.com
    http:**
      paths:
      - backend:
          serviceName: jenkins
          servicePort: 8080
  tls:
  - hosts:
    - redacted.com
    secretName: redacted
status:
  loadBalancer:
    ingress:
    - ip: redacted

The values.yaml I am using is posted above. I am using certmanager and nginx ingress helm charts.

[torstenwalter]

Thanks for the update. Do you want to create a PR to fix the chart?

[torstenwalter]

Looks like there are two parameters where the hostname can be configured:

master.hostName master.ingress.hostName => has been introduces as part of #12946

If I look at the template only master.ingress.hostName is used now. NOTES.txt still references the wrong value. I just created #13318 to correct the notes.

[marcingluc-clearobject]

Thank you for the clarification. I'll correct my values and this should fix it.

[marcingluc-clearobject]

I have tried setting my hostname redacted.com on both master.hostName or master.ingress.hostName. Neither populates the rules field within the ingress rules section. My workaround is to manually modify the ingress so our systems are operational. Just bringing this back up to the community in case anyone else is having a problem.

[torstenwalter]

@marcingluc-clearobject I tried to reproduce your issue using the following steps:

git clone https://github.com/helm/charts.git
cd charts/stable/jenkins
helm template . -x templates/jenkins-master-ingress.yaml --set master.ingress.enabled=true --set master.ingress.hostName=redacted.com
---
# Source: jenkins/templates/jenkins-master-ingress.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  labels:
    "app.kubernetes.io/name": 'jenkins'
    "helm.sh/chart": "jenkins-1.1.10"
    "app.kubernetes.io/managed-by": "Tiller"
    "app.kubernetes.io/instance": "release-name"
    "app.kubernetes.io/component": "jenkins-master"
  name: release-name-jenkins
spec:
  rules:
  - http:
      paths:
      - backend:
          serviceName: release-name-jenkins
          servicePort: 8080
    host: "redacted.com"

It contains host: "redacted.com", so what's wrong with the generated ingress?

[marcingluc-clearobject]

Looks like it was an indentation error in my values.yaml. Closing issue.