Upgrading deis from 2.5.0 to 2.6.0 fails on Kubernetes 1.4

[felipejfc]

Just followed the instructions from here

This is the log:

$ helmc install workflow-{$DESIRED_WORKFLOW_RELEASE}
---> Running `kubectl create -f` ...
namespace "deis" configured

secret "builder-ssh-private-keys" created

secret "builder-key-auth" created

secret "django-secret-key" created

secret "database-creds" created

secret "logger-redis-creds" configured

secret "objectstorage-keyfile" created

secret "registry-secret" created

secret "deis-router-dhparam" created

serviceaccount "deis-builder" created

serviceaccount "deis-controller" created

serviceaccount "deis-database" created

serviceaccount "deis-logger-fluentd" created

serviceaccount "deis-logger" created

serviceaccount "deis-monitor-telegraf" created

serviceaccount "deis-nsqd" created

serviceaccount "deis-router" created

serviceaccount "deis-workflow-manager" created

service "deis-builder" created

service "deis-controller" created

service "deis-database" created

service "deis-logger-redis" configured

service "deis-logger" created

service "deis-monitor-grafana" created

service "deis-monitor-influxapi" created

service "deis-monitor-influxui" created

service "deis-nsqd" created

proto: no encoder for TypeMeta unversioned.TypeMeta [GetProperties]
proto: tag has too few fields: "-"
proto: no coders for struct *reflect.rtype
proto: no encoder for sec int64 [GetProperties]
proto: no encoder for nsec int32 [GetProperties]
proto: no encoder for loc *time.Location [GetProperties]
proto: no encoder for Time time.Time [GetProperties]
proto: no coders for intstr.Type
proto: no encoder for Type intstr.Type [GetProperties]
The Service "deis-router" is invalid.

* metadata.resourceVersion: Invalid value: "": must be specified for an update
* spec.clusterIP: Invalid value: "": field is immutable

[ERROR] Failed to upload manifests: exit status 1

[bacongobbler]

pinging @tmc as I believe he has seen this specific issue before and knows a workaround :)

[bacongobbler]

@felipejfc have you resolved this issue yourself? If so how did you resolve it? IIRC it has something to do with when you modify the deis-router service's annotations... Or something of that nature.

[felipejfc]

@bacongobbler I've created and replaced the deployments manually and didn't touch the router service as it had no upgrades

[bacongobbler]

Can you please show your service manifest from the old release? That way we can try to identify which field cannot be updated and hopefully we can come up with a fix.

[felipejfc]

@bacongobbler Here it is:

$ kubectl edit svc deis-router --namespace deis

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
kind: Service
metadata:
  annotations:
    chart.helm.sh/description: Deis Workflow
    chart.helm.sh/file: /home/felipe/.helmc/workspace/charts/workflow-v2.5.0/manifests/deis-router-service.yaml
    chart.helm.sh/name: workflow-v2.5.0
    chart.helm.sh/version: v2.5.0
    helm-keep: "true"
    kubectl.kubernetes.io/last-applied-configuration: '{"kind":"Service","apiVersion":"v1","metadata":{"name":"deis-router","namespace":"deis","selfLink":"/api/v1/namespaces/deis/services/deis-router","uid":"3cbf8d61-5b52-11e6-8414-0a656e9cbc8d","resourceVersion":"6906941","creationTimestamp":"2016-08-05T21:19:04Z","labels":{"heritage":"deis"},"annotations":{"chart.helm.sh/description":"Deis
      Workflow","chart.helm.sh/file":"/home/felipe/.helmc/workspace/charts/workflow-v2.5.0/manifests/deis-router-service.yaml","chart.helm.sh/name":"workflow-v2.5.0","chart.helm.sh/version":"v2.5.0","helm-keep":"true"}},"spec":{"ports":[{"name":"http","protocol":"TCP","port":80,"targetPort":8080,"nodePort":30372},{"name":"https","protocol":"TCP","port":443,"targetPort":6443,"nodePort":32228},{"name":"builder","protocol":"TCP","port":2222,"targetPort":2222,"nodePort":30928},{"name":"healthz","protocol":"TCP","port":9090,"targetPort":9090,"nodePort":31437}],"selector":{"app":"deis-router"},"clusterIP":"100.71.77.67","type":"LoadBalancer","sessionAffinity":"None"},"status":{"loadBalancer":{"ingress":[{"hostname":"xxxxxxxxxxxxxxxxxxxxx.us-east-1.elb.amazonaws.com"}]}}}'
  creationTimestamp: 2016-08-05T21:19:04Z
  labels:
    heritage: deis
  name: deis-router
  namespace: deis
  resourceVersion: "6909803"
  selfLink: /api/v1/namespaces/deis/services/deis-router
  uid: 3cbf8d61-5b52-11e6-8414-0a656e9cbc8d
spec:
  clusterIP: 100.71.77.67
  ports:
  - name: http
    nodePort: 30372
    port: 80
    protocol: TCP
    targetPort: 8080
  - name: https
    nodePort: 32228
    port: 443
    protocol: TCP
    targetPort: 6443
  - name: builder
    nodePort: 30928
    port: 2222
    protocol: TCP
    targetPort: 2222
  - name: healthz
    nodePort: 31437
    port: 9090
    protocol: TCP
    targetPort: 9090
  selector:
    app: deis-router
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer:
    ingress:
    - hostname: xxxxxxxxxxxxxxxxxx.us-east-1.elb.amazonaws.com

[bnerd]

I ran into the same issue while upgrading from deis v2.6 to v2.7 on k8s v1.2.6. I deleted the deis-router service

kubectl delete svc deis-router --namespace deis

and succesfully ran the installation again. Afterwards I edited the deis-router service and set the nodePorts back to the values matching my current LB setup.

[sstarcher]

I'm seeing health check failures on some of the deis router pods in 1 my environments. These are now running 2.6.4 the issue started appearing for me some where after 2.6.0 I believe. This is on K8S 1.4.0

deis-router-1301926735-2cnfe   1/1       Running   14         2d
deis-router-1301926735-g2d5y   1/1       Running   1          2d
deis-router-1301926735-goieo   1/1       Running   2          2d
deis-router-1301926735-xexd1   1/1       Running   56         2d

[bacongobbler]

@sstarcher can you please post a new issue? That doesn't seem related to the OP's issue.

[sstarcher]

@bacongobbler I upgraded to 2.6.5 and the problem seems to have ceased. I'm not sure what the root cause was for the issue.

[bacongobbler]

Perhaps https://github.com/deis/router/commit/646b1c69faf9b8493687af2da53fc723a92be8e8?

EDIT: sorry, https://github.com/deis/charts/pull/368

[sstarcher]

@bacongobbler My issue seemed to be with the healthz failing after a while. The initial delay would not affect a currently running container would it?

[bacongobbler]

Not a long-running container, no.

[bacongobbler]

Closing as fixed with helm v2.