v0.14.0-upgrade job fails

AceHack commented 4 years ago

Describe the bug When installing the knative eventing operator the v0.14.0-upgrade jobs fails with the following error. It appears the service account it uses: eventing-controller does have permission to namespaces so I'm not sure what the problem is.

{"level":"warn","ts":1587242704.338062,"logger":"fallback","caller":"v0.14.0/upgrader.go:42","msg":"Failed to list namespaces: Get https://172.20.0.1:443/api/v1/namespaces: dial tcp 172.20.0.1:443: connect: connection refu
Broker Upgrade failed with: Get https://172.20.0.1:443/api/v1/namespaces: dial tcp 172.20.0.1:443: connect: connection refused

Expected behavior Job to complete successfully

To Reproduce Install eventing operator

Knative release version 0.14.0

AceHack commented 4 years ago

Also, this job should likely disable istio with annotations or else it will never complete. See this related issue https://github.com/istio/istio/issues/6324

AceHack commented 4 years ago

Istio is what breaks it, it seems it's not waiting for envoy to complete come up and there are no retries so it fails and never recovers. Should add the sidecar.istio.io/inject: "false" annotation to the pod as I did here. I had to create a 2nd job, I can't actually patch the first one so it stays failed forever. This is very annoying, it would be great to get this fix into the operator.

apiVersion: batch/v1
kind: Job
metadata:
  labels:
    eventing.knative.dev/release: v0.14.0
  name: v0.14.0-upgrade-fix
  namespace: knative-eventing
spec:
  backoffLimit: 6
  completions: 1
  parallelism: 1
  template:
    metadata:
      annotations:
        sidecar.istio.io/inject: "false"
    spec:
      containers:
      - image: gcr.io/knative-releases/knative.dev/eventing/cmd/upgrade/v0.14.0@sha256:3f796b810f763325678730849021bf53ce7b6016848f880f27298e4db9112d0f
        imagePullPolicy: IfNotPresent
        name: upgrade-brokers
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Never
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: eventing-controller
      serviceAccountName: eventing-controller
      terminationGracePeriodSeconds: 30

AceHack commented 4 years ago

An alternative to disabling istio would be to use something like envoy pre-flight.

https://github.com/monzo/envoy-preflight

AceHack commented 4 years ago

Fixed as of 0.14.1

knative / eventing-operator

v0.14.0-upgrade job fails #194