GCR as Private Registry when Not On GKE

[grifonas]

Hi, I've been having trouble configuring image polling with GCR. We are using GCR as our Docker registry however the cluster doesn't live on Google Cloud.

Pub Sub I tried pub/sub, but that appears to require instance metadata, which, obviously, my cluster nodes don't have.

Polling

Configuration:

• Keel service account has a reference to the secret containing the GCR credentials json key file:

apiVersion: v1
imagePullSecrets:
- name: gcr-json-key
kind: ServiceAccount
metadata:
  labels:
    app: keel
    chart: keel-0.8.16
    heritage: Tiller
    release: keel
  name: keel

• Note that similar configuration works with the other deployments and they are able to pull the images using the exact same secret.

Keel logs:

time="2019-12-09T13:14:37Z" level=warning msg="secrets.defaultGetter: registry not found among secrets" image="keelhq/keel:0.15.0-rc1" secret_ref=gcr-json-key
time="2019-12-09T13:14:37Z" level=warning msg="secrets.defaultGetter.lookupSecrets: secret found but couldn't detect authentication for the desired registry" image=index.docker.io/keelhq/keel namespace=kube-system provider=helm registry=index.docker.io secrets="[gcr-json-key]"
time="2019-12-09T13:14:38Z" level=info msg="trigger.poll.RepositoryWatcher: new watch repository tags job added" digest="sha256:28a9226ca76b99d845fab6ea43b46a1cf1695ccbebd6d1693c2a871c4624334d" image="keelhq/keel:0.15.0-rc1" job_name=index.docker.io/keelhq/keel schedule="@every 3m"
time="2019-12-09T13:14:38Z" level=warning msg="secrets.defaultGetter: registry not found among secrets" image="keelhq/keel:0.15.0-rc1" secret_ref=gcr-json-key
time="2019-12-09T13:14:38Z" level=warning msg="secrets.defaultGetter.lookupSecrets: secret found but couldn't detect authentication for the desired registry" image=index.docker.io/keelhq/keel namespace=kube-system provider=helm registry=index.docker.io secrets="[gcr-json-key]"

☝️ This seems to indicate that Keel is able to detect and use the secret.

• Relevant logs for my deployment:

  time="2019-12-09T13:16:51Z" level=error msg="trigger.poll.RepositoryWatcher.addJob: failed to get image digest" error="Get https://eu.gcr.io/v2/MY_GOOGLE_PROJECT/MY_IMAGE_NAME/manifests/v5.1.1: http: non-successful response (status=401 body=\"{\\\"errors\\\":[{\\\"code\\\":\\\"UNAUTHORIZED\\\",\\\"message\\\":\\\"You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication\\\"}]}\")" image="MY_GOOGLE_PROJECT/MY_IMAGE_NAME:v5.1.1" password= username=
  time="2019-12-09T13:16:51Z" level=error msg="trigger.poll.RepositoryWatcher.Watch: failed to add image watch job" error="Get https://eu.gcr.io/v2/MY_GOOGLE_PROJECT/MY_IMAGE_NAME/manifests/v5.1.1: http: non-successful response (status=401 body=\"{\\\"errors\\\":[{\\\"code\\\":\\\"UNAUTHORIZED\\\",\\\"message\\\":\\\"You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication\\\"}]}\")" image="namespace:default,image:eu.gcr.io/MY_GOOGLE_PROJECT/MY_IMAGE_NAME:v5.1.1,provider:kubernetes,trigger:poll,sched:@every 1m,secrets:[]"
  time="2019-12-09T13:16:51Z" level=error msg="trigger.poll.manager: got error(-s) while watching images" error="encountered errors while adding images: Get https://eu.gcr.io/v2/MY_GOOGLE_PROJECT/MY_IMAGE_NAME/manifests/v5.1.1: http: non-successful response (status=401 body=\"{\\\"errors\\\":[{\\\"code\\\":\\\"UNAUTHORIZED\\\",\\\"message\\\":\\\"You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication\\\"}]}\")"

Is there a way to use polling with GCR?

Thank you!

[rchenzheng]

Did you ensure the service account has storage viewer role to the registry?

[kenneyfortressiq]

@rchenzheng I have pub/sub editor role and it has a parsing bug with index.docker.io. All my images are in GCR including keel. It is annoying and leads to me thinking it is failing to update..

steps to reproduce: 1) 0.16.0 chart 2) have value.yaml below that is working on GKE helm chart 3) change cluster name and move to Azure AKS 4) run: helm upgrade --install keel . --namespace kube-system -f values.yaml 5) see issue

error: time="2020-04-14T23:15:07Z" level=debug msg="performing scan" time="2020-04-14T23:15:07Z" level=debug msg="registry index.docker.io is not a GCR, skipping"

values.yaml

polling:
  enabled: false

gcr:
  enabled: true
  projectId: "xxx"
  clusterName: "tr-us-000"
keel:
  # keel policy (all/major/minor/patch/force)
  policy: all
  # trigger type, defaults to events such as pubsub, webhooks
  trigger: poll
  # polling schedule
  pollSchedule: "@every 3m"
  # images to track and update
  images:
    - repository: gcr.io/..
      tag: latest