Closed sharadg closed 5 years ago
My theory is that the most probable root cause is a missing "storage.admin" role binding on the service account.
Unfortunately I was unable to repro that this would cause the failure above on our team GCP project - even when i initialize the namespace using a secret for a serviceaccount with no bindings at all.
@trisberg perhaps you could confirm that a missing role binding would indeed cause the failure.
Do you have the Google Container Registry enabled?
Error says
Please enable Google Container Registry API in Cloud Console at https://console.cloud.google.com/apis/api/containerregistry.googleapis.com/overview?project=fe-shgpupta before performing this operation.
You could try something like the following:
enable GCR:
gcloud services enable containerregistry.googleapis.com
create service account with storage.admin role:
gcloud iam service-accounts create push-image
export GCP_PROJECT=$(gcloud config get-value core/project)
gcloud projects add-iam-policy-binding $GCP_PROJECT \
--member serviceAccount:push-image@$GCP_PROJECT.iam.gserviceaccount.com \
--role roles/storage.admin
create JSON key for the service account:
export GCP_PROJECT=$(gcloud config get-value core/project)
gcloud iam service-accounts keys create \
--iam-account "push-image@$GCP_PROJECT.iam.gserviceaccount.com" \
$HOME/push-image.json
initialize the default namespace with the push credentials you created:
riff namespace init default --gcr $HOME/push-image.json
So, I made little progress (by the way of cloud services enable step)… but it fails at a different error this time:
riff function create node square --git-repo https://github.com/projectriff-samples/node-square --artifact package.json --image gcr.io/$GCP_PROJECT/square --wait -v
Waiting for LatestCreatedRevisionName:. square-00001
default/square-00001-ngljj[build-step-riff-init]: Initializing with node invoker
default/square-00001-ngljj[build-step-riff-init]: Initializing /workspace/Dockerfile
default/square-00001-ngljj[build-step-riff-init]: Initializing /workspace/.dockerignore
default/square-00001-ngljj[build-step-build-and-push]: INFO[0000] Downloading base image projectriff/node-function-invoker:0.0.8
default/square-00001-ngljj[build-step-build-and-push]: ERROR: logging before flag.Parse: E1015 14:23:53.696991 1 metadata.go:142] while reading 'google-dockercfg' metadata: http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg
default/square-00001-ngljj[build-step-build-and-push]: ERROR: logging before flag.Parse: E1015 14:23:53.699961 1 metadata.go:159] while reading 'google-dockercfg-url' metadata: http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg-url
default/square-00001-ngljj[build-step-build-and-push]: 2018/10/15 14:23:53 No matching credentials were found, falling back on anonymous
default/square-00001-ngljj[build-step-build-and-push]: INFO[0000] Executing 0 build triggers
default/square-00001-ngljj[build-step-build-and-push]: INFO[0000] Extracting layer 0
default/square-00001-ngljj[build-step-build-and-push]: INFO[0001] Extracting layer 1
default/square-00001-ngljj[build-step-build-and-push]: INFO[0003] Extracting layer 2
default/square-00001-ngljj[build-step-build-and-push]: INFO[0004] Extracting layer 3
default/square-00001-ngljj[build-step-build-and-push]: INFO[0004] Extracting layer 4
default/square-00001-ngljj[build-step-build-and-push]: INFO[0004] Extracting layer 5
default/square-00001-ngljj[build-step-build-and-push]: INFO[0006] Taking snapshot of full filesystem...
default/square-00001-ngljj[build-step-build-and-push]: INFO[0011] ENV FUNCTION_URI /functions/
default/square-00001-ngljj[build-step-build-and-push]: INFO[0011] Using files from context: [/workspace]
default/square-00001-ngljj[build-step-build-and-push]: INFO[0011] COPY . ${FUNCTION_URI}
default/square-00001-ngljj[build-step-build-and-push]: INFO[0011] Taking snapshot of files...
default/square-00001-ngljj[build-step-build-and-push]: INFO[0012] RUN (cd ${FUNCTION_URI} && npm install --production --unsafe-perm)
default/square-00001-ngljj[build-step-build-and-push]: INFO[0012] cmd: /bin/sh
default/square-00001-ngljj[build-step-build-and-push]: INFO[0012] args: [-c (cd ${FUNCTION_URI} && npm install --production --unsafe-perm)]
default/square-00001-ngljj[build-step-build-and-push]: npm notice created a lockfile as package-lock.json. You should commit this file.
default/square-00001-ngljj[build-step-build-and-push]: up to date in 0.1s
default/square-00001-ngljj[build-step-build-and-push]: INFO[0012] Taking snapshot of full filesystem...
default/square-00001-ngljj[build-step-build-and-push]: error pushing image: failed to push to destination gcr.io/fe-shgupta/square:latest: no token in bearer response:
default/square-00001-ngljj[build-step-build-and-push]: {"errors":[{"code":"DENIED","message":"Token exchange failed for project 'fe-shgupta'. Caller does not have permission 'storage.buckets.get'. To configure permissions, follow instructions at: https://cloud.google.com/container-registry/docs/access-control"}]}
Error: function creation failed: RevisionMissing: Configuration "square" does not have any ready Revision.; Revision "square-00001" failed with message: "build step \"build-step-build-and-push\" exited with code 1 (image: \"docker-pullable://gcr.io/kaniko-project/executor@sha256:a59007aa409ea7734107e31e7ee323a4a8bd85f863de59fdd7018e85ed3c2b5a\"); for logs run: kubectl -n default logs square-00001-ngljj -c build-step-build-and-push".
I do have "Storage Admin" on the service account
did you re-initialize the namespace with the new push-image.json
file created from the new service account?
I did. How can I confirm that all the pre-reqs are in place? Tried again:
riff function create node square --git-repo https://github.com/projectriff-samples/node-square --artifact package.json --image gcr.io/$GCP_PROJECT/square --wait -v
Waiting for LatestCreatedRevisionName:. square-00001
default/square-00001-s5zlp[build-step-credential-initializer]: {"level":"info","ts":1539615012.912515,"logger":"fallback-logger","caller":"creds-init/main.go:40","msg":"Credentials initialized."}
default/square-00001-s5zlp[build-step-git-source]: {"level":"info","ts":1539615017.0998971,"logger":"fallback-logger","caller":"git-init/main.go:74","msg":"Successfully cloned \"https://github.com/projectriff-samples/node-square\" @ \"master\""}
default/square-00001-s5zlp[build-step-riff-init]: Initializing with node invoker
default/square-00001-s5zlp[build-step-riff-init]: Initializing /workspace/Dockerfile
default/square-00001-s5zlp[build-step-riff-init]: Initializing /workspace/.dockerignore
default/square-00001-s5zlp[build-step-build-and-push]: INFO[0000] Downloading base image projectriff/node-function-invoker:0.0.8
default/square-00001-s5zlp[build-step-build-and-push]: ERROR: logging before flag.Parse: E1015 14:50:26.408557 1 metadata.go:142] while reading 'google-dockercfg' metadata: http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg
default/square-00001-s5zlp[build-step-build-and-push]: ERROR: logging before flag.Parse: E1015 14:50:26.411219 1 metadata.go:159] while reading 'google-dockercfg-url' metadata: http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg-url
default/square-00001-s5zlp[build-step-build-and-push]: 2018/10/15 14:50:26 No matching credentials were found, falling back on anonymous
default/square-00001-s5zlp[build-step-build-and-push]: INFO[0000] Executing 0 build triggers
default/square-00001-s5zlp[build-step-build-and-push]: INFO[0000] Extracting layer 0
default/square-00001-s5zlp[build-step-build-and-push]: INFO[0001] Extracting layer 1
default/square-00001-s5zlp[build-step-build-and-push]: INFO[0003] Extracting layer 2
default/square-00001-s5zlp[build-step-build-and-push]: INFO[0003] Extracting layer 3
default/square-00001-s5zlp[build-step-build-and-push]: INFO[0003] Extracting layer 4
default/square-00001-s5zlp[build-step-build-and-push]: INFO[0003] Extracting layer 5
default/square-00001-s5zlp[build-step-build-and-push]: INFO[0005] Taking snapshot of full filesystem...
default/square-00001-s5zlp[build-step-build-and-push]: INFO[0010] ENV FUNCTION_URI /functions/
default/square-00001-s5zlp[build-step-build-and-push]: INFO[0010] Using files from context: [/workspace]
default/square-00001-s5zlp[build-step-build-and-push]: INFO[0010] COPY . ${FUNCTION_URI}
default/square-00001-s5zlp[build-step-build-and-push]: INFO[0010] Taking snapshot of files...
default/square-00001-s5zlp[build-step-build-and-push]: INFO[0010] RUN (cd ${FUNCTION_URI} && npm install --production --unsafe-perm)
default/square-00001-s5zlp[build-step-build-and-push]: INFO[0010] cmd: /bin/sh
default/square-00001-s5zlp[build-step-build-and-push]: INFO[0010] args: [-c (cd ${FUNCTION_URI} && npm install --production --unsafe-perm)]
default/square-00001-s5zlp[build-step-build-and-push]: npm notice created a lockfile as package-lock.json. You should commit this file.
default/square-00001-s5zlp[build-step-build-and-push]: up to date in 0.104s
default/square-00001-s5zlp[build-step-build-and-push]: INFO[0011] Taking snapshot of full filesystem...
default/square-00001-s5zlp[build-step-build-and-push]: error pushing image: failed to push to destination gcr.io/fe-shgupta/square:latest: no token in bearer response:
default/square-00001-s5zlp[build-step-build-and-push]: {"errors":[{"code":"DENIED","message":"Token exchange failed for project 'fe-shgupta'. Caller does not have permission 'storage.buckets.get'. To configure permissions, follow instructions at: https://cloud.google.com/container-registry/docs/access-control"}]}
Error: function creation failed: RevisionMissing: Configuration "square" does not have any ready Revision.; Revision "square-00001" failed with message: "build step \"build-step-build-and-push\" exited with code 1 (image: \"docker-pullable://gcr.io/kaniko-project/executor@sha256:a59007aa409ea7734107e31e7ee323a4a8bd85f863de59fdd7018e85ed3c2b5a\"); for logs run: kubectl -n default logs square-00001-s5zlp -c build-step-build-and-push".
You could run something like:
kubectl get secret push-credentials -o "jsonpath={.data.password}" | base64 -D
and verify you have the right JSON credentials in the secret you are using
Confirmed that the JSON is exactly the same as was uploaded from $HOME/push-image.json... Questions:
default/square-00001-s5zlp[build-step-build-and-push]: 2018/10/15 14:50:26 No matching credentials were found, falling back on anonymous
default/square-00001-s5zlp[build-step-build-and-push]: error pushing image: failed to push to destination gcr.io/fe-shgupta/square:latest: no token in bearer response
this tells me that authentication hasn't happened with container registry to have a bearer token?Exact same setup, but on GKE works perfectly fine.
riff function create node square --git-repo https://github.com/projectriff-samples/node-square --artifact package.json --image gcr.io/$GCP_PROJECT/square --wait -v
Waiting for LatestCreatedRevisionName:. square-00001
default/square-00001-4gh45[build-step-build-and-push]: INFO[0000] Downloading base image projectriff/node-function-invoker:0.0.8
default/square-00001-4gh45[build-step-build-and-push]: ERROR: logging before flag.Parse: E1015 18:20:55.047820 1 metadata.go:142] while reading 'google-dockercfg' metadata: http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg
default/square-00001-4gh45[build-step-build-and-push]: ERROR: logging before flag.Parse: E1015 18:20:55.050255 1 metadata.go:159] while reading 'google-dockercfg-url' metadata: http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg-url
default/square-00001-4gh45[build-step-build-and-push]: 2018/10/15 18:20:55 No matching credentials were found, falling back on anonymous
default/square-00001-4gh45[build-step-build-and-push]: INFO[0000] Executing 0 build triggers
default/square-00001-4gh45[build-step-build-and-push]: INFO[0000] Extracting layer 0
default/square-00001-4gh45[build-step-build-and-push]: INFO[0001] Extracting layer 1
default/square-00001-4gh45[build-step-build-and-push]: INFO[0002] Extracting layer 2
default/square-00001-4gh45[build-step-build-and-push]: INFO[0003] Extracting layer 3
default/square-00001-4gh45[build-step-build-and-push]: INFO[0003] Extracting layer 4
default/square-00001-4gh45[build-step-build-and-push]: INFO[0003] Extracting layer 5
default/square-00001-4gh45[build-step-build-and-push]: INFO[0004] Taking snapshot of full filesystem...
default/square-00001-4gh45[build-step-build-and-push]: INFO[0007] ENV FUNCTION_URI /functions/
default/square-00001-4gh45[build-step-build-and-push]: INFO[0007] Using files from context: [/workspace]
default/square-00001-4gh45[build-step-build-and-push]: INFO[0007] COPY . ${FUNCTION_URI}
default/square-00001-4gh45[build-step-build-and-push]: INFO[0007] Taking snapshot of files...
default/square-00001-4gh45[build-step-build-and-push]: INFO[0007] RUN (cd ${FUNCTION_URI} && npm install --production --unsafe-perm)
default/square-00001-4gh45[build-step-build-and-push]: INFO[0007] cmd: /bin/sh
default/square-00001-4gh45[build-step-build-and-push]: INFO[0007] args: [-c (cd ${FUNCTION_URI} && npm install --production --unsafe-perm)]
default/square-00001-4gh45[build-step-build-and-push]: npm notice created a lockfile as package-lock.json. You should commit this file.
default/square-00001-4gh45[build-step-build-and-push]: up to date in 0.088s
default/square-00001-4gh45[build-step-build-and-push]: INFO[0008] Taking snapshot of full filesystem...
default/square-00001-4gh45[build-step-build-and-push]: 2018/10/15 18:21:05 existing blob: sha256:605ce1bd3f3164f2949a30501cc596f52a72de05da1306ab360055f0d7130c32
default/square-00001-4gh45[build-step-build-and-push]: 2018/10/15 18:21:07 pushed blob sha256:98cfc19c2da6c1afc0b7de59ed7d351fd4939561387057b5128a52fefbd27ffa
default/square-00001-4gh45[build-step-build-and-push]: 2018/10/15 18:21:07 pushed blob sha256:3f0ff71f6ba10ec6ba666d261c4794b81d2ce0ef92b7e01cf806a3ced5a1e8d6
default/square-00001-4gh45[build-step-build-and-push]: 2018/10/15 18:21:07 pushed blob sha256:b9120239cfdf2d1f88c148a98c9867a8f7d6261e770554d5de4e4382abea905d
default/square-00001-4gh45[build-step-build-and-push]: 2018/10/15 18:21:07 pushed blob sha256:793b21730bffa835e14b49234d5e0cf637d3af96b5d01bb1c2b8e1f04efc7163
default/square-00001-4gh45[build-step-build-and-push]: 2018/10/15 18:21:07 pushed blob sha256:343e34c41f87fd40fa77e38946458ec27e88aedd8607f8a60e1c92c0bb25e123
default/square-00001-4gh45[build-step-build-and-push]: 2018/10/15 18:21:07 pushed blob sha256:1467c8ee45df61e24a02ca3de448e45a287485f3bfd700b7b52462b70eabc101
default/square-00001-4gh45[build-step-build-and-push]: 2018/10/15 18:21:08 pushed blob sha256:e727997e671263b3e33f36f22e96ed5ce246a2314ea28bfd7008e6593cb6e6b3
default/square-00001-4gh45[build-step-build-and-push]: 2018/10/15 18:21:08 pushed blob sha256:0511902e1bcd00eb5002093590197ce40e984c181e63719167e393b6a06cf944
default/square-00001-4gh45[build-step-build-and-push]: 2018/10/15 18:21:08 gcr.io/fe-shgupta/square:latest: digest: sha256:afa9704fab47c8cae57501e63f3ff1d54092eb23ec59d5c67350a2405a47c0a0 size: 1566
default/square-00001-deployment-547dc6bcdc-2bdww[user-container]: Node started in 90ms
default/square-00001-deployment-547dc6bcdc-2bdww[user-container]: Server starting with undefined interaction model and undefined argument type
default/square-00001-deployment-547dc6bcdc-2bdww[user-container]: gRPC loaded in 153ms
default/square-00001-deployment-547dc6bcdc-2bdww[user-container]: HTTP loaded in 95ms
default/square-00001-deployment-547dc6bcdc-2bdww[user-container]: gRPC running on localhost:10382
default/square-00001-deployment-547dc6bcdc-2bdww[user-container]: HTTP running on localhost:8080
default/square-00001-deployment-547dc6bcdc-2bdww[user-container]: Function invoker started in 343ms
riff function create completed successfully
Interesting, I'll see if I can find someone who is more familiar with Knative builds on PKS to take a look.
If it helps, the other difference is the GKE installation is 1.10.7 and PKS is 1.11.2 for K8s.
That version should be fine, I run with 11.x on minikube without problems
I am also having this issue. Let me know if you need any additional information to help resolve.
Also, similar problem on minikube - https://github.com/projectriff/riff/issues/723
I created an issue in knative/build for similar issue on GKE pushing to GCR running in a different GCP project - https://github.com/knative/build/issues/450
Please document how to allow PKS to push images to GCR (running on google cloud).
Now that riff uses the Cloud Native Buildpacks v3 and no longer uses Kaniko, this is not a problem any more.
You need to initialize the namespace with proper credentials for pushing images using:
riff namespace init default --gcr <path-to-json-key-with-image-push-credentials>
You also need to add imagePullSecret
with image-pull credentials for the namespace. That can be done using:
export PULL_IMAGE_JSON_KEY=<path-to-json-key-with-image-pull-credentials>
kubectl create secret docker-registry "gcr" \
--docker-server=gcr.io \
--docker-username=_json_key \
--docker-password="$(cat $PULL_IMAGE_JSON_KEY)" \
--docker-email=$(gcloud config get-value core/account) \
--namespace "default"
kubectl patch serviceaccount "default" \
--patch '{"imagePullSecrets": [{"name": "gcr"}]}' \
--namespace "default"
We can close this issue once riff 0.2.0 is released
Thanks @trisberg - just one question:
Could you clarify if the image pull secret is required for builds when riff itself has been installed on PKS from images on public OSS repos? -- i.o.w Do builds require both pull and push credentials?
the pull secret would still be required since you are pushing the function image to private GCR repo
closing per https://github.com/projectriff/riff/issues/879#issuecomment-439450739 now that v0.2.0 has been released.
Hi, I am following the Getting Started on GKE guide, but I am using PKS 1.2 deployed on GCP. I am following the instructions but getting this error while trying to create function:
I did initialize by using:
riff namespace init default --gcr gcr-storage-admin.json
and I can see the service account being created but push-credentials don't look right?