Open trisberg opened 5 years ago
/assign pivotal-nader-ziada
For Kaniko to be able to push to a Google Cloud Registry on a different project, it needs a Kubernetes secret, which contains the auth required to push the final image.
From the Kaniko docs: How to run Kaniko on a Kubernetes cluster
you must first create the secret
kubectl create secret generic kaniko-secret --from-file=<path-to-service-account.json>
And make the Pod look like this
apiVersion: v1
kind: Pod
metadata:
name: kaniko
spec:
containers:
- name: kaniko
image: gcr.io/kaniko-project/executor:latest
args: ["--dockerfile=<path to Dockerfile within the build context>",
"--context=gs://<GCS bucket>/<path to .tar.gz>",
"--destination=<gcr.io/$PROJECT/$IMAGE:$TAG>"]
volumeMounts:
- name: kaniko-secret
mountPath: /secret
env:
- name: GOOGLE_APPLICATION_CREDENTIALS
value: /secret/kaniko-secret.json
restartPolicy: Never
volumes:
- name: kaniko-secret
secret:
secretName: kaniko-secret
but currently in Kaniko build-template, the volume and secret are not supported
we have the following options:
ApplyTemplate
logic in build to somehow allow the user to append to the templateI'm able to push to a Google Cloud Registry from a PKS cluster using the following custom template: (@trisberg if you need a workaround)
apiVersion: build.knative.dev/v1alpha1
kind: BuildTemplate
metadata:
name: kaniko-for-external-cluster
spec:
parameters:
- name: IMAGE
description: The name of the image to push
- name: DOCKERFILE
description: Path to the Dockerfile to build.
default: /workspace/Dockerfile
steps:
- name: build-and-push
image: gcr.io/kaniko-project/executor
args:
- --dockerfile=${DOCKERFILE}
- --destination=${IMAGE}
volumeMounts:
- name: kaniko-secret
mountPath: /secret
env:
- name: GOOGLE_APPLICATION_CREDENTIALS
value: /secret/kaniko-secret.json
volumes:
- name: kaniko-secret
secret:
secretName: kaniko-secret
@ImJasonH @shashwathi wdyt?
We already create a secret for the service account, are you saying that we need an additional secret, that seems like the wrong solution? I'm having same issue using the buildpack template as well, so it seems to be a general problem that the build doesn't use the secret provided to the service account.
The Kaniko build-template doesn't pick up the secret unless its in the format mentioned here: https://github.com/GoogleContainerTools/kaniko#kubernetes-secret
Even if you have the secret, Kaniko doesn't read it.
Using the custom template above, I can push an image to gcr from a pks cluster
It seems to use the service account secret when pushing to Docker Hub, so I wonder why GCR is different.
In the Kaniko docs here https://github.com/GoogleContainerTools/kaniko it says there are different methods of authentication based on the target registry
@trisberg
... I'm having same issue using the buildpack template as well, so it seems to be a general problem that the build doesn't use the secret provided to the service account.
I neither tried Kaniko nor GCR, but had the same issue using the buildpack template on an AWS K8s cluster using ECR as docker registry.
The reason for this behavior is that the buildpack export step uses credential helpers for GCP, AWS, and Azure by default. On AWS this helper then creates a docker configuration based on the IAM role of the EC2 instance that runs the build pod.
Disabling these helpers results in the service account's secret to be used. See https://github.com/knative/build-templates/pull/83
What's the current status on this issue? I ran into a similar problem, following the kaniko example and pushing to GCR. Is the problem here with kaniko or is this a knative issue?
Following the breadcrumbs, I think this is resolved by https://github.com/knative/build-templates/pull/87
@dlorenc is that right?
Correct, it wasn't really an issue with knative or kaniko, just the interaction in between.
@dlorenc @jonjohnsonjr Thanks for the quick response! I'll see if that works and report back.
@dlorenc That worked!
Expected Behavior
I can have a build push images to GCR from cluster that is not a GKE cluster within the same project as GCR
Actual Behavior
Getting auth errors. Looks like some GCR Access Control is missing when building and pushing images from non GKE cluster or a GKE cluster that is running under a different project.
Steps to Reproduce the Problem
Create a Kubernetes cluster using a different project and install recent Knative build and serving.
Create secret and k8s service account
where
kaniko-sa.yaml
is:kaniko-square.yaml
and check the build status and logs.
Additional Info
I got this: