search

apache / openwhisk-deploy-kube

The Apache OpenWhisk Kubernetes Deployment repository supports deploying the Apache OpenWhisk system on Kubernetes and OpenShift clusters.
https://openwhisk.apache.org/
Apache License 2.0
302 stars 232 forks source link

Unable to pull pre-warm images from private docker registry #701

Closed pankaj-dahiya-devops closed 3 years ago

pankaj-dahiya-devops commented 3 years ago

I am unable to pull pre-warm image - "myregistry/action-nodejs-v10" from my private gitlab registry.

Steps to replicate:

  1. Pull open whisk helm chart.
  2. Edit values.yml and provide docker.registry.name, username, password:
  3. Helm install

All images will be pulled successfully and all the pods will get into a running state. But Pre-warm pods will give an error as "ErrImagePull".

Events: Type Reason Age From Message

Normal Scheduled 36s default-scheduler Successfully assigned default/wskpankaj-invoker-00-10-prewarm-nodejs10 to ip-172-31-39-89 Normal BackOff 13s (x2 over 35s) kubelet Back-off pulling image "registry.gitlab.com//openwhisk/action-nodejs-v10:1.16.0" Warning Failed 13s (x2 over 35s) kubelet Error: ImagePullBackOff Normal Pulling 1s (x3 over 35s) kubelet Pulling image "registry.gitlab.com/MYREP>/openwhisk/action-nodejs-v10:1.16.0" Warning Failed 1s (x3 over 35s) kubelet Failed to pull image "registry.gitlab.com/MYREPO/openwhisk/action-nodejs-v10:1.16.0": rpc error: code = Unknown desc = failed to pull and unpack image "registry.gitlab.com/MYREPO/openwhisk/action-nodejs-v10:1.16.0": failed to resolve reference "registry.gitlab.com/MYREPO/openwhisk/action-nodejs-v10:1.16.0": failed to authorize: failed to fetch anonymous token: unexpected status: 403 Forbidden Warning Failed 1s (x3 over 35s) kubelet Error: ErrImagePull

dgrove-oss commented 3 years ago

Is it only the preWarm pods that are having trouble pulling the image, or is it a problem for any pod being created by an invoker using this image to run user actions?

I'd speculate you are probably seeing the later, which means the imagePullSecret needs to be propagated into the pod template the invoker uses to create user action pods. There is poorly documented support for doing this by injecting a value into the invoker's configuration value: whisk.kubernetes.podTemplate https://github.com/apache/openwhisk/blob/master/core/invoker/src/main/resources/application.conf#L87-#L91.

pankaj-dahiya-devops commented 3 years ago

Hi, @dgrove-oss I am deploying Openwhisk via helm carts, In helm charts, there is an option in values.yaml which asks us to private docker registry and its credentials which are further passed to templates/_invoker-helpers.tpl where we define "openwhisk.docker_pull_runtimes" which includes these credentials.

So, as per me, providing registry credentials in values.yaml must be enough!

One more thing, as per industry standards, the User should provide a .Values.imagePullSecretsin values.yaml and that should be propagated to everywhere else but in Openwhisk this is completely useless as this .Values.imagePullSecrets is not passed anywhere to any pods of Openwhisk and we must provide our registry credentials to Openwhisk that too in plain text format!.

dgrove-oss commented 3 years ago

The helm chart supports using a private registry with the DockerContainerFactory. It is missing support for using the KubernetesContainerFactory with a private registry. Contributions would be welcome.

pankaj-dahiya-devops commented 3 years ago

Hi ... this thing should also be supported in KubernatesContainerFactory. Will raise a PR by next month for this. (Bit busy now). So closing the issue as of now.