Incomplete deployment with private Docker registry

[mcopik]

Hi!

I've been following the documentation on how to set up OpenWhisk with a private Docker registry. After configuring authorization with a self-signed certificate, which works fine, and finding out all versions of images that need to be transported to the private registry, I got the deployment with helm almost ready. Everything seems to be deployed except for invoker pods:

openwhisk            owdev-alarmprovider-85c6cb4f6d-gsfnz                         1/1     Running            0          3m54s
openwhisk            owdev-apigateway-64c77ddb4-2gvr8                             1/1     Running            0          3m54s
openwhisk            owdev-controller-0                                           1/1     Running            0          3m54s
openwhisk            owdev-couchdb-d75b8cf5c-xlxz5                                1/1     Running            0          3m54s
openwhisk            owdev-gen-certs-6tpbz                                        0/1     Completed          0          3m54s
openwhisk            owdev-init-couchdb-7cqtq                                     0/1     Completed          0          3m54s
openwhisk            owdev-install-packages-fhg2l                                 0/1     Init:0/1           0          3m54s
openwhisk            owdev-invoker-0                                              1/1     Running            0          3m54s
openwhisk            owdev-kafka-0                                                1/1     Running            0          3m54s
openwhisk            owdev-kafkaprovider-696dcc45f9-24m4c                         1/1     Running            0          3m54s
openwhisk            owdev-nginx-65775cb5d6-zzsng                                 1/1     Running            0          3m54s
openwhisk            owdev-redis-7775bb848f-f2twc                                 1/1     Running            0          3m54s
openwhisk            owdev-seccomp-6q9cd                                          1/1     Running            0          3m54s
openwhisk            owdev-seccomp-n5x8m                                          1/1     Running            0          3m54s
openwhisk            owdev-wskadmin                                               1/1     Running            0          3m54s
openwhisk            owdev-zookeeper-0                                            1/1     Running            0          3m54s
openwhisk            wskowdev-invoker-00-6-whisksystem-invokerhealthtestaction0   0/1     ErrImagePull       0          40s
openwhisk            wskowdev-invoker-00-7-prewarm-nodejs14                       0/1     ErrImagePull       0          39s

A closer inspection suggests that pods are not authorized to access the registry and it does not look like any certificate and network availability issue, but an incorrect/missing authorization:

Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  31s                default-scheduler  Successfully assigned openwhisk/wskowdev-invoker-00-1-prewarm-nodejs14 to kind-worker2
  Normal   Pulling    16s (x2 over 30s)  kubelet            Pulling image "192.168.0.19:5000/openwhisk/action-nodejs-v14:1.19.0"
  Warning  Failed     16s (x2 over 30s)  kubelet            Failed to pull image "192.168.0.19:5000/openwhisk/action-nodejs-v14:1.19.0": rpc error: code = Unknown desc = failed to pull and unpack image "192.168.0.19:5000/openwhisk/action-nodejs-v14:1.19.0": failed to resolve reference "192.168.0.19:5000/openwhisk/action-nodejs-v14:1.19.0": unexpected status code [manifests 1.19.0]: 401 Unauthorized
  Warning  Failed     16s (x2 over 30s)  kubelet            Error: ErrImagePull
  Normal   BackOff    3s (x2 over 29s)   kubelet            Back-off pulling image "192.168.0.19:5000/openwhisk/action-nodejs-v14:1.19.0"
  Warning  Failed     3s (x2 over 29s)   kubelet            Error: ImagePullBackOff

I'm not using any custom images and I have not even tried to create any actions - this is also from standard deployment. The image is available in my registry. Does it look like there's a missing docker login on these pods?

I am using OpenWhisk with helm 3.8 and kind 0.11.

[mcopik]

Could my issue be a duplicate of #701?

[dgrove-oss]

Hi. Yes, I think you are getting hit with the same problem. We need to propagate the imagePullSecret into the pod template that is being used by the invoker when it asks Kubernetes to create pods for running functions.

[IsaacNez]

Hi!

I saw one of your comments to propagate the imagePullSecret using pod-template but I cannot find how to define this template since the action pods seem to not be defined as a regular one, e.g, invoker or controller. Is that correct?

[jrebmann]

I think it's the same issue as https://github.com/apache/openwhisk-deploy-kube/issues/528.

Is there any idea how to solve the problem for now?