Failed to pull a container image on quay.io

[yakiyoshi]

OpenShift 4.6 HPE CSI Driver Operator v1.4.0 (via OperatorHub)

Pulling hpestorage container image which is specified HPE CSI Driver Operatorv1.4.0 was failed.

• Operator version

$ oc get csv
NAME                                           DISPLAY                           VERSION                 REPLACES   PHASE
hpe-csi-operator.v1.4.0                        HPE CSI Operator for Kubernetes   1.4.0                              Succeeded

• HPECSIDriver.yaml

  $ cat HPECSIDriver.yaml
  apiVersion: storage.hpe.com/v1
  kind: HPECSIDriver
  metadata:
  name: csi-driver
  namespace: hpe-csi-driver
  spec:
  disableNodeConformance: false
  imagePullPolicy: IfNotPresent
  iscsi:
  chapPassword: ""
  chapUser: ""
  logLevel: info
  registry: quey.io

• Pods status that was created by operator

  $ oc get pods
  NAME                                       READY   STATUS             RESTARTS   AGE
  hpe-csi-controller-7c954d99c4-p2qs5        4/9     ImagePullBackOff   0          109m
  hpe-csi-driver-operator-84cb7b8995-sqwf4   1/1     Running            0          124m
  hpe-csi-node-6rhmz                         1/2     ImagePullBackOff   0          109m
  hpe-csi-node-grrp8                         1/2     ImagePullBackOff   0          109m
  hpe-csi-node-k468c                         1/2     ImagePullBackOff   0          109m
  nimble-csp-5d56c9d4b6-b9rx5                0/1     ErrImagePull       0          109m
  primera3par-csp-9ccf675b6-56gcs            0/1     ErrImagePull       0          109m

• kubernetes Events on the project for hpe csi driver

  $ oc get event
  LAST SEEN   TYPE      REASON                OBJECT                                          MESSAGE
  114m        Normal    Scheduled             pod/hpe-csi-controller-7c954d99c4-p2qs5         Successfully assigned hpe-csi-driver/hpe-csi-controller-7c954d99c4-p2qs5 to openshift-worker-0
  114m        Normal    Pulling               pod/hpe-csi-controller-7c954d99c4-p2qs5         Pulling image "quay.io/k8scsi/csi-provisioner:v1.5.0"
  114m        Normal    Pulled                pod/hpe-csi-controller-7c954d99c4-p2qs5         Successfully pulled image "quay.io/k8scsi/csi-provisioner:v1.5.0" in 6.307556687s
  114m        Normal    Created               pod/hpe-csi-controller-7c954d99c4-p2qs5         Created container csi-provisioner
  114m        Normal    Started               pod/hpe-csi-controller-7c954d99c4-p2qs5         Started container csi-provisioner
  114m        Normal    Pulling               pod/hpe-csi-controller-7c954d99c4-p2qs5         Pulling image "quay.io/k8scsi/csi-attacher:v2.1.1"
  114m        Normal    Pulled                pod/hpe-csi-controller-7c954d99c4-p2qs5         Successfully pulled image "quay.io/k8scsi/csi-attacher:v2.1.1" in 8.435693571s
  114m        Normal    Created               pod/hpe-csi-controller-7c954d99c4-p2qs5         Created container csi-attacher
  114m        Normal    Started               pod/hpe-csi-controller-7c954d99c4-p2qs5         Started container csi-attacher
  114m        Normal    Pulling               pod/hpe-csi-controller-7c954d99c4-p2qs5         Pulling image "quay.io/k8scsi/csi-snapshotter:v3.0.2"
  114m        Normal    Pulled                pod/hpe-csi-controller-7c954d99c4-p2qs5         Successfully pulled image "quay.io/k8scsi/csi-snapshotter:v3.0.2" in 10.22304632s
  114m        Normal    Created               pod/hpe-csi-controller-7c954d99c4-p2qs5         Created container csi-snapshotter
  114m        Normal    Started               pod/hpe-csi-controller-7c954d99c4-p2qs5         Started container csi-snapshotter
  114m        Normal    Pulling               pod/hpe-csi-controller-7c954d99c4-p2qs5         Pulling image "quay.io/k8scsi/csi-resizer:v0.4.0"
  114m        Normal    Pulled                pod/hpe-csi-controller-7c954d99c4-p2qs5         Successfully pulled image "quay.io/k8scsi/csi-resizer:v0.4.0" in 8.341872818s
  114m        Normal    Created               pod/hpe-csi-controller-7c954d99c4-p2qs5         Created container csi-resizer
  114m        Normal    Started               pod/hpe-csi-controller-7c954d99c4-p2qs5         Started container csi-resizer
  114m        Normal    Pulling               pod/hpe-csi-controller-7c954d99c4-p2qs5         Pulling image "quey.io/hpestorage/csi-driver:v1.4.0"
  114m        Warning   Failed                pod/hpe-csi-controller-7c954d99c4-p2qs5         Failed to pull image "quey.io/hpestorage/csi-driver:v1.4.0": rpc error: code = Unknown desc = invalid character '<' looking for beginning of value
  114m        Warning   Failed                pod/hpe-csi-controller-7c954d99c4-p2qs5         Error: ErrImagePull
  114m        Normal    Pulling               pod/hpe-csi-controller-7c954d99c4-p2qs5         Pulling image "quey.io/hpestorage/volume-mutator:v1.2.0"
  114m        Warning   Failed                pod/hpe-csi-controller-7c954d99c4-p2qs5         Failed to pull image "quey.io/hpestorage/volume-mutator:v1.2.0": rpc error: code = Unknown desc = invalid character '<' looking for beginning of value
  114m        Warning   Failed                pod/hpe-csi-controller-7c954d99c4-p2qs5         Error: ErrImagePull
  114m        Normal    Pulling               pod/hpe-csi-controller-7c954d99c4-p2qs5         Pulling image "quey.io/hpestorage/volume-group-snapshotter:v1.0.0"
  114m        Warning   Failed                pod/hpe-csi-controller-7c954d99c4-p2qs5         Failed to pull image "quey.io/hpestorage/volume-group-snapshotter:v1.0.0": rpc error: code = Unknown desc = invalid character '<' looking for beginning of value
  114m        Warning   Failed                pod/hpe-csi-controller-7c954d99c4-p2qs5         Error: ErrImagePull
  94m         Normal    BackOff               pod/hpe-csi-controller-7c954d99c4-p2qs5         Back-off pulling image "quey.io/hpestorage/csi-driver:v1.4.0"
  8m30s       Normal    Pulled                pod/hpe-csi-controller-7c954d99c4-p2qs5         Container image "quay.io/k8scsi/csi-provisioner:v1.5.0" already present on machine
  8m30s       Warning   Failed                pod/hpe-csi-controller-7c954d99c4-p2qs5         Error: services have not yet been read at least once, cannot construct envvars
  <ommit>

  All events is bellow. oc_get_event.txt

And also hpe csi driver operator didn't have correct authority and couldn't deploy some clusterroles. hpe-csi-operator-sa need patch operation for storage.hpe.com API. I adding it in the clusterrole attached the SA manually.

• I could find the bellow messages on operator container logger

  $ oc logs hpe-csi-driver-operator-84cb7b8995-sqwf4
  <ommit>
  I0117 13:35:40.958479       1 request.go:645] Throttling request took 1.048280091s, request: GET:https://172.30.0.1:443/apis/batch/v1beta1?timeout=32s
  {"level":"error","ts":1610890547.0710552,"logger":"helm.controller","msg":"Release failed","namespace":"hpe-csi-driver","name":"csi-driver","apiVersion":"storage.hpe.com/v1","kind":"HPECSIDriver","release":"csi-driver","error":"failed to install release: clusterroles.rbac.authorization.k8s.io \"hpe-csi-driver-role\" is forbidden: user \"system:serviceaccount:hpe-csi-driver:hpe-csi-operator-sa\" (groups=[\"system:serviceaccounts\" \"system:serviceaccounts:hpe-csi-driver\" \"system:authenticated\"]) is attempting to grant RBAC permissions not currently held:\n{APIGroups:[\"storage.hpe.com\"], Resources:[\"hpesnapshotgroupinfos\"], Verbs:[\"patch\"]}\n{APIGroups:[\"storage.hpe.com\"], Resources:[\"hpevolumegroupinfos\"], Verbs:[\"patch\"]}","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/travis/gopath/pkg/mod/github.com/go-logr/zapr@v0.2.0/zapr.go:132\ngithub.com/operator-framework/operator-sdk/internal/helm/controller.HelmOperatorReconciler.Reconcile\n\toperator-sdk/internal/helm/controller/reconcile.go:180\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/travis/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/travis/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:235\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.1\n\t/home/travis/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:198\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1\n\t/home/travis/gopath/pkg/mod/k8s.io/apimachinery@v0.19.4/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/home/travis/gopath/pkg/mod/k8s.io/apimachinery@v0.19.4/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/home/travis/gopath/pkg/mod/k8s.io/apimachinery@v0.19.4/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/travis/gopath/pkg/mod/k8s.io/apimachinery@v0.19.4/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext\n\t/home/travis/gopath/pkg/mod/k8s.io/apimachinery@v0.19.4/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.UntilWithContext\n\t/home/travis/gopath/pkg/mod/k8s.io/apimachinery@v0.19.4/pkg/util/wait/wait.go:99"}
  {"level":"error","ts":1610890547.0751362,"logger":"controller-runtime.manager.controller.hpecsidriver-controller","msg":"Reconciler error","name":"csi-driver","namespace":"hpe-csi-driver","error":"failed to install release: clusterroles.rbac.authorization.k8s.io \"hpe-csi-driver-role\" is forbidden: user \"system:serviceaccount:hpe-csi-driver:hpe-csi-operator-sa\" (groups=[\"system:serviceaccounts\" \"system:serviceaccounts:hpe-csi-driver\" \"system:authenticated\"]) is attempting to grant RBAC permissions not currently held:\n{APIGroups:[\"storage.hpe.com\"], Resources:[\"hpesnapshotgroupinfos\"], Verbs:[\"patch\"]}\n{APIGroups:[\"storage.hpe.com\"], Resources:[\"hpevolumegroupinfos\"], Verbs:[\"patch\"]}","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/travis/gopath/pkg/mod/github.com/go-logr/zapr@v0.2.0/zapr.go:132\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/travis/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:267\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/travis/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:235\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.1\n\t/home/travis/gopath/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:198\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1\n\t/home/travis/gopath/pkg/mod/k8s.io/apimachinery@v0.19.4/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/home/travis/gopath/pkg/mod/k8s.io/apimachinery@v0.19.4/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/home/travis/gopath/pkg/mod/k8s.io/apimachinery@v0.19.4/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/travis/gopath/pkg/mod/k8s.io/apimachinery@v0.19.4/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext\n\t/home/travis/gopath/pkg/mod/k8s.io/apimachinery@v0.19.4/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.UntilWithContext\n\t/home/travis/gopath/pkg/mod/k8s.io/apimachinery@v0.19.4/pkg/util/wait/wait.go:99"}
  <ommit>

• I updated clusterrole manually

  
  $ oc get clusterrole hpe-csi-operator.v1.4.0-6d789d6f5c -o yaml
  apiVersion: rbac.authorization.k8s.io/v1
  kind: ClusterRole
  metadata:
  name: hpe-csi-operator.v1.4.0-6d789d6f5c
  rules:

• apiGroups:

  • storage.hpe.com resources:
  • '*' verbs:
  • get
  • watch
  • list
  • delete
  • update
  • create
  • patch <---------- I added this

• apiGroups:

  • "" resources:
  • namespaces
  • pods
  • services
  • endpoints ```

[raunakkumar]

Hi @yakiyoshi Looks like there is a typo in the HPECSIDriver.yaml for the key registry. It looks to be quey.io instead of quay.io. The official document on operator hub refers to quay.io. Did you modify the value?

[datamattsson]

@raunakkumar this reveals the bug that we've forgotten to prefix the CSI sidecars with the registry variable.

[raunakkumar]

We have prefixed the side cars with registry prefix for all the hpe images. https://github.com/hpe-storage/co-deployments/blob/14cae90fe95ba6a297275e0d0f206af1da4f957b/helm/charts/hpe-csi-driver/templates/hpe-csi-controller.yaml#L99-L100

[yakiyoshi]

Oops. Sorry for the typo. Thank you for pointing that out.

I'v already released ocp cluster and hardwares are sharing with another. I can't use any more. If no other disscusions, I would like to close this issue.