HPECSIDriver Conditions: Initialized, ReleaseFailed - failed to install release: clusterroles.rbac.authorization.k8s.io ...

[justin-cohesity]

Operator v1.4 OCP 4.6.9

Pre-created the scc per the install documentation before installing the csi driver

cat hpe-csi-scc.yaml

kind: SecurityContextConstraints apiVersion: security.openshift.io/v1 metadata: name: hpe-csi-scc allowHostDirVolumePlugin: true allowHostIPC: true allowHostNetwork: true allowHostPID: true allowHostPorts: true allowPrivilegeEscalation: true allowPrivilegedContainer: true allowedCapabilities:

• '*' defaultAddCapabilities: [] fsGroup: type: RunAsAny groups: [] priority: readOnlyRootFilesystem: false requiredDropCapabilities: [] runAsUser: type: RunAsAny seLinuxContext: type: RunAsAny supplementalGroups: type: RunAsAny users:
• system:serviceaccount:hpe-csi-driver:hpe-csi-controller-sa
• system:serviceaccount:hpe-csi-driver:hpe-csi-node-sa
• system:serviceaccount:hpe-csi-driver:hpe-csp-sa
• system:serviceaccount:hpe-csi-driver:hpe-csi-operator-sa
• system:serviceaccount:hpe-nfs:hpe-csi-nfs-sa volumes:
• '*'

oc create -f hpe-csi-scc.yaml securitycontextconstraints.security.openshift.io/hpe-csi-scc created

Errors when deploying/creating the HPECSIDriver instance:

status: conditions:

• lastTransitionTime: '2021-01-28T14:48:31Z' status: 'True' type: Initialized

• lastTransitionTime: '2021-01-28T14:48:35Z' message: >- failed to install release: clusterroles.rbac.authorization.k8s.io "hpe-csi-volumegroup-role" is forbidden: user "system:serviceaccount:hpe-csi-driver:hpe-csi-operator-sa" (groups=["system:serviceaccounts" "system:serviceaccounts:hpe-csi-driver" "system:authenticated"]) is attempting to grant RBAC permissions not currently held:

  {APIGroups:["storage.hpe.com"], Resources:["volumegroupclasses"],
  Verbs:["patch"]}
  
  {APIGroups:["storage.hpe.com"], Resources:["volumegroupcontents"],
  Verbs:["patch"]}
  
  {APIGroups:["storage.hpe.com"],
  Resources:["volumegroupcontents/status"], Verbs:["patch"]}
  
  {APIGroups:["storage.hpe.com"], Resources:["volumegroups"],
  Verbs:["patch"]}
  
  {APIGroups:["storage.hpe.com"], Resources:["volumegroups/status"],
  Verbs:["patch"]}

  reason: InstallError status: 'True' type: ReleaseFailed

Is this a known issue? I don't recall running into the issue with OCP v4.6 /w HPE CSI Driver v1.3, however, with trying to set this up with v1.4 I'm running into the above.

Thanks

[datamattsson]

That's really strange and particular. Those RBAC permissions are part of the hpe-csi-volumegroup-role: https://github.com/hpe-storage/co-deployments/blob/4eaefe31eeb09d9e8a1de3f0c36fbd362a2473a2/helm/charts/hpe-csi-driver/templates/hpe-csi-rbac.yaml#L268

Are you creating the HPECSIDriver instance in the "hpe-csi-driver" namespace/project?

cc: @c-snell

[justin-cohesity]

Yes, it is being created in the "hpe-csi-driver" namespace/project.

[justin-cohesity]

I see a few clusterroles missing on the OpenShift cluster for which HPECSIDriver instance is not installing:

On an OCP 4.6 cluster /w working v1.3 HPE CSI driver:

oc get clusterrole | grep hpe hpe-csi-attacher-role 2021-01-14T14:40:14Z hpe-csi-driver-nj6zj-admin 2021-01-14T13:42:27Z hpe-csi-driver-nj6zj-edit 2021-01-14T13:42:27Z hpe-csi-driver-nj6zj-view 2021-01-14T13:42:27Z hpe-csi-driver-role 2021-01-14T14:40:14Z hpe-csi-operator.v1.3.0-6556f4556 2021-01-14T13:43:26Z hpe-csi-provisioner-role 2021-01-14T14:40:14Z hpe-csi-snapshotter-role 2021-01-14T14:40:14Z hpecsidrivers.storage.hpe.com-v1-admin 2021-01-14T13:43:43Z hpecsidrivers.storage.hpe.com-v1-crdview 2021-01-14T13:43:43Z hpecsidrivers.storage.hpe.com-v1-edit 2021-01-14T13:43:43Z hpecsidrivers.storage.hpe.com-v1-view 2021-01-14T13:43:43Z

On an OCP 4.6 cluster that has issues /w v1.4 HPE CSI driver:

oc get clusterrole | grep hpe hpe-csi-driver-s7bkf-admin 2021-01-28T21:21:34Z hpe-csi-driver-s7bkf-edit 2021-01-28T21:21:34Z hpe-csi-driver-s7bkf-view 2021-01-28T21:21:34Z hpe-csi-operator.v1.4.0-6d789d6f5c 2021-01-28T21:24:36Z hpecsidrivers.storage.hpe.com-v1-admin 2021-01-28T21:24:42Z hpecsidrivers.storage.hpe.com-v1-crdview 2021-01-28T21:24:42Z hpecsidrivers.storage.hpe.com-v1-edit 2021-01-28T21:24:42Z hpecsidrivers.storage.hpe.com-v1-view 2021-01-28T21:24:42Z

[datamattsson]

@justin-cohesity sorry to leave you hanging. We've confirmed there's an issue. Hang on.

[justin-cohesity]

Sure, no worries @datamattsson Thanks for looking into this issue!

[datamattsson]

@justin-cohesity this seems to be the same issue as #237 (not the pull image issue, but the role issue) but we have pushed version 1.4.1 of the operator that has this fixed but we're seeing an issue in our environment pulling 1.4.1 so we need to open a ticket with Red Hat. Please stay tuned as we sort this out. In the meantime you can install 1.4.0 and edit the role manually as per #237

[raunakkumar]

@justin-cohesity, we just got an update from Red Hat that the image publish issue has been fixed. Could you try to install the 1.4.1 of the operator on the cluster and see if it works for you?