search

dell / csm

Dell Container Storage Modules (CSM)
Apache License 2.0
67 stars 15 forks source link

[BUG]: PowerStore fails to provision after upgrade #1405

Closed a-chartier closed 1 month ago

a-chartier commented 1 month ago

Bug Description

We recently upgraded our PowerStore and all existing NFS connection work however we are unable to provision new ones. This is only impacting NFS , Block continues to work fine.

I have included a snippet of the relevant section of the provisioning log.

Storage Class

allowVolumeExpansion: true
allowedTopologies:
- matchLabelExpressions:
  - key: csi-powerstore.dellemc.com/10.1.1.1-nfs
    values:
    - "true"
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    storageclass.kubernetes.io/is-default-class: "false"
  name: dell-powerstore-nfs
parameters:
  FsType: nfs
  allowRoot: "true"
  arrayID: SLD-CNSN1
provisioner: csi-powerstore.dellemc.com
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

Logs

I0729 16:39:48.337652       1 controller.go:1359] provision "default/testing" class "dell-powerstore-nfs": started
W0729 16:39:48.337716       1 controller.go:620] "fstype" is deprecated and will be removed in a future release, please use "csi.storage.k8s.io/fstype" instead
I0729 16:39:48.337752       1 connection.go:193] GRPC call: /csi.v1.Controller/CreateVolume
I0729 16:39:48.337765       1 event.go:298] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"testing", UID:"01956b46-76dc-423d-89c0-78fecf19fa6f", APIVersion:"v1", ResourceVersion:"1287635746", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/testing"
I0729 16:39:48.337762       1 connection.go:194] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"csi-powerstore.dellemc.com/10.1.1.1-nfs":"true"}}],"requisite":[{"segments":{"csi-powerstore.dellemc.com/10.1.1.1-nfs":"true"}}]},"capacity_range":{"required_bytes":3221225472},"name":"csivol-01956b4676","parameters":{"FsType":"nfs","allowRoot":"true","arrayID":"SLD-CNSN1","csi.storage.k8s.io/pv/name":"csivol-01956b4676","csi.storage.k8s.io/pvc/name":"testing","csi.storage.k8s.io/pvc/namespace":"default"},"volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"nfs"}},"access_mode":{"mode":5}}]}
I0729 16:39:48.921410       1 connection.go:200] GRPC response: {}
I0729 16:39:48.921433       1 connection.go:201] GRPC error: rpc error: code = Internal desc = can't find filesystem 'csivol-01956b4676':
I0729 16:39:48.921454       1 controller.go:816] CreateVolume failed, supports topology = true, node selected true => may reschedule = true => state = Finished: rpc error: code = Internal desc = can't find filesystem 'csivol-01956b4676':
I0729 16:39:48.921468       1 capacity.go:459] Capacity Controller: enqueuing {segment:0xc0010ae810 storageClassName:dell-powerstore-nfs} because of the storage class
I0729 16:39:48.921489       1 capacity.go:459] Capacity Controller: enqueuing {segment:0xc0010ae840 storageClassName:dell-powerstore-nfs} because of the storage class
I0729 16:39:48.921541       1 controller.go:1075] Final error received, removing PVC 01956b46-76dc-423d-89c0-78fecf19fa6f from claims in progress
W0729 16:39:48.921552       1 controller.go:934] Retrying syncing claim "01956b46-76dc-423d-89c0-78fecf19fa6f", failure 0
E0729 16:39:48.921572       1 controller.go:957] error syncing claim "01956b46-76dc-423d-89c0-78fecf19fa6f": failed to provision volume with StorageClass "dell-powerstore-nfs": rpc error: code = Internal desc = can't find filesystem 'csivol-01956b4676':
I0729 16:39:48.921573       1 capacity.go:574] Capacity Controller: refreshing {segment:0xc0010ae810 storageClassName:dell-powerstore-nfs}
I0729 16:39:48.921611       1 connection.go:193] GRPC call: /csi.v1.Controller/GetCapacity
I0729 16:39:48.921656       1 event.go:298] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"testing", UID:"01956b46-76dc-423d-89c0-78fecf19fa6f", APIVersion:"v1", ResourceVersion:"1287635746", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "dell-powerstore-nfs": rpc error: code = Internal desc = can't find filesystem 'csivol-01956b4676':
I0729 16:39:48.921622       1 connection.go:194] GRPC request: {"accessible_topology":{"segments":{"csi-powerstore.dellemc.com/10.1.1.1-iscsi":"true","csi-powerstore.dellemc.com/10.1.1.1-nfs":"true"}},"parameters":{"FsType":"nfs","allowRoot":"true","arrayID":"SLD-CNSN1"},"volume_capabilities":[{"AccessType":{"Mount":null},"access_mode":{}}]}
I0729 16:39:49.209539       1 connection.go:200] GRPC response: {"available_capacity":12663281020205}
I0729 16:39:49.209556       1 connection.go:201] GRPC error: <nil>
I0729 16:39:49.209570       1 capacity.go:656] Capacity Controller: no need to update csisc-4vhnn for {segment:0xc0010ae810 storageClassName:dell-powerstore-nfs}, same capacity 12663281020205, same maximumVolumeSize <nil> and correct owner
I0729 16:39:49.209597       1 capacity.go:574] Capacity Controller: refreshing {segment:0xc0010ae840 storageClassName:dell-powerstore-nfs}
I0729 16:39:49.209614       1 connection.go:193] GRPC call: /csi.v1.Controller/GetCapacity
I0729 16:39:49.209618       1 connection.go:194] GRPC request: {"accessible_topology":{"segments":{"csi-powerstore.dellemc.com/10.1.1.1-nfs":"true"}},"parameters":{"FsType":"nfs","allowRoot":"true","arrayID":"SLD-CNSN1"},"volume_capabilities":[{"AccessType":{"Mount":null},"access_mode":{}}]}
I0729 16:39:49.450140       1 connection.go:200] GRPC response: {"available_capacity":12663281020205}
I0729 16:39:49.450163       1 connection.go:201] GRPC error: <nil>
I0729 16:39:49.450181       1 capacity.go:656] Capacity Controller: no need to update csisc-ftkc4 for {segment:0xc0010ae840 storageClassName:dell-powerstore-nfs}, same capacity 12663281020205, same maximumVolumeSize <nil> and correct owner
I0729 16:39:49.921977       1 controller.go:1359] provision "default/testing" class "dell-powerstore-nfs": started
W0729 16:39:49.922077       1 controller.go:620] "fstype" is deprecated and will be removed in a future release, please use "csi.storage.k8s.io/fstype" instead
I0729 16:39:49.922141       1 connection.go:193] GRPC call: /csi.v1.Controller/CreateVolume
I0729 16:39:49.922135       1 event.go:298] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"testing", UID:"01956b46-76dc-423d-89c0-78fecf19fa6f", APIVersion:"v1", ResourceVersion:"1287635746", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/testing"
I0729 16:39:49.922149       1 connection.go:194] GRPC request: {"accessibility_requirements":{"preferred":[{"segments":{"csi-powerstore.dellemc.com/10.1.1.1-nfs":"true"}}],"requisite":[{"segments":{"csi-powerstore.dellemc.com/10.1.1.1-nfs":"true"}}]},"capacity_range":{"required_bytes":3221225472},"name":"csivol-01956b4676","parameters":{"FsType":"nfs","allowRoot":"true","arrayID":"SLD-CNSN1","csi.storage.k8s.io/pv/name":"csivol-01956b4676","csi.storage.k8s.io/pvc/name":"testing","csi.storage.k8s.io/pvc/namespace":"default"},"volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"nfs"}},"access_mode":{"mode":5}}]}
I0729 16:39:50.214794       1 connection.go:200] GRPC response: {}
I0729 16:39:50.214813       1 connection.go:201] GRPC error: rpc error: code = Internal desc = can't find filesystem 'csivol-01956b4676':
I0729 16:39:50.214831       1 controller.go:816] CreateVolume failed, supports topology = true, node selected true => may reschedule = true => state = Finished: rpc error: code = Internal desc = can't find filesystem 'csivol-01956b4676':
I0729 16:39:50.214844       1 capacity.go:459] Capacity Controller: enqueuing {segment:0xc0010ae810 storageClassName:dell-powerstore-nfs} because of the storage class
I0729 16:39:50.214861       1 capacity.go:459] Capacity Controller: enqueuing {segment:0xc0010ae840 storageClassName:dell-powerstore-nfs} because of the storage class
I0729 16:39:50.214887       1 controller.go:1075] Final error received, removing PVC 01956b46-76dc-423d-89c0-78fecf19fa6f from claims in progress
W0729 16:39:50.214898       1 controller.go:934] Retrying syncing claim "01956b46-76dc-423d-89c0-78fecf19fa6f", failure 1
E0729 16:39:50.214915       1 controller.go:957] error syncing claim "01956b46-76dc-423d-89c0-78fecf19fa6f": failed to provision volume with StorageClass "dell-powerstore-nfs": rpc error: code = Internal desc = can't find filesystem 'csivol-01956b4676':
I0729 16:39:50.214927       1 capacity.go:574] Capacity Controller: refreshing {segment:0xc0010ae810 storageClassName:dell-powerstore-nfs}
I0729 16:39:50.214944       1 connection.go:193] GRPC call: /csi.v1.Controller/GetCapacity
I0729 16:39:50.214979       1 event.go:298] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"testing", UID:"01956b46-76dc-423d-89c0-78fecf19fa6f", APIVersion:"v1", ResourceVersion:"1287635746", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "dell-powerstore-nfs": rpc error: code = Internal desc = can't find filesystem 'csivol-01956b4676':
I0729 16:39:50.214950       1 connection.go:194] GRPC request: {"accessible_topology":{"segments":{"csi-powerstore.dellemc.com/10.1.1.1-iscsi":"true","csi-powerstore.dellemc.com/10.1.1.1-nfs":"true"}},"parameters":{"FsType":"nfs","allowRoot":"true","arrayID":"SLD-CNSN1"},"volume_capabilities":[{"AccessType":{"Mount":null},"access_mode":{}}]}
I0729 16:39:50.360839       1 connection.go:200] GRPC response: {"available_capacity":12663281020205}
I0729 16:39:50.360854       1 connection.go:201] GRPC error: <nil>
I0729 16:39:50.360868       1 capacity.go:656] Capacity Controller: no need to update csisc-4vhnn for {segment:0xc0010ae810 storageClassName:dell-powerstore-nfs}, same capacity 12663281020205, same maximumVolumeSize <nil> and correct owner
I0729 16:39:50.360896       1 capacity.go:574] Capacity Controller: refreshing {segment:0xc0010ae840 storageClassName:dell-powerstore-nfs}
I0729 16:39:50.360912       1 connection.go:193] GRPC call: /csi.v1.Controller/GetCapacity
I0729 16:39:50.360916       1 connection.go:194] GRPC request: {"accessible_topology":{"segments":{"csi-powerstore.dellemc.com/10.1.1.1-nfs":"true"}},"parameters":{"FsType":"nfs","allowRoot":"true","arrayID":"SLD-CNSN1"},"volume_capabilities":[{"AccessType":{"Mount":null},"access_mode":{}}]}
I0729 16:39:50.506866       1 connection.go:200] GRPC response: {"available_capacity":12663281020205}
I0729 16:39:50.506882       1 connection.go:201] GRPC error: <nil>
I0729 16:39:50.506895       1 capacity.go:656] Capacity Controller: no need to update csisc-ftkc4 for {segment:0xc0010ae840 storageClassName:dell-powerstore-nfs}, same capacity 12663281020205, same maximumVolumeSize <nil> and correct owner

Screenshots

No response

Additional Environment Information

PowerStore Version = 3.6.1.2 OpenShift version = 4.12

Steps to Reproduce

Upgraded the PowreStore Version

Expected Behavior

To be able to continue to provision NFS mounts

CSM Driver(s)

image

Installation Type

Operator

Container Storage Modules Enabled

resiliency

Container Orchestrator

OpenShift

Operating System

coreos

csmbot commented 1 month ago

@a-chartier: Thank you for submitting this issue!

The issue is currently awaiting triage. Please make sure you have given us as much context as possible.

If the maintainers determine this is a relevant issue, they will remove the needs-triage label and respond appropriately.

We want your feedback! If you have any questions or suggestions regarding our contributing process/workflow, please reach out to us at container.storage.modules@dell.com.

gallacher commented 1 month ago

Hi @a-chartier. Container Storage Modules support for PowerStore 3.6.x was introduced in CSM 1.10. Based on our prerequisites documentation , you will need a minimum OCP version of 4.14 along with an update to CSM 1.10.

a-chartier commented 1 month ago

@gallacher - So 4.12 is still supported by CSM, and has been regardless of what that says. But if you are just saying 3.6.X needs 1.10 which is only supported on 4.14 then that's fine.

I will need to see this this can be rolled back then. Thanks.

gallacher commented 1 month ago

@a-chartier - With every release of CSM, we support N-1 versions of OCP. Support for OCP 4.12 was introduced in CSM 1.7 and it was removed in CSM 1.9.

a-chartier commented 1 month ago

Seems like the PowerStore is a one way upgrade, which now leaves me in an awkward position of not having a function storage system on 4.12

AkshaySainiDell commented 1 month ago

@a-chartier It is recommended to use CSM 1.10 with PowerStore 3.6. I looked at your logs and found "can't find filesystem 'csivol-01956b4676'" error. Could you please check if the file system 'csivol-01956b4676' is created on the array and see if there are any events/alerts reported for it.

Also, please share below info:

  1. array version before upgrade
  2. csm-operator version
  3. csi-powerstore driver version
  4. logs of driver container - kubectl logs -n csi-powerstore <powerstore-controller-pod> -c driver
a-chartier commented 1 month ago

@AkshaySainiDell

Thanks for the response. It's not created on the PowerStore. Block devices are created just fine but NFS are not and I see no errors or alerts on the PowerStore indicating something failed 1) 3.2 2) 1.2 (Installed through certified operator) 3) 2.7 4) I will need a place to upload the logs

But here is a driver snippet of it trying to create an NFS volume (new one)

{"level":"debug","msg":"release a lock","time":"2024-07-30T12:13:19.289400375Z"}
{"level":"info","msg":"/csi.v1.Controller/GetCapacity: REP 0233: AvailableCapacity=12662181963365, XXX_NoUnkeyedLiteral={}, XXX_sizecache=0","time":"2024-07-30T12:13:19.289453817Z"}
{"level":"info","msg":"/csi.v1.Controller/CreateVolume: REQ 0234: Name=csivol-141c8f9aad, CapacityRange=required_bytes:3221225472 , VolumeCapabilities=[mount:\u003cfs_type:\"nfs\" \u003e access_mode:\u003cmode:MULTI_NODE_MULTI_WRITER \u003e ], Parameters=map[FsType:nfs allowRoot:true arrayID:SLD-CNSN1 csi.storage.k8s.io/pv/name:csivol-141c8f9aad csi.storage.k8s.io/pvc/name:testing csi.storage.k8s.io/pvc/namespace:default], AccessibilityRequirements=requisite:\u003csegments:\u003ckey:\"csi-powerstore.dellemc.com/10.1.51.165-nfs\" value:\"true\" \u003e \u003e preferred:\u003csegments:\u003ckey:\"csi-powerstore.dellemc.com/10.1.51.165-nfs\" value:\"true\" \u003e \u003e , XXX_NoUnkeyedLiteral={}, XXX_sizecache=0","time":"2024-07-30T12:13:35.003426538Z"}
{"level":"debug","msg":"REQUEST: GET /api/rest/nas_server?name=eq.POWERSTORE_NAS_1\u0026select=description%2Cid%2Cname%2Ccurrent_node_id%2Coperational_status%2Ccurrent_preferred_IPv4_interface_id%2Cnfs_servers HTTP/1.1 Host: 10.1.51.165 Application-Type: CSI Driver for Dell EMC PowerStore/2.7.0 Authorization: ******  ","time":"2024-07-30T12:13:35.00355934Z"}
{"level":"debug","msg":"acquire a lock","time":"2024-07-30T12:13:35.003573224Z"}
{"level":"debug","msg":"RESPONSE: HTTP/1.1 200 OK Content-Length: 309 Cache-Control: no-cache Cache-Control: no-store Cache-Control: must-revalidate Cache-Control: max-age=0 Content-Language: en-US Content-Type: application/json Dell-Emc-Token: ****** Expires: -1 Set-Cookie: auth_cookie=******; Path=/; Secure; HTTPOnly X-Content-Type-Options: nosniff  [{\"description\":\"\",\"id\":\"643da243-a9bf-8bc3-3e0c-0e923a9c5317\",\"name\":\"POWERSTORE_NAS_1\",\"current_node_id\":\"SLD-CNSN1-appliance-1-node-A\",\"operational_status\":\"Started\",\"current_preferred_IPv4_interface_id\":\"643da256-790d-f266-11c5-0e923a9c5317\",\"nfs_servers\":[{\"id\":\"643da256-88f4-b8e6-ec4b-0e923a9c5317\"}]}]\n","time":"2024-07-30T12:13:35.014599301Z"}
{"level":"debug","msg":"release a lock","time":"2024-07-30T12:13:35.014648462Z"}
{"level":"debug","msg":"REQUEST: POST /api/rest/file_system HTTP/1.1 Host: 10.1.51.165 Application-Type: CSI Driver for Dell EMC PowerStore/2.7.0 Authorization: ****** X-Csi-Pv-Claimname: testing X-Csi-Pv-Name: csivol-141c8f9aad X-Csi-Pv-Namespace: default  {\"description\":\"testing-default\",\"name\":\"csivol-141c8f9aad\",\"nas_server_id\":\"643da243-a9bf-8bc3-3e0c-0e923a9c5317\",\"size_total\":4831838208}","time":"2024-07-30T12:13:35.014732603Z"}
{"level":"debug","msg":"acquire a lock","time":"2024-07-30T12:13:35.014745731Z"}
{"level":"info","msg":"/podmon.v1.Podmon/ValidateVolumeHostConnectivity: REQ 0235: NodeId=csi-node-b2da558a23294137b80bbce669f9eb3a-10.1.55.82, IoSampleTimeSeconds=0","time":"2024-07-30T12:13:35.134443487Z"}
{"level":"info","msg":"ValidateVolumeHostConnectivity called node_id:\"csi-node-b2da558a23294137b80bbce669f9eb3a-10.1.55.82\"","time":"2024-07-30T12:13:35.134525105Z"}
{"level":"info","msg":"neither globalId nor volumeID is present in request","time":"2024-07-30T12:13:35.134533882Z"}
{"level":"info","msg":"Checking if array SLD-CNSN1 is connected to node csi-node-b2da558a23294137b80bbce669f9eb3a-10.1.55.82","time":"2024-07-30T12:13:35.134540987Z"}
{"level":"debug","msg":"Received response \u0026{Status:200 OK StatusCode:200 Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Content-Length:[51] Content-Type:[text/plain; charset=utf-8] Date:[Tue, 30 Jul 2024 12:13:35 GMT]] Body:0xc000316860 ContentLength:51 TransferEncoding:[] Close:false Uncompressed:false Trailer:map[] Request:0xc0007b6600 TLS:\u003cnil\u003e} for url http://10.1.55.82:8083/array-status/SLD-CNSN1","time":"2024-07-30T12:13:35.136649437Z"}
{"level":"info","msg":"API Response received is {LastSuccess:1722341604 LastAttempt:1722341604}\n","time":"2024-07-30T12:13:35.136700228Z"}
{"level":"debug","msg":"use pollingFrequency as 60 seconds","time":"2024-07-30T12:13:35.136716529Z"}
{"level":"debug","msg":"last connectivity was  0 sec back, tolerance is 60 sec","time":"2024-07-30T12:13:35.136722825Z"}
{"level":"info","msg":"array SLD-CNSN1 is connected to node csi-node-b2da558a23294137b80bbce669f9eb3a-10.1.55.82","time":"2024-07-30T12:13:35.136731826Z"}
{"level":"info","msg":"ValidateVolumeHostConnectivity reply connected:true messages:\"array SLD-CNSN1 is connected to node csi-node-b2da558a23294137b80bbce669f9eb3a-10.1.55.82\"","time":"2024-07-30T12:13:35.136753126Z"}
{"level":"info","msg":"/podmon.v1.Podmon/ValidateVolumeHostConnectivity: REP 0235: Connected=true, IosInProgress=false, Messages=[array SLD-CNSN1 is connected to node csi-node-b2da558a23294137b80bbce669f9eb3a-10.1.55.82]","time":"2024-07-30T12:13:35.136777865Z"}
{"level":"debug","msg":"RESPONSE: HTTP/1.1 422 Unprocessable Entity Transfer-Encoding: chunked Cache-Control: no-cache Cache-Control: no-store Cache-Control: must-revalidate Cache-Control: max-age=0 Content-Language: en-US Content-Type: application/json Dell-Emc-Token: ****** Expires: -1 X-Content-Type-Options: nosniff  113 {\"messages\":[{\"code\":\"0xE08010080451\",\"severity\":\"Error\",\"message_l10n\":\"New file system can not be created. The limit of 125 file systems for the NAS server 643da243-a9bf-8bc3-3e0c-0e923a9c5317 has been reached.\",\"arguments\":[\"125\",\"643da243-a9bf-8bc3-3e0c-0e923a9c5317\"]}]} 0  \n","time":"2024-07-30T12:13:35.329257697Z"}
{"level":"debug","msg":"release a lock","time":"2024-07-30T12:13:35.329319937Z"}
{"level":"debug","msg":"REQUEST: GET /api/rest/file_system?name=eq.csivol-141c8f9aad\u0026select=description%2Cid%2Cname%2Cnas_server_id%2Cfilesystem_type%2Csize_total%2Csize_used%2Cparent_id HTTP/1.1 Host: 10.1.51.165 Application-Type: CSI Driver for Dell EMC PowerStore/2.7.0 Authorization: ******  ","time":"2024-07-30T12:13:35.32939539Z"}
{"level":"debug","msg":"acquire a lock","time":"2024-07-30T12:13:35.329408858Z"}
{"level":"debug","msg":"RESPONSE: HTTP/1.1 200 OK Content-Length: 2 Cache-Control: no-cache Cache-Control: no-store Cache-Control: must-revalidate Cache-Control: max-age=0 Content-Language: en-US Content-Type: application/json Dell-Emc-Token: ****** Expires: -1 Set-Cookie: auth_cookie=******; Path=/; Secure; HTTPOnly X-Content-Type-Options: nosniff  []\n","time":"2024-07-30T12:13:35.341653795Z"}
{"level":"debug","msg":"release a lock","time":"2024-07-30T12:13:35.34168075Z"}
{"level":"info","msg":"/csi.v1.Controller/CreateVolume: REP 0234: rpc error: code = Internal desc = can't find filesystem 'csivol-141c8f9aad': ","time":"2024-07-30T12:13:35.341704384Z"}
{"level":"info","msg":"/csi.v1.Controller/GetCapacity: REQ 0236: VolumeCapabilities=[mount:\u003c\u003e access_mode:\u003c\u003e ], Parameters=map[FsType:nfs allowRoot:true arrayID:SLD-CNSN1], AccessibleTopology=segments:\u003ckey:\"csi-powerstore.dellemc.com/10.1.51.165-iscsi\" value:\"true\" \u003e segments:\u003ckey:\"csi-powerstore.dellemc.com/10.1.51.165-nfs\" value:\"true\" \u003e , XXX_NoUnkeyedLiteral={}, XXX_sizecache=0","time":"2024-07-30T12:13:35.34239405Z"}
{"level":"debug","msg":"REQUEST: POST /api/rest/metrics/generate?select=physical_total%2Cphysical_used HTTP/1.1 Host: 10.1.51.165 Application-Type: CSI Driver for Dell EMC PowerStore/2.7.0 Authorization: ******  {\"entity\":\"space_metrics_by_cluster\",\"entity_id\":\"0\",\"interval\":\"Five_Mins\"}","time":"2024-07-30T12:13:35.342484151Z"}
{"level":"debug","msg":"acquire a lock","time":"2024-07-30T12:13:35.342526741Z"}
AkshaySainiDell commented 1 month ago

Thanks @a-chartier for sharing the logs. Found below error :

severity\":\"Error\",\"message_l10n\":\"New file system can not be created. The limit of 125 file systems for the NAS server 643da243-a9bf-8bc3-3e0c-0e923a9c5317 has been reached.\"

PowerStore allows 125 file systems per NAS server. Does provisioning work with a different NAS which has not crossed this limit ?

a-chartier commented 1 month ago

@AkshaySainiDell

Interesting, I don't believe that was enforced previously, because I am looking a cluster running 3.2 and it has 167 files systems

This works on 3.2 QA1 image

This does not work on 3.6

QA2 image

Dev image

So what does this look like in terms of configuration to support multiple NAS Servers?

AkshaySainiDell commented 1 month ago

Yes, this changes was introduced in 3.5. PowerStore 3.6.1.2 Release Notes image

If all looks good, let me know if I can close this ticket.

a-chartier commented 1 month ago

@AkshaySainiDell

Before that - Considering the PowerStore configuration is in the following format

arrays:
   - endpoint: "https://10.1.1.1/api/rest"
     globalID: "SLD-CNSN1"
     username: "abc"
     password: "123"
     skipCertificateValidation: true
     isDefault: true
     blockProtocol: "auto"                
     nasName: "POWERSTORE_NAS_1"
     nfsAcls: "0777"

It seems to limit us to one nasName per PowerStore array. what does this look like for multiple nasNames

AkshaySainiDell commented 1 month ago

At this point, CSI-PowerStore supports only one NAS server per array in secret.yaml.

As an alternative, you can utilize the nasName parameter in the storage class by creating multiple storage classes, each configured for a different NAS server. https://github.com/dell/csi-powerstore/blob/1254a31034eafb44090d35cdbb3223e19500b790/samples/storageclass/powerstore-nfs.yaml#L44

a-chartier commented 1 month ago

Okay so let me get this right, You are now enforcing this limit but have not updated the downstream software to support allowing for multiple nasServers ?

That's kind of a problem and creating multiple storageClass's as we keep expanding our array's is a problem.

that being said I have found a number of stale artifacts from before the transition to CSM from the powerstore operator. Cleaning these up has allow us to provision elements at least now.

However this change and the lack of support for multiple nasServers means a re-architecture of our storage classes.

vyshakav-dell commented 1 month ago

Hi @a-chartier, the feature to support multiple NAS servers is part of our backlog and is planned for a future CSM release.

a-chartier commented 1 month ago

@vyshakav-dell

Thanks for the response. Do you have an ETA on that feature by chance? I suspect not but it will help influence design decisions on our part.

This issue can be closed as well then.

Thanks

vyshakav-dell commented 1 month ago

@a-chartier - It will not be part of the next release (CSM 1.12). There are more pressing requirements that we need to deliver for PowerStore CSM before we focus our attention on the multiple NAS server support.

AkshaySainiDell commented 1 month ago

Closing it based on above info.