search

hpe-storage / csi-driver

A Container Storage Interface (CSI) driver from HPE
https://scod.hpedev.io
Apache License 2.0
55 stars 53 forks source link

CSI Driver failed to mount PV to a Pod!!! #380

Open arupdevops opened 4 months ago

arupdevops commented 4 months ago

Hi Team,

While trying to mount a exsisting PV to a deployment, we are getting the below error.

Please help here where we are making mistake

================================================

17m Warning FailedAttachVolume pod/nginx-arup-7c7d84bf74-gwz5g AttachVolume.Attach failed for volume "pvc-ee5428bb-bae9-42d1-a739-e3d4e499503f" : rpc error: code = Internal desc = Failed to add ACL to volume pvc-ee5428bb-bae9-42d1-a739-e3d4e499503f for node &{ ocp-cp3.cluster.ocp.aspire.infra de78e75c-94e5-21de-7f15-cebdacc7d26f [0xc000920b80] [0xc000920bc0 0xc000920bd0 0xc000920be0 0xc000920bf0] [] } via CSP, err: Request failed with status code 500 and errors Error code (Internal Server Error) and message (VLUN creation error: failed to find any ready target port on array 192.168.236.192)

regards Arup

datamattsson commented 4 months ago

VLUN creation error: failed to find any ready target port on array 192.168.236.192

This is usually a mismatch in protocols. Does the StorageClass have the correct accessProtocol set?

arupdevops commented 4 months ago

Hi

Below is the YAML for the Storage class, Access protocol is ISCSI

=============================

kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: hpe-scod uid: 66d582be-02c2-4c4a-8c2d-206b31a95d01 resourceVersion: '136206' creationTimestamp: '2024-01-29T16:52:21Z' managedFields:

regards Arup

datamattsson commented 4 months ago

If it's iSCSI then my best guess is that you don't have any iSCSI ports on the array. If you check the logs of the CSP, it may reveal more clues. k logs -nhpe-storage deploy/primera3par-csp

arupdevops commented 4 months ago

Thanx, what is the exact command to check the logs "k logs -nhpe-storage deploy/primera3par-csp"?

datamattsson commented 4 months ago

kubectl logs -nhpe-storage deploy/primera3par-csp didn't work?

arupdevops commented 4 months ago

============================================================== time="2024-02-28T07:05:17Z" level=info msg="Response status is 200 OK" file="rest_client.go:72" time="2024-02-28T07:05:17Z" level=info msg="Get ports response is : &{0xc000a4c870 0xc000317380}" file="port_client.go:23" time="2024-02-28T07:05:17Z" level=info msg="=== GET Ports RESP FROM ARRAY: ===\n &{200 OK 200 HTTP/2.0 2 0 map[Cache-Control:[no-cache, no-store, private, max-age=-1, must-revalidate no-cache] Content-Language:[en-US] Content-Security-Policy:[default-src 'self' https://www.google-analytics.com/ ; script-src 'unsafe-eval' 'self' https://www.google-analytics.com/analytics.js 'sha256-v0Q8X6xSNmwWcfldSEWqW+3imFpJurfHYdywm3hw6+w=' ; style-src 'nonce-d29tZSBjb29sIHN0cmluZyB3aWxsIHBvcCB1cCAxMjM=' 'self' blob:; frame-src 'self'; object-src 'self'; media-src 'self' data: blob:; font-src 'self' data:; frame-ancestors 'self'] Content-Type:[application/json] Date:[Wed, 28 Feb 2024 06:37:34 GMT] Pragma:[no-cache no-cache] Server:[hp3par-wsapi] Strict-Transport-Security:[max-age=31536000; includeSubDomains;] Vary:[Accept-Encoding Origin] X-Content-Type-Options:[nosniff] X-Frame-Options:[SAMEORIGIN] X-Xss-Protection:[1; mode=block]] 0xc000327440 -1 [] false true map[] 0xc0006b2600 0xc0001fb600}" file="managed_client.go:214" time="2024-02-28T07:05:17Z" level=info msg="RESULT of get: &{{ } {pvc-02119201-2f72-479b-b7bd-464144ffecf0 bf4e96d1-5464-4ffb-b646-994f0582f5c9 4933799 1 2024-02-09 05:30:43 +0000 UTC map[] map[] [] [] [{main Update storage.hpe.com/v2 2024-02-09 05:30:43 +0000 UTC FieldsV1 {\"f:spec\":{\".\":{},\"f:record\":{\".\":{},\"f:Compression\":{},\"f:Description\":{},\"f:HostEncryption\":{},\"f:HostSeesVLUN\":{},\"f:Id\":{},\"f:Name\":{},\"f:ProvisioningType\":{},\"f:Published\":{},\"f:Size\":{},\"f:isVolumeReplicated\":{}},\"f:uuid\":{}}} }]} {pvc-02119201-2f72-479b-b7bd-464144ffecf0 {pvc-02119201-2f72-479b-b7bd-464144ffecf0 pvc-02119201-2f72-479b-b7bd-464 tpvv false 1073741824 Volume created by the HPE CSI Driver for Kubernetes false false false false }}}" file="crd_utils.go:117" time="2024-02-28T07:05:17Z" level=debug msg="[ REQUEST-ID 102101 ] -- CRD volume metadata found: map[Compression:false Description:Volume created by the HPE CSI Driver for Kubernetes HostEncryption:false HostSeesVLUN:false Id:pvc-02119201-2f72-479b-b7bd-464144ffecf0 Name:pvc-02119201-2f72-479b-b7bd-464 ProvisioningType:tpvv Published:false Size:1073741824 isVolumeReplicated:false]" file="create_vlun_cmd.go:190" time="2024-02-28T07:05:17Z" level=info msg="Appending port 0:3:1 , no env set for fc ports" file="create_vlun_cmd.go:258" time="2024-02-28T07:05:17Z" level=info msg="Appending port 0:3:2 , no env set for fc ports" file="create_vlun_cmd.go:258" time="2024-02-28T07:05:17Z" level=info msg="Appending port 1:3:1 , no env set for fc ports" file="create_vlun_cmd.go:258" time="2024-02-28T07:05:17Z" level=info msg="Appending port 1:3:2 , no env set for fc ports" file="create_vlun_cmd.go:258" time="2024-02-28T07:05:17Z" level=info msg="[ REQUEST-ID 102101 ] -- [Array: 192.168.236.192] LIST OF PORTS []" file="create_vlun_cmd.go:327" time="2024-02-28T07:05:17Z" level=info msg="[ REQUEST-ID 102101 ] -- [Array: 192.168.236.192] TARGET IQNS []" file="create_vlun_cmd.go:329" time="2024-02-28T07:05:17Z" level=info msg="[ REQUEST-ID 102101 ] -- [Array: 192.168.236.192] DISCOVERY IPS []" file="create_vlun_cmd.go:330" time="2024-02-28T07:05:17Z" level=info msg="[ REQUEST-ID 102101 ] -- [Array: 192.168.236.192] Checking If volume pvc-02119201-2f72-479b-b7bd-464 has already existing LUNs associated with host ocp-cp2" file="create_vlun_cmd.go:333" time="2024-02-28T07:05:17Z" level=info msg="Get session returning existing session for user 3paradm on array 192.168.236.192" file="managed_client.go:169" time="2024-02-28T07:05:17Z" level=info msg="Query luns string: /vluns?query=%22volumeName%20EQ%20pvc-02119201-2f72-479b-b7bd-464%22" file="vlun_client.go:82" time="2024-02-28T07:05:17Z" level=trace msg="Header: {X-HP3PAR-WSAPI-SessionKey : *****}\n" file="utils.go:30" time="2024-02-28T07:05:17Z" level=trace msg="Header: {Content-type : application/json}\n" file="utils.go:32" time="2024-02-28T07:05:17Z" level=info msg="Request: action=GET path=https://192.168.236.192:443/api/v1//vluns?query=%22volumeName%20EQ%20pvc-02119201-2f72-479b-b7bd-464%22" file="http.go:65" time="2024-02-28T07:05:17Z" level=info msg="HTTP Response: {\n \"total\": 0,\n \"members\": []\n }" file="utils.go:46" time="2024-02-28T07:05:17Z" level=info msg="Response status is 200 OK" file="rest_client.go:72" time="2024-02-28T07:05:17Z" level=info msg="=== GET HOST VLUNs RESP FROM ARRAY: ===\n &{200 OK 200 HTTP/2.0 2 0 map[Cache-Control:[no-cache, no-store, private, max-age=-1, must-revalidate no-cache] Content-Language:[en-US] Content-Length:[24] Content-Security-Policy:[default-src 'self' https://www.google-analytics.com/ ; script-src 'unsafe-eval' 'self' https://www.google-analytics.com/analytics.js 'sha256-v0Q8X6xSNmwWcfldSEWqW+3imFpJurfHYdywm3hw6+w=' ; style-src 'nonce-d29tZSBjb29sIHN0cmluZyB3aWxsIHBvcCB1cCAxMjM=' 'self' blob:; frame-src 'self'; object-src 'self'; media-src 'self' data: blob:; font-src 'self' data:; frame-ancestors 'self'] Content-Type:[application/json] Date:[Wed, 28 Feb 2024 06:37:34 GMT] Pragma:[no-cache no-cache] Server:[hp3par-wsapi] Strict-Transport-Security:[max-age=31536000; includeSubDomains;] Vary:[Accept-Encoding Origin] X-Content-Type-Options:[nosniff] X-Frame-Options:[SAMEORIGIN] X-Xss-Protection:[1; mode=block]] {0xc0006ec180} 24 [] false false map[] 0xc0006b2000 0xc0001fa580}" file="managed_client.go:638" time="2024-02-28T07:05:17Z" level=error msg="[ REQUEST-ID 102101 ] -- VLUN creation error: failed to find any ready target port on array 192.168.236.192" file="create_vlun_cmd.go:364" time="2024-02-28T07:05:17Z" level=info msg="Raising panic with following message: VLUN creation error: failed to find any ready target port on array 192.168.236.192" file="utils.go:166" time="2024-02-28T07:05:17Z" level=info msg="[ REQUEST-ID 102101 ] -- <<<<< Create VLUN Cmd: [Array: 192.168.236.192] - Host ocp-cp2 and Volume pvc-02119201-2f72-479b-b7bd-464144ffecf0" file="panic.go:884" time="2024-02-28T07:05:17Z" level=trace msg="Releasing mutex lock for PublishVolume:pvc-02119201-2f72-479b-b7bd-464144ffecf0:6bf9b0b6-3886-d3ac-e443-3286cd7302ab" file="concurrent.go:67" time="2024-02-28T07:05:17Z" level=info msg="[ REQUEST-ID 102101 ] -- <<<<< Publish Volume Cmd for volume pvc-02119201-2f72-479b-b7bd-464144ffecf0 and host 6bf9b0b6-3886-d3ac-e443-3286cd7302ab" file="panic.go:884" time="2024-02-28T07:05:17Z" level=info msg="CspManager: Recovering from panic..." file="csp_manager.go:42" time="2024-02-28T07:05:17Z" level=info msg="Sending following message to CSI driver: VLUN creation error: failed to find any ready target port on array 192.168.236.192" file="utils.go:173" time="2024-02-28T07:05:17Z" level=info msg="Error response to be sent is &models.ErrorsPayload{Errors:[]models.ErrorObject{models.ErrorObject{Code:\"Internal Server Error\", Message:\"VLUN creation error: failed to find any ready target port on array 192.168.236.192\"}}}" file="utils.go:181" [root@ocp-svc ~]#

datamattsson commented 4 months ago 
time="2024-02-28T07:05:17Z" level=info msg="[ REQUEST-ID 102101 ] -- [Array: 192.168.236.192] LIST OF PORTS []" file="create_vlun_cmd.go:327"
time="2024-02-28T07:05:17Z" level=info msg="[ REQUEST-ID 102101 ] -- [Array: 192.168.236.192] TARGET IQNS []" file="create_vlun_cmd.go:329"
time="2024-02-28T07:05:17Z" level=info msg="[ REQUEST-ID 102101 ] -- [Array: 192.168.236.192] DISCOVERY IPS []" file="create_vlun_cmd.go:330"

Doesn't look like the CSP can find any iSCSI ports.

arupdevops commented 4 months ago

just now deploy a PV with fc protocol but mounting again got failed. Attached yaml for fc

kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: hpe-scod-fc uid: d7ffeed6-bb09-4d00-8d8e-d228d1345600 resourceVersion: '14910417' creationTimestamp: '2024-02-28T06:47:22Z' managedFields:

arupdevops commented 4 months ago

pvc_mount_issue

datamattsson commented 4 months ago

Is the worker node zoned (since you're switching to FC) with the array?

arupdevops commented 4 months ago

Yes...all are zoned!!!

Redhat OCP-----3 Controller & 2 Worker nodes(All VMs from the Ovirt Engine Infra). Ovirts Physical hosts are properly zoned with Storage controller

arupdevops commented 4 months ago

storage_pvc_mount_error

We have found something & checking on that, will update!!!

datamattsson commented 4 months ago

Redhat OCP-----3 Controller & 2 Worker nodes(All VMs from the Ovirt Engine Infra). Ovirts Physical hosts are properly zoned with Storage controller

This won't work, you have to use the oVirt CSI driver for persistent storage if the workers are VMs. The HPE CSI Driver only works on virtual machines when using iSCSI.

arupdevops commented 4 months ago

The HPE CSI Driver only works on virtual machines when using iSCSI--------While using SCSI, as you mentioned it is unable to find any scsi port in the array.

The backend storage is HPE Primera Storage.

datamattsson commented 4 months ago

The array needs iSCSI cards and have those configured. You also need two vNICs on the oVirt VMs in ideally two different subnets shared with your array iSCSI interfaces.

arupdevops commented 4 months ago

@datamattsson : Thanks for the support. We are checking on the iSCSI part. As of now we don't see any iscsi port on the HPE Primera. On the VLAN part, we have allowed traffic for inter VLANa already.

arupdevops commented 4 months ago

Hi logged a case with HPE for Primera, so they checking the same on the iSCSI part,I have a confusion here no where in documents it is mentioned that to use access protocol iSCSI,the supported Primera storage should be iSCSI configured,also how to configure all these steps,please help with respect to CSI driver

datamattsson commented 4 months ago

The documentation for the 3PAR CSP (Alletra 9K/MP, Primera etc): https://scod.hpedev.io/container_storage_provider/hpe_alletra_9000/#storageclass_parameters

arupdevops commented 4 months ago

FC should work as we have FC ports available in the physical storage & by default CSI driver will use the available FC ports.

datamattsson commented 4 months ago

The HPE CSI Driver does not support virtual FC HPAs (NPIV). Run OpenShift on bare-metal instead and FC will work.

arupdevops commented 4 months ago

Can you please help us with the ovirt csi operator link, as we are unable to find any good resource on that. Appreciate for your extending support here....