Cant install on kubernetes v1.22.3

[v0l]

Hi,

One of the CustomResreouces is using a deprecated kind.

https://kubernetes.io/docs/reference/using-api/deprecation-guide/#customresourcedefinition-v122

Error: INSTALLATION FAILED: failed to install CRD crds/hpe-replicated-device-info-crd.yaml: unable to recognize "": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"

[datamattsson]

Hey @v0l this is a know issue with the Helm chart at the moment. It will be fixed in the next version of the upstream chart. We expect it to drop towards end of January. There might be a beta chart before that. In the meantime, deploy using the YAML manifest method.

[v0l]

Thanks @datamattsson I installed directly from the manifest, I have volumes created in TrueNAS but it cant seem to mount them over ISCSI, how does it mount these if the portal doesnt have any targets/luns?

AttachVolume.Attach failed for volume "pvc-a0f0b25a-e987-43ee-b3c2-05cdfca47fa8" : Attach timeout for volume BLK_hpe-csi_pvc-a0f0b25a-e987-43ee-b3c2-05cdfca47fa8

truenas-csp logs:

Tue, 23 Nov 2021 19:48:42 +0000 backend ERROR Not found: Volume with name pvc-a0f0b25a-e987-43ee-b3c2-05cdfca47fa8 not found.
Tue, 23 Nov 2021 19:48:43 +0000 backend INFO Volume created: pvc-a0f0b25a-e987-43ee-b3c2-05cdfca47fa8

csi-provisioner logs:

I1123 19:48:42.588372 1 controller.go:1199] provision "erp/mariadb-data" class "hpe-k8s": started
I1123 19:48:42.595494 1 controller.go:523] CreateVolumeRequest {Name:pvc-a0f0b25a-e987-43ee-b3c2-05cdfca47fa8 CapacityRange:required_bytes:536870912000 VolumeCapabilities:[mount:<fs_type:"xfs" > access_mode:<mode:SINGLE_NODE_WRITER > ] Parameters:map[allowOverrides:sparse,compression,deduplication,volblocksize,sync,description csi.storage.k8s.io/controller-expand-secret-name:truenas-secret csi.storage.k8s.io/controller-expand-secret-namespace:hpe-storage csi.storage.k8s.io/controller-publish-secret-name:truenas-secret csi.storage.k8s.io/controller-publish-secret-namespace:hpe-storage csi.storage.k8s.io/fstype:xfs csi.storage.k8s.io/node-publish-secret-name:truenas-secret csi.storage.k8s.io/node-publish-secret-namespace:hpe-storage csi.storage.k8s.io/node-stage-secret-name:truenas-secret csi.storage.k8s.io/node-stage-secret-namespace:hpe-storage csi.storage.k8s.io/provisioner-secret-name:truenas-secret csi.storage.k8s.io/provisioner-secret-namespace:hpe-storage root:BLK/hpe-csi] Secrets:map[] VolumeContentSource:<nil> AccessibilityRequirements:<nil> XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I1123 19:48:42.595668 1 event.go:281] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"erp", Name:"mariadb-data", UID:"a0f0b25a-e987-43ee-b3c2-05cdfca47fa8", APIVersion:"v1", ResourceVersion:"3119611", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "erp/mariadb-data"
I1123 19:48:42.597416 1 connection.go:182] GRPC call: /csi.v1.Controller/CreateVolume
I1123 19:48:42.597431 1 connection.go:183] GRPC request: {"capacity_range":{"required_bytes":536870912000},"name":"pvc-a0f0b25a-e987-43ee-b3c2-05cdfca47fa8","parameters":{"allowOverrides":"sparse,compression,deduplication,volblocksize,sync,description","root":"BLK/hpe-csi"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"xfs"}},"access_mode":{"mode":1}}]}
I1123 19:48:43.313370 1 connection.go:185] GRPC response: {"volume":{"capacity_bytes":536870912000,"volume_context":{"allowOverrides":"sparse,compression,deduplication,volblocksize,sync,description","compression":"LZ4","deduplication":"OFF","fsType":"xfs","hostEncryption":"false","root":"BLK/hpe-csi","sync":"STANDARD","targetScope":"volume","volblocksize":"8K","volumeAccessMode":"mount"},"volume_id":"BLK_hpe-csi_pvc-a0f0b25a-e987-43ee-b3c2-05cdfca47fa8"}}
I1123 19:48:43.314047 1 connection.go:186] GRPC error: <nil>
I1123 19:48:43.314062 1 controller.go:578] create volume rep: {CapacityBytes:536870912000 VolumeId:BLK_hpe-csi_pvc-a0f0b25a-e987-43ee-b3c2-05cdfca47fa8 VolumeContext:map[allowOverrides:sparse,compression,deduplication,volblocksize,sync,description compression:LZ4 deduplication:OFF fsType:xfs hostEncryption:false root:BLK/hpe-csi sync:STANDARD targetScope:volume volblocksize:8K volumeAccessMode:mount] ContentSource:<nil> AccessibleTopology:[] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I1123 19:48:43.314148 1 controller.go:660] successfully created PV pvc-a0f0b25a-e987-43ee-b3c2-05cdfca47fa8 for PVC mariadb-data and csi volume name BLK_hpe-csi_pvc-a0f0b25a-e987-43ee-b3c2-05cdfca47fa8
I1123 19:48:43.314157 1 controller.go:676] successfully created PV {GCEPersistentDisk:nil AWSElasticBlockStore:nil HostPath:nil Glusterfs:nil NFS:nil RBD:nil ISCSI:nil Cinder:nil CephFS:nil FC:nil Flocker:nil FlexVolume:nil AzureFile:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil PortworxVolume:nil ScaleIO:nil Local:nil StorageOS:nil CSI:&CSIPersistentVolumeSource{Driver:csi.hpe.com,VolumeHandle:BLK_hpe-csi_pvc-a0f0b25a-e987-43ee-b3c2-05cdfca47fa8,ReadOnly:false,FSType:xfs,VolumeAttributes:map[string]string{allowOverrides: sparse,compression,deduplication,volblocksize,sync,description,compression: LZ4,deduplication: OFF,fsType: xfs,hostEncryption: false,root: BLK/hpe-csi,storage.kubernetes.io/csiProvisionerIdentity: 1637691507621-8081-csi.hpe.com,sync: STANDARD,targetScope: volume,volblocksize: 8K,volumeAccessMode: mount,},ControllerPublishSecretRef:&SecretReference{Name:truenas-secret,Namespace:hpe-storage,},NodeStageSecretRef:&SecretReference{Name:truenas-secret,Namespace:hpe-storage,},NodePublishSecretRef:&SecretReference{Name:truenas-secret,Namespace:hpe-storage,},ControllerExpandSecretRef:&SecretReference{Name:truenas-secret,Namespace:hpe-storage,},}}
I1123 19:48:43.314266 1 controller.go:1281] provision "erp/mariadb-data" class "hpe-k8s": volume "pvc-a0f0b25a-e987-43ee-b3c2-05cdfca47fa8" provisioned
I1123 19:48:43.314294 1 controller.go:1298] provision "erp/mariadb-data" class "hpe-k8s": succeeded
I1123 19:48:43.314300 1 volume_store.go:154] Saving volume pvc-a0f0b25a-e987-43ee-b3c2-05cdfca47fa8
I1123 19:48:43.319504 1 volume_store.go:157] Volume pvc-a0f0b25a-e987-43ee-b3c2-05cdfca47fa8 saved
I1123 19:48:43.319561 1 controller.go:1014] Claim processing succeeded, removing PVC a0f0b25a-e987-43ee-b3c2-05cdfca47fa8 from claims in progress
I1123 19:48:43.319625 1 event.go:281] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"erp", Name:"mariadb-data", UID:"a0f0b25a-e987-43ee-b3c2-05cdfca47fa8", APIVersion:"v1", ResourceVersion:"3119611", FieldPath:""}): type: 'Normal' reason: 'ProvisioningSucceeded' Successfully provisioned volume pvc-a0f0b25a-e987-43ee-b3c2-05cdfca47fa8

[v0l]

I switched to csi-driver edge and that seems to get me a few steps further, i have the targets set on TrueNAS now but mount claims they are not found:

MountVolume.MountDevice failed for volume "pvc-98909345-bcb3-4300-ba4a-e8f68743da2a" : rpc error: code = Internal desc = Failed to stage volume BLK_hpe-csi_pvc-98909345-bcb3-4300-ba4a-e8f68743da2a, err: rpc error: code = Internal desc = Error creating device for volume BLK_hpe-csi_pvc-98909345-bcb3-4300-ba4a-e8f68743da2a, err: device not found with serial 6589cfc00000098cb404eaaaa4195ac3 or target

[v0l]

system logs show this repeatedly on the k8s node:

Nov 23 22:42:56 k8-02 kernel: [39991.966089] sd 7:0:0:0: [sdb] Synchronizing SCSI cache
Nov 23 22:42:56 k8-02 kernel: [39992.045023] scsi 7:0:0:0: alua: Detached
Nov 23 22:42:56 k8-02 kernel: [39992.046190] scsi 7:0:0:0: Direct-Access     TrueNAS  iSCSI Disk       0123 PQ: 0 ANSI: 7
Nov 23 22:42:56 k8-02 kernel: [39992.047022] scsi 7:0:0:0: alua: supports implicit TPGS
Nov 23 22:42:56 k8-02 kernel: [39992.047025] scsi 7:0:0:0: alua: device naa.6589cfc00000098cb404eaaaa4195ac3 port group 1 rel port 4
Nov 23 22:42:56 k8-02 kernel: [39992.047172] sd 7:0:0:0: Attached scsi generic sg2 type 0
Nov 23 22:42:56 k8-02 kernel: [39992.048529] sd 7:0:0:0: [sdb] 1048576000 512-byte logical blocks: (537 GB/500 GiB)
Nov 23 22:42:56 k8-02 kernel: [39992.048530] sd 7:0:0:0: [sdb] 8192-byte physical blocks
Nov 23 22:42:56 k8-02 kernel: [39992.048679] sd 7:0:0:0: [sdb] Write Protect is off
Nov 23 22:42:56 k8-02 kernel: [39992.048681] sd 7:0:0:0: [sdb] Mode Sense: 7f 00 10 08
Nov 23 22:42:56 k8-02 kernel: [39992.048991] sd 7:0:0:0: [sdb] Write cache: enabled, read cache: enabled, supports DPO and FUA
Nov 23 22:42:56 k8-02 kernel: [39992.049455] sd 7:0:0:0: [sdb] Optimal transfer size 1048576 bytes
Nov 23 22:42:56 k8-02 kernel: [39992.065424] sd 7:0:0:0: alua: transition timeout set to 60 seconds
Nov 23 22:42:56 k8-02 kernel: [39992.065426] sd 7:0:0:0: alua: port group 01 state A non-preferred supports TolUSNA
Nov 23 22:42:56 k8-02 kernel: [39992.087376] sd 7:0:0:0: [sdb] Attached SCSI disk

Its mounted but then it disappears

[datamattsson]

Hey! Thanks for digging in here. What you're seeing here is that the CSI driver can't discover the disk on the host. It only supports multipath devices today, even if you only have a single IP interface. Do you have any idea what the worker node OS is? You might be missing a couple of mandatory stanzas in your /etc/multipath.conf or the multipathing service isn't running properly.

[v0l]

I couldnt see a problem with my multipath setup, it seems to mount the disk fine on the OS but its not mounted by kubelet on the container.

OS: Ubuntu 20.04.03

[datamattsson]

There's an issue with Ubuntu with no fix in sight. The multipath.conf you have is provided by the stock package and not the HPE CSI Driver. Remove /etc/multipath.conf from the worker nodes and restart the driver:

kubectl rollout restart -n hpe-storage ds/hpe-csi-node

[ishioni]

@datamattsson any news on that fresh version?

[datamattsson]

@ishioni yes! Version 2.1.0 of the HPE CSI Driver will be released next week. I'm hoping to have a fresh cut of the TrueNAS CSP built for this version the following weekend. The multipath issue mentioned in this issue has been addressed too and that will simplify deployment on stock Ubuntu.

[datamattsson]

Please give v2.1.0 a whirl and reopen this bug if problems persist.