search

NetApp / trident

Storage orchestrator for containers
Apache License 2.0
760 stars 222 forks source link

can not mount iSCSI PVs after upgrading to 21.01.0 #523

Closed GyeonSeon closed 3 years ago

GyeonSeon commented 3 years ago

Describe the bug After upgrading to 21.01.0 from 20.10.0, iSCSI volumes can not be mounted.

$ k -n harbor-prod describe pod harbor-postgresql-postgresql-primary-0
...
Events:
  Type     Reason       Age    From               Message
  ----     ------       ----   ----               -------
  Normal   Scheduled    2m10s  default-scheduler  Successfully assigned harbor-prod/harbor-postgresql-postgresql-primary-0 to knode001
  Warning  FailedMount  12s    kubelet            MountVolume.MountDevice failed for volume "pvc-f6a53352-f529-434e-a2ea-5bb4e0bec0f1" : rpc error: code = Internal desc = iSCSI session error: failed to parse node list: "10.251.178.29:3260,-1 iqn.1992-08.com.netapp:sn.4788f617034911e9865d00a098d10258:vs.3"
  Warning  FailedMount  7s     kubelet            Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[custom-init-scripts dshm data default-token-4zz24]: timed out waiting for the condition

$ k -n trident-prod logs trident-csi-66kdv trident-main
...
time="2021-02-08T06:21:11Z" level=error msg="Failed to parse node list" linenum=1 output="10.251.178.29:3260,-1 iqn.1992-08.com.netapp:sn.4788f617034911e9865d00a098d10258:vs.3\n10.251.178.28:3260,-1 iqn.1992-08.com.netapp:sn.4788f617034911e9865d00a098d10258:vs.3\n" requestID=074cbebc-6959-4d7a-83a5-78909e99333c requestSource=CSI
time="2021-02-08T06:21:11Z" level=error msg="GRPC error: rpc error: code = Internal desc = iSCSI session error: failed to parse node list: \"10.251.178.29:3260,-1 iqn.1992-08.com.netapp:sn.4788f617034911e9865d00a098d10258:vs.3\"" requestID=074cbebc-6959-4d7a-83a5-78909e99333c requestSource=CSI

$ k describe pv pvc-f6a53352-f529-434e-a2ea-5bb4e0bec0f1
Name:            pvc-f6a53352-f529-434e-a2ea-5bb4e0bec0f1
Labels:          <none>
Annotations:     pv.kubernetes.io/provisioned-by: csi.trident.netapp.io
Finalizers:      [kubernetes.io/pv-protection external-attacher/csi-trident-netapp-io]
StorageClass:    san-ssd1-stage
Status:          Bound
Claim:           harbor-prod/data-harbor-postgresql-postgresql-primary-0
Reclaim Policy:  Delete
Access Modes:    RWO
VolumeMode:      Filesystem
Capacity:        8Gi
Node Affinity:   <none>
Message:         
Source:
    Type:              CSI (a Container Storage Interface (CSI) volume source)
    Driver:            csi.trident.netapp.io
    FSType:            ext4
    VolumeHandle:      pvc-f6a53352-f529-434e-a2ea-5bb4e0bec0f1
    ReadOnly:          false
    VolumeAttributes:      backendUUID=a8534943-340d-4701-b78e-66720e5721c6
                           internalName=k8s_t01_pvc_f6a53352_f529_434e_a2ea_5bb4e0bec0f1
                           name=pvc-f6a53352-f529-434e-a2ea-5bb4e0bec0f1
                           protocol=block
                           storage.kubernetes.io/csiProvisionerIdentity=1605833967849-8081-csi.trident.netapp.io
Events:                <none>

$ k -n trident-prod describe tridentvolumes pvc-f6a53352-f529-434e-a2ea-5bb4e0bec0f1
Name:          pvc-f6a53352-f529-434e-a2ea-5bb4e0bec0f1
Namespace:     trident-prod
Labels:        <none>
Annotations:   <none>
API Version:   trident.netapp.io/v1
Backend UUID:  a8534943-340d-4701-b78e-66720e5721c6
Config:
  Access Information:
    Iscsi Igroup:      k8s_t01
    Iscsi Lun Serial:  80B1J]MpNvgj
    Iscsi Portals:
      10.251.178.29
    Iscsi Target Iqn:            iqn.1992-08.com.netapp:sn.4788f617034911e9865d00a098d10258:vs.3
    Iscsi Target Portal:         10.251.178.28
  Access Mode:                   ReadWriteOnce
  Block Size:                    
  Clone Source Snapshot:         
  Clone Source Volume:           
  Clone Source Volume Internal:  
  Encryption:                    
  File System:                   ext4
  Internal Name:                 k8s_t01_pvc_f6a53352_f529_434e_a2ea_5bb4e0bec0f1
  Name:                          pvc-f6a53352-f529-434e-a2ea-5bb4e0bec0f1
  Protocol:                      block
  Security Style:                
  Size:                          8589934592
  Space Reserve:                 
  Split On Clone:                
  Storage Class:                 san-ssd1-stage
  Version:                       1
  Volume Mode:                   Filesystem
Kind:                            TridentVolume
Metadata:
  Creation Timestamp:  2020-11-20T01:10:42Z
  Finalizers:
    trident.netapp.io
  Generation:  1
  Managed Fields:
    API Version:  trident.netapp.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:backendUUID:
      f:config:
        .:
        f:accessInformation:
          .:
          f:iscsiIgroup:
          f:iscsiLunSerial:
          f:iscsiPortals:
          f:iscsiTargetIqn:
          f:iscsiTargetPortal:
        f:accessMode:
        f:blockSize:
        f:cloneSourceSnapshot:
        f:cloneSourceVolume:
        f:cloneSourceVolumeInternal:
        f:encryption:
        f:fileSystem:
        f:internalName:
        f:name:
        f:protocol:
        f:securityStyle:
        f:size:
        f:spaceReserve:
        f:splitOnClone:
        f:storageClass:
        f:version:
        f:volumeMode:
      f:metadata:
        f:finalizers:
          .:
          v:"trident.netapp.io":
      f:orphaned:
      f:pool:
      f:state:
    Manager:         trident_orchestrator
    Operation:       Update
    Time:            2020-11-20T01:10:42Z
  Resource Version:  5312
  UID:               9fcda0bc-24b0-40df-adc3-65e29b410d08
Orphaned:            false
Pool:                aggr_ssd21
State:               online
Events:              <none>

$ k -n trident-prod describe tridentbackends tbe-w5jtx
Name:          tbe-w5jtx
Namespace:     trident-prod
Labels:        <none>
Annotations:   <none>
API Version:   trident.netapp.io/v1
Backend Name:  san-pax2b
Backend UUID:  a8534943-340d-4701-b78e-66720e5721c6
Config:
  ontap_config:
    Aggregate:  
    Auto Export CID Rs:
      0.0.0.0/0
      ::/0
    Auto Export Policy:            false
    Backend Name:                  san-pax2b
    Chap Initiator Secret:         
    Chap Target Initiator Secret:  
    Chap Target Username:          
    Chap Username:                 
    Data LIF:                      10.251.178.29
    Debug:                         false
    Debug Trace Flags:             <nil>
    Defaults:
      Encryption:                          false
      Export Policy:                       default
      File System Type:                    ext4
      Security Style:                      unix
      Size:                                1G
      Snapshot Dir:                        false
      Snapshot Policy:                     none
      Snapshot Reserve:                    0
      Space Allocation:                    true
      Space Reserve:                       none
      Split On Clone:                      false
      Tiering Policy:                      
      Unix Permissions:                    ---rwxrwxrwx
    Disable Delete:                        false
    Empty Flexvol Deferred Delete Period:  
    Igroup Name:                           k8s_t01
    Labels:                                <nil>
    Limit Aggregate Usage:                 95%
    Limit Volume Size:                     
    Management LIF:                        10.251.178.21
    Nfs Mount Options:                     
    Password:                              secret:tbe-a8534943-340d-4701-b78e-66720e5721c6
    Qtree Prune Flexvols Period:           
    Qtree Quota Resize Period:             
    Qtrees Per Flexvol:                    
    Region:                                
    Serial Numbers:
      941748000002
      941748000004
    Storage:               <nil>
    Storage Driver Name:   ontap-san
    Storage Prefix:        k8s_t01_
    Supported Topologies:  <nil>
    Svm:                   vpax2
    Usage Heartbeat:       
    Use CHAP:              false
    Username:              secret:tbe-a8534943-340d-4701-b78e-66720e5721c6
    Version:               1
    Zone:                  
Kind:                      TridentBackend
Metadata:
  Creation Timestamp:  2020-11-20T01:01:47Z
  Finalizers:
    trident.netapp.io
  Generate Name:  tbe-
  Generation:     1
  Managed Fields:
    API Version:  trident.netapp.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:backendName:
      f:backendUUID:
      f:config:
        .:
        f:ontap_config:
          .:
          f:aggregate:
          f:autoExportCIDRs:
          f:autoExportPolicy:
          f:backendName:
          f:chapInitiatorSecret:
          f:chapTargetInitiatorSecret:
          f:chapTargetUsername:
          f:chapUsername:
          f:dataLIF:
          f:debug:
          f:debugTraceFlags:
          f:defaults:
            .:
            f:encryption:
            f:exportPolicy:
            f:fileSystemType:
            f:securityStyle:
            f:size:
            f:snapshotDir:
            f:snapshotPolicy:
            f:snapshotReserve:
            f:spaceAllocation:
            f:spaceReserve:
            f:splitOnClone:
            f:tieringPolicy:
            f:unixPermissions:
          f:disableDelete:
          f:emptyFlexvolDeferredDeletePeriod:
          f:igroupName:
          f:labels:
          f:limitAggregateUsage:
          f:limitVolumeSize:
          f:managementLIF:
          f:nfsMountOptions:
          f:password:
          f:qtreePruneFlexvolsPeriod:
          f:qtreeQuotaResizePeriod:
          f:qtreesPerFlexvol:
          f:region:
          f:serialNumbers:
          f:storage:
          f:storageDriverName:
          f:storagePrefix:
          f:supportedTopologies:
          f:svm:
          f:usageHeartbeat:
          f:useCHAP:
          f:username:
          f:version:
          f:zone:
      f:metadata:
        f:finalizers:
          .:
          v:"trident.netapp.io":
        f:generateName:
      f:online:
      f:state:
      f:version:
    Manager:         trident_orchestrator
    Operation:       Update
    Time:            2020-11-20T01:01:47Z
  Resource Version:  3578
  UID:               782497ac-2941-4743-aaf1-c25581c9e8de
Online:              true
State:               online
Version:             1
Events:              <none>

Environment Trident version: 20.10.0 -> 21.01.0 Container runtime : docker://20.10.3 Kubernetes version: 1.20.2 Kubernetes orchestrator: self managed cluster on-prem OS : Ubuntu 20.04.2 LTS NetApp backend types: ONTAP SAN

gnarl commented 3 years ago

@GyeonSeon,

The team is working on a fix for this issue. We will update this issue when the fix is available.

GyeonSeon commented 3 years ago

This issue is now fixed in the Trident v21.01.1 release. I tested the new release on kubernetes 1.19.x. and 1.20.x. Finally, there is no problems.

Thank you for your efforts.