IOPS parameter behavior does not match its description

scaleoutsean commented 5 years ago

Trident 19.07
OS: Ubuntu 18.04
Docker: 18.06.2-ce
Kubernetes: v1.13.9
Element OS: 11.3 (VM)

1) solidfire-san backend:

    "Types": [{"Type": "bronze", "Qos": {"minIOPS": 100, "maxIOPS": 200, "burstIOPS": 400}},
              {"Type": "silver", "Qos": {"minIOPS": 400, "maxIOPS": 600, "burstIOPS": 800}},
              {"Type": "gold", "Qos": {"minIOPS": 600, "maxIOPS": 800, "burstIOPS": 1000}}]

2) Case A - create a storage class "solidfire-gold-xfs" (I forgot to change fsType but lets ignore that for a second):

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: solidfire-gold-xfs
provisioner: netapp.io/trident
parameters:
  backendType: "solidfire-san"
  IOPS: "555"
  fsType: "ext4"```

Now a Stateful set created to make use of "solidfire-gold-xfs" storage class creates PVCs that result in "silver" QoS on Element storage. scaled to 2, maybe it'd get random after that.

3) Case B - create a SC "gold" which should supposedly map exactly to "gold" in backend.json:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gold
provisioner: netapp.io/trident
parameters:
  backendType: "solidfire-san"
  fsType: "xfs"

Create a stateful set with 1 replica - the first volume gets bronze QoS. Scale it to 2, the next volume gets a gold, scale it to 3, the next volume gets a bronze, scale it to 4, it's a silver.

**sean@master:~/trident-installer$ kubectl describe sc gold**
Name:                  gold
IsDefaultClass:        No
Annotations:           <none>
Provisioner:           netapp.io/trident
Parameters:            backendType=solidfire-san,fsType=xfs
AllowVolumeExpansion:  <unset>
MountOptions:          <none>
ReclaimPolicy:         Delete
VolumeBindingMode:     Immediate
Events:                <none>

**sean@master:~/trident-installer$ kubectl describe sc gold-xfs**
Name:                  gold-xfs
IsDefaultClass:        No
Annotations:           <none>
Provisioner:           netapp.io/trident
Parameters:            IOPS=555,backendType=solidfire-san,fsType=xfs
AllowVolumeExpansion:  <unset>
MountOptions:          <none>
ReclaimPolicy:         Delete
VolumeBindingMode:     Immediate
Events:                <none>

**sean@master:~/trident-installer$ kubectl describe pvc**
Name:          www-web-0
Namespace:     default
StorageClass:  gold
Status:        Bound
Volume:        default-www-web-0-3d6f9
Labels:        app=nginx
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: netapp.io/trident
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      1073741824
Access Modes:  RWO
VolumeMode:    Filesystem
Events:
  Type       Reason                Age                From                         Message
  ----       ------                ----               ----                         -------
  Normal     ExternalProvisioning  13m (x2 over 13m)  persistentvolume-controller  waiting for a volume to be created, either by external provisioner "netapp.io/trident" or manually created by system administrator
  Normal     ProvisioningSuccess   13m                netapp.io/trident            provisioned a volume and a PV for the PVC.
Mounted By:  web-0

Name:          www-web-1
Namespace:     default
StorageClass:  gold
Status:        Bound
Volume:        default-www-web-1-f5ca4
Labels:        app=nginx
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: netapp.io/trident
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      1073741824
Access Modes:  RWO
VolumeMode:    Filesystem
Events:
  Type       Reason                Age                    From                         Message
  ----       ------                ----                   ----                         -------
  Normal     ExternalProvisioning  8m48s (x2 over 8m48s)  persistentvolume-controller  waiting for a volume to be created, either by external provisioner "netapp.io/trident" or manually created by system administrator
  Normal     ProvisioningSuccess   8m47s                  netapp.io/trident            provisioned a volume and a PV for the PVC.
Mounted By:  web-1

Name:          www-web-2
Namespace:     default
StorageClass:  gold
Status:        Bound
Volume:        default-www-web-2-71736
Labels:        app=nginx
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: netapp.io/trident
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      1073741824
Access Modes:  RWO
VolumeMode:    Filesystem
Events:
  Type       Reason                Age                    From                         Message
  ----       ------                ----                   ----                         -------
  Normal     ExternalProvisioning  5m20s (x2 over 5m20s)  persistentvolume-controller  waiting for a volume to be created, either by external provisioner "netapp.io/trident" or manually created by system administrator
  Normal     ProvisioningSuccess   5m20s                  netapp.io/trident            provisioned a volume and a PV for the PVC.
Mounted By:  web-2

Name:          www-web-3
Namespace:     default
StorageClass:  gold
Status:        Bound
Volume:        default-www-web-3-9b0fd
Labels:        app=nginx
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: netapp.io/trident
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      1073741824
Access Modes:  RWO
VolumeMode:    Filesystem
Events:
  Type       Reason                Age    From                         Message
  ----       ------                ----   ----                         -------
  Normal     ExternalProvisioning  3m59s  persistentvolume-controller  waiting for a volume to be created, either by external provisioner "netapp.io/trident" or manually created by system administrator
  Normal     ProvisioningSuccess   3m58s  netapp.io/trident            provisioned a volume and a PV for the PVC.
Mounted By:  web-3

**sean@master:~/trident-installer$ kubectl describe pv**
Name:            default-www-web-0-3d6f9
Labels:          <none>
Annotations:     pv.kubernetes.io/provisioned-by: netapp.io/trident
                 volume.beta.kubernetes.io/storage-class: gold
Finalizers:      [kubernetes.io/pv-protection]
StorageClass:    gold
Status:          Bound
Claim:           default/www-web-0
Reclaim Policy:  Delete
Access Modes:    RWO
VolumeMode:      Filesystem
Capacity:        1073741824
Node Affinity:   <none>
Message:         
Source:
    Type:               ISCSI (an ISCSI Disk resource that is attached to a kubelet's host machine and then exposed to the pod)
    TargetPortal:       10.0.0.6:3260
    IQN:                iqn.2010-01.com.solidfire:pjoj.default-www-web-0-3d6f9.35
    Lun:                0
    ISCSIInterface      default
    FSType:             xfs
    ReadOnly:           false
    Portals:            []
    DiscoveryCHAPAuth:  true
    SessionCHAPAuth:    true
    SecretRef:          &SecretReference{Name:trident-chap-a3ebdf14-a28f-4b1b-89e1-d273de53c35b-trident,Namespace:trident,}
    InitiatorName:      <none>
Events:                 <none>

Name:            default-www-web-1-f5ca4
Labels:          <none>
Annotations:     pv.kubernetes.io/provisioned-by: netapp.io/trident
                 volume.beta.kubernetes.io/storage-class: gold
Finalizers:      [kubernetes.io/pv-protection]
StorageClass:    gold
Status:          Bound
Claim:           default/www-web-1
Reclaim Policy:  Delete
Access Modes:    RWO
VolumeMode:      Filesystem
Capacity:        1073741824
Node Affinity:   <none>
Message:         
Source:
    Type:               ISCSI (an ISCSI Disk resource that is attached to a kubelet's host machine and then exposed to the pod)
    TargetPortal:       10.0.0.6:3260
    IQN:                iqn.2010-01.com.solidfire:pjoj.default-www-web-1-f5ca4.36
    Lun:                0
    ISCSIInterface      default
    FSType:             xfs
    ReadOnly:           false
    Portals:            []
    DiscoveryCHAPAuth:  true
    SessionCHAPAuth:    true
    SecretRef:          &SecretReference{Name:trident-chap-a3ebdf14-a28f-4b1b-89e1-d273de53c35b-trident,Namespace:trident,}
    InitiatorName:      <none>
Events:                 <none>

Name:            default-www-web-2-71736
Labels:          <none>
Annotations:     pv.kubernetes.io/provisioned-by: netapp.io/trident
                 volume.beta.kubernetes.io/storage-class: gold
Finalizers:      [kubernetes.io/pv-protection]
StorageClass:    gold
Status:          Bound
Claim:           default/www-web-2
Reclaim Policy:  Delete
Access Modes:    RWO
VolumeMode:      Filesystem
Capacity:        1073741824
Node Affinity:   <none>
Message:         
Source:
    Type:               ISCSI (an ISCSI Disk resource that is attached to a kubelet's host machine and then exposed to the pod)
    TargetPortal:       10.0.0.6:3260
    IQN:                iqn.2010-01.com.solidfire:pjoj.default-www-web-2-71736.37
    Lun:                0
    ISCSIInterface      default
    FSType:             xfs
    ReadOnly:           false
    Portals:            []
    DiscoveryCHAPAuth:  true
    SessionCHAPAuth:    true
    SecretRef:          &SecretReference{Name:trident-chap-a3ebdf14-a28f-4b1b-89e1-d273de53c35b-trident,Namespace:trident,}
    InitiatorName:      <none>
Events:                 <none>

Name:            default-www-web-3-9b0fd
Labels:          <none>
Annotations:     pv.kubernetes.io/provisioned-by: netapp.io/trident
                 volume.beta.kubernetes.io/storage-class: gold
Finalizers:      [kubernetes.io/pv-protection]
StorageClass:    gold
Status:          Bound
Claim:           default/www-web-3
Reclaim Policy:  Delete
Access Modes:    RWO
VolumeMode:      Filesystem
Capacity:        1073741824
Node Affinity:   <none>
Message:         
Source:
    Type:               ISCSI (an ISCSI Disk resource that is attached to a kubelet's host machine and then exposed to the pod)
    TargetPortal:       10.0.0.6:3260
    IQN:                iqn.2010-01.com.solidfire:pjoj.default-www-web-3-9b0fd.38
    Lun:                0
    ISCSIInterface      default
    FSType:             xfs
    ReadOnly:           false
    Portals:            []
    DiscoveryCHAPAuth:  true
    SessionCHAPAuth:    true
    SecretRef:          &SecretReference{Name:trident-chap-a3ebdf14-a28f-4b1b-89e1-d273de53c35b-trident,Namespace:trident,}
    InitiatorName:      <none>
Events:                 <none>

Element OS volumes:

VolumeID                    : 35
Name                        : default-www-web-0-3d6f9
AccountID                   : 1
CreateTime                  : 2019-09-01T13:05:58Z
VolumeConsistencyGroupUUID  : d3758c99-779d-4537-8b0c-cc364f8b60d0
VolumeUUID                  : 502d89e6-0abc-4cf1-b118-33cf7d79719c
EnableSnapMirrorReplication : False
Status                      : active
Access                      : readWrite
Enable512e                  : True
Iqn                         : iqn.2010-01.com.solidfire:pjoj.default-www-web-0-3d6f9.35
ScsiEUIDeviceID             : 706a6f6a00000023f47acc0100000000
ScsiNAADeviceID             : 6f47acc100000000706a6f6a00000023
Qos                         : {"MinIOPS" = 100, "MaxIOPS" = 200, "BurstIOPS" = 400, "BurstTime" = 60}
QosPolicyID                 : 
VolumeAccessGroups          : {}
VolumePairs                 : {}
DeleteTime                  : 
PurgeTime                   : 
LastAccessTime              : 2019-09-01T13:06:02Z
LastAccessTimeIO            : 
SliceCount                  : 1
TotalSize                   : 1073741824
BlockSize                   : 4096
VirtualVolumeID             : 
Attributes                  : {trident, docker-name, fstype}

VolumeID                    : 36
Name                        : default-www-web-1-f5ca4
AccountID                   : 1
CreateTime                  : 2019-09-01T13:11:07Z
VolumeConsistencyGroupUUID  : d8e522ff-a339-475a-a239-af9f9053023d
VolumeUUID                  : ba0c1b82-1b6d-4c3e-8b13-a8e5df288061
EnableSnapMirrorReplication : False
Status                      : active
Access                      : readWrite
Enable512e                  : True
Iqn                         : iqn.2010-01.com.solidfire:pjoj.default-www-web-1-f5ca4.36
ScsiEUIDeviceID             : 706a6f6a00000024f47acc0100000000
ScsiNAADeviceID             : 6f47acc100000000706a6f6a00000024
Qos                         : {"MinIOPS" = 600, "MaxIOPS" = 800, "BurstIOPS" = 1000, "BurstTime" = 60}
QosPolicyID                 : 
VolumeAccessGroups          : {}
VolumePairs                 : {}
DeleteTime                  : 
PurgeTime                   : 
LastAccessTime              : 2019-09-01T13:11:12Z
LastAccessTimeIO            : 
SliceCount                  : 1
TotalSize                   : 1073741824
BlockSize                   : 4096
VirtualVolumeID             : 
Attributes                  : {trident, fstype, docker-name}

VolumeID                    : 37
Name                        : default-www-web-2-71736
AccountID                   : 1
CreateTime                  : 2019-09-01T13:14:35Z
VolumeConsistencyGroupUUID  : a7693b7f-0852-429d-aa42-21adee86cd63
VolumeUUID                  : f1b9dd2b-6e8b-4f48-81f1-b5e60d3a5905
EnableSnapMirrorReplication : False
Status                      : active
Access                      : readWrite
Enable512e                  : True
Iqn                         : iqn.2010-01.com.solidfire:pjoj.default-www-web-2-71736.37
ScsiEUIDeviceID             : 706a6f6a00000025f47acc0100000000
ScsiNAADeviceID             : 6f47acc100000000706a6f6a00000025
Qos                         : {"MinIOPS" = 100, "MaxIOPS" = 200, "BurstIOPS" = 400, "BurstTime" = 60}
QosPolicyID                 : 
VolumeAccessGroups          : {}
VolumePairs                 : {}
DeleteTime                  : 
PurgeTime                   : 
LastAccessTime              : 2019-09-01T13:14:45Z
LastAccessTimeIO            : 
SliceCount                  : 1
TotalSize                   : 1073741824
BlockSize                   : 4096
VirtualVolumeID             : 
Attributes                  : {trident, fstype, docker-name}

VolumeID                    : 38
Name                        : default-www-web-3-9b0fd
AccountID                   : 1
CreateTime                  : 2019-09-01T13:15:56Z
VolumeConsistencyGroupUUID  : a6df792e-8adf-447b-b154-668da573adf9
VolumeUUID                  : 61b84970-e7e6-4e77-ad14-25d1f8e64f66
EnableSnapMirrorReplication : False
Status                      : active
Access                      : readWrite
Enable512e                  : True
Iqn                         : iqn.2010-01.com.solidfire:pjoj.default-www-web-3-9b0fd.38
ScsiEUIDeviceID             : 706a6f6a00000026f47acc0100000000
ScsiNAADeviceID             : 6f47acc100000000706a6f6a00000026
Qos                         : {"MinIOPS" = 400, "MaxIOPS" = 600, "BurstIOPS" = 800, "BurstTime" = 60}
QosPolicyID                 : 
VolumeAccessGroups          : {}
VolumePairs                 : {}
DeleteTime                  : 
PurgeTime                   : 
LastAccessTime              : 2019-09-01T13:16:05Z
LastAccessTimeIO            : 
SliceCount                  : 1
TotalSize                   : 1073741824
BlockSize                   : 4096
VirtualVolumeID             : 
Attributes                  : {trident, fstype, docker-name}

wonderland commented 5 years ago

Your solidfire-gold-xfs storage class defines an IOPS level that falls in the range of the silver definition in the backend (silver ranges from 400 to 600, the sc defines 555). Your gold class does not specify any IOPS level, therefore all classes defined in the backend match. I guess that is why you see that "random" behaviour, there is nothing in the sc that would allow trident to choose a particular type so it spreads them across all.

scaleoutsean commented 5 years ago

solidfire-gold-xfs: I see there's logic to your theory (Trident picks a "range" in which IOPS "roughly" fits in), but IMO that's not how it's supposed to work - https://netapp-trident.readthedocs.io/en/stable-v19.07/kubernetes/concepts/objects.html#kubernetes-storageclass-objects says "Pool is capable of guaranteeing IOPS in this range". But even in that case, such behavior must be documented. I do have a SC with 555+ IOPS (gold, Min 600 IOPS), why doesn't it use that? If it cannot do that, I guess at least 50% of users would prefer to have Trident doesn't deliver those storage resources and then they could edit & resubmit or add needed QoS to my back-end definition (if it's not at all available). The current behavior is undocumented and silently introduces bad behavior - both of which seems wrong.
gold: also, that's a good point, but this randomness completely contradicts expected behavior of Element software. If I had a 3-pod stateful set of Cassandra nodes and Trident sprayed my PVs around all storage classes/SLAs, that would be worse than picking the lowest QoS and sticking with it (because at least all pods from Stateful Set would be equally slow, whereas now it's all over the place which doesn't make the whole set faster but it does waste Min IOPS allocated to all but the slowest Pod).

Your gold class does not specify any IOPS level, therefore all classes defined in the backend match.

I did few other tests (in one, I made the SC with IOPS 555 default SC) and when pod had no SC assignment, Trident would still assign a non-default class to each pod in Stateful Set (all 3 PVs). This is also unexpected. All right, maybe there's also some logic to it (can't meet the default QoS SLA, so let's round-robin around), but I don't like that logic.

innergy commented 5 years ago

Re: your solidfire-gold-xfs storage class, I believe what you're saying is that you want the IOPS parameter to mean "use any pool with minIOPS greather than or equal to this number", whereas today it means "use any pool where this number is between minIOPS and maxIOPS", and therefore silver qualifies rather than gold. I agree that would make more sense, especially when we say "guaranteeing IOPS" in the description. We'll have to figure out if we want to change the behavior or just the meaning to make it more clear.

Re: your gold storage class, that's working as designed. Its definition (with only backendType defined) says "any pool from a solidfire-san backend is viable".

scaleoutsean commented 3 years ago

@innergy it took me a while to figure out why I was extra confused when I reported this - even when there's a default storage class, K8s reports that all PVs use it:

But in reality they do not (this part we knew before); I have 2 classes and 1 in 3 got storage settings QoS from a non-default storage class.

If default SC is set, K8s shows PVCs and PVs are compliant and consistent whereas that's not necessarily true. When there's no default storage class, the behavior may be easier to spot (I haven't verified if in that case K8s would show different storage classes, but even if it did, users would probably be more curious what settings got applied.)

These screenshots are from a more recent release:

Rancher v1.18.12
Trident v20.10
Element OS 12.2

NetApp / trident

IOPS parameter behavior does not match its description #281