pamandrejko
commented
1 year ago These tests were performed on cluster: pk7-aro url: https://console-openshift-console.apps.jce5djuj.eastus.aroapp.io/k8s/cluster/storageclasses az aro list-credentials \ --name pk7-aro --resource-group cpbu-sdlc-rg Contact me for the credentials
When you deploy an ODF storage cluster on ARO, you have to select a "Backing storage type" from the existing storage classes available:
From the screenshot you can see only 2 options are available on ARO:
managed-csimanaged-premium
The ARO ODF documentation says to select managed-premium.
Note: The validation tests always test BOTH RWO and RWX storage classes. It always tests RWO first, followed by RWX. But if the RWO tests FAIL, then the test aborts, and never runs the RWX storage tests.
Test #1
So let's just run the tests with the managed-premium RWO storage class, and remove ODF from the equation:
If I run the storage validation tests against the underlying backing storage managed-premium they PASS,
msg": "######################## MOUNT TESTS PASSED FOR ReadWriteOnce Volume #################################"but fail for the RWX storage. (I expect the RWX tests to fail at this point.) In this case, in params.yml I had specified:
storageClass_ReadWriteMany: managed-csi
storageClass_ReadWriteOnce: managed-premiummanaged-csi is bogus, but I had to specify something to test the managed-premium and the managed-premium storage class tests passed!
run_k8s_storage_test
PLAY [localhost] ***************************************************************
TASK [ocp login using creds] ***************************************************
changed: [localhost]
TASK [ocp login using token] ***************************************************
skipping: [localhost]
TASK [debug] *******************************************************************
ok: [localhost] => {
"login_creds.stdout_lines": [
"Login successful.",
"",
"You have access to 69 projects, the list has been suppressed. You can list all projects with 'oc projects'",
"",
"Using project \"default\".",
"Welcome! See 'oc help' to get started."
]
}
TASK [debug] *******************************************************************
skipping: [localhost]
TASK [Storage Readiness] *******************************************************
TASK [storage-readiness : Create namespace k8s-validation if not present] ******
changed: [localhost]
TASK [storage-readiness : Run simple mount test for ReadWriteOnce] *************
included: /opt/ansible/roles/storage-readiness/tasks/mount-test.yaml for localhost
TASK [storage-readiness : Test mount ReadWriteOnce volumes for readiness] ******
changed: [localhost] => (item={'name': 'create-volume.yaml.j2'})
changed: [localhost] => (item={'name': 'mount-job.yaml.j2'})
TASK [storage-readiness : Verify mount completed for ReadWriteOnce] ************
FAILED - RETRYING: Verify mount completed for ReadWriteOnce (20 retries left).
FAILED - RETRYING: Verify mount completed for ReadWriteOnce (19 retries left).
ok: [localhost]
TASK [storage-readiness : debug] ***********************************************
ok: [localhost] => {
"msg": "######################## MOUNT TESTS PASSED FOR ReadWriteOnce Volume #################################"
}
TASK [storage-readiness : Run simple mount test for ReadWriteMany] *************
included: /opt/ansible/roles/storage-readiness/tasks/mount-test.yaml for localhost
TASK [storage-readiness : Test mount ReadWriteMany volumes for readiness] ******
changed: [localhost] => (item={'name': 'create-volume.yaml.j2'})
changed: [localhost] => (item={'name': 'mount-job.yaml.j2'})
TASK [storage-readiness : Verify mount completed for ReadWriteMany] ************
FAILED - RETRYING: Verify mount completed for ReadWriteMany (20 retries left).
FAILED - RETRYING: Verify mount completed for ReadWriteMany (19 retries left).
FAILED - RETRYING: Verify mount completed for ReadWriteMany (18 retries left).
FAILED - RETRYING: Verify mount completed for ReadWriteMany (17 retries left).
FAILED - RETRYING: Verify mount completed for ReadWriteMany (16 retries left).
FAILED - RETRYING: Verify mount completed for ReadWriteMany (15 retries left).
FAILED - RETRYING: Verify mount completed for ReadWriteMany (14 retries left).
FAILED - RETRYING: Verify mount completed for ReadWriteMany (13 retries left).
FAILED - RETRYING: Verify mount completed for ReadWriteMany (12 retries left).
FAILED - RETRYING: Verify mount completed for ReadWriteMany (11 retries left).
FAILED - RETRYING: Verify mount completed for ReadWriteMany (10 retries left).
FAILED - RETRYING: Verify mount completed for ReadWriteMany (9 retries left).
FAILED - RETRYING: Verify mount completed for ReadWriteMany (8 retries left).
FAILED - RETRYING: Verify mount completed for ReadWriteMany (7 retries left).
FAILED - RETRYING: Verify mount completed for ReadWriteMany (6 retries left).
FAILED - RETRYING: Verify mount completed for ReadWriteMany (5 retries left).
FAILED - RETRYING: Verify mount completed for ReadWriteMany (4 retries left).
FAILED - RETRYING: Verify mount completed for ReadWriteMany (3 retries left).
FAILED - RETRYING: Verify mount completed for ReadWriteMany (2 retries left).
FAILED - RETRYING: Verify mount completed for ReadWriteMany (1 retries left).
fatal: [localhost]: FAILED! => {"attempts": 20, "changed": false, "resources": [{"apiVersion": "batch/v1", "kind": "Job", "metadata": {"creationTimestamp": "2022-12-06T20:20:15Z", "generation": 1, "labels": {"controller-uid": "65ea6529-81ec-4ec4-a40e-c5412ad4d69d", "job-name": "readiness-mount-job-readwritemany"}, "managedFields": [{"apiVersion": "batch/v1", "fieldsType": "FieldsV1", "fieldsV1": {"f:spec": {"f:backoffLimit": {}, "f:completionMode": {}, "f:completions": {}, "f:parallelism": {}, "f:suspend": {}, "f:template": {"f:metadata": {"f:name": {}}, "f:spec": {"f:containers": {"k:{\"name\":\"mount-job\"}": {".": {}, "f:args": {}, "f:command": {}, "f:image": {}, "f:imagePullPolicy": {}, "f:name": {}, "f:resources": {}, "f:terminationMessagePath": {}, "f:terminationMessagePolicy": {}, "f:volumeMounts": {".": {}, "k:{\"mountPath\":\"/mnt/data\"}": {".": {}, "f:mountPath": {}, "f:name": {}}}}}, "f:dnsPolicy": {}, "f:restartPolicy": {}, "f:schedulerName": {}, "f:securityContext": {}, "f:terminationGracePeriodSeconds": {}, "f:volumes": {".": {}, "k:{\"name\":\"tmp-store\"}": {".": {}, "f:name": {}, "f:persistentVolumeClaim": {".": {}, "f:claimName": {}}}}}}}}, "manager": "OpenAPI-Generator", "operation": "Update", "time": "2022-12-06T20:20:15Z"}, {"apiVersion": "batch/v1", "fieldsType": "FieldsV1", "fieldsV1": {"f:status": {"f:active": {}, "f:startTime": {}}}, "manager": "kube-controller-manager", "operation": "Update", "subresource": "status", "time": "2022-12-06T20:20:15Z"}], "name": "readiness-mount-job-readwritemany", "namespace": "k8s-validation", "resourceVersion": "48001", "uid": "65ea6529-81ec-4ec4-a40e-c5412ad4d69d"}, "spec": {"backoffLimit": 6, "completionMode": "NonIndexed", "completions": 1, "parallelism": 1, "selector": {"matchLabels": {"controller-uid": "65ea6529-81ec-4ec4-a40e-c5412ad4d69d"}}, "suspend": false, "template": {"metadata": {"creationTimestamp": null, "labels": {"controller-uid": "65ea6529-81ec-4ec4-a40e-c5412ad4d69d", "job-name": "readiness-mount-job-readwritemany"}, "name": "mount-job"}, "spec": {"containers": [{"args": ["-c", "echo \"volume mounted\";echo \"writing data\" > /mnt/data/readiness"], "command": ["sh"], "image": "quay.io/centos/amd64", "imagePullPolicy": "IfNotPresent", "name": "mount-job", "resources": {}, "terminationMessagePath": "/dev/termination-log", "terminationMessagePolicy": "File", "volumeMounts": [{"mountPath": "/mnt/data", "name": "tmp-store"}]}], "dnsPolicy": "ClusterFirst", "restartPolicy": "Never", "schedulerName": "default-scheduler", "securityContext": {}, "terminationGracePeriodSeconds": 30, "volumes": [{"name": "tmp-store", "persistentVolumeClaim": {"claimName": "readiness-readwritemany"}}]}}}, "status": {"active": 1, "startTime": "2022-12-06T20:20:15Z"}}]}
PLAY RECAP *********************************************************************
localhost : ok=9 changed=4 unreachable=0 failed=1 skipped=2 rescued=0 ignored=0 Test #2
I followed these instructions to deploy a new NFS backed POSIX compliant azure-file storage (currently in Tech Preview).
This time I run the storage validation tests with:
storageClass_ReadWriteMany: azure-file
storageClass_ReadWriteOnce: managed-premiumAnd this time the RWO tests PASS,
"msg": "######################## MOUNT TESTS PASSED FOR ReadWriteOnce Volume #################################"but my RWX tests fail:
spec: Forbidden: spec is immutable after creation except resources.requests for bound claims
run_k8s_storage_test
PLAY [localhost] ***************************************************************
TASK [ocp login using creds] ***************************************************
changed: [localhost]
TASK [ocp login using token] ***************************************************
skipping: [localhost]
TASK [debug] *******************************************************************
ok: [localhost] => {
"login_creds.stdout_lines": [
"Login successful.",
"",
"You have access to 70 projects, the list has been suppressed. You can list all projects with 'oc projects'",
"",
"Using project \"default\".",
"Welcome! See 'oc help' to get started."
]
}
TASK [debug] *******************************************************************
skipping: [localhost]
TASK [Storage Readiness] *******************************************************
TASK [storage-readiness : Create namespace k8s-validation if not present] ******
ok: [localhost]
TASK [storage-readiness : Run simple mount test for ReadWriteOnce] *************
included: /opt/ansible/roles/storage-readiness/tasks/mount-test.yaml for localhost
TASK [storage-readiness : Test mount ReadWriteOnce volumes for readiness] ******
ok: [localhost] => (item={'name': 'create-volume.yaml.j2'})
ok: [localhost] => (item={'name': 'mount-job.yaml.j2'})
TASK [storage-readiness : Verify mount completed for ReadWriteOnce] ************
ok: [localhost]
TASK [storage-readiness : debug] ***********************************************
ok: [localhost] => {
"msg": "######################## MOUNT TESTS PASSED FOR ReadWriteOnce Volume #################################"
}
TASK [storage-readiness : Run simple mount test for ReadWriteMany] *************
included: /opt/ansible/roles/storage-readiness/tasks/mount-test.yaml for localhost
TASK [storage-readiness : Test mount ReadWriteMany volumes for readiness] ******
failed: [localhost] (item={'name': 'create-volume.yaml.j2'}) => {"ansible_loop_var": "item", "changed": false, "error": 422, "item": {"name": "create-volume.yaml.j2"}, "msg": "Failed to patch object: b'{\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"PersistentVolumeClaim \\\\\"readiness-readwritemany\\\\\" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests for bound claims\\\\n core.PersistentVolumeClaimSpec{\\\\n \\\\t... // 2 identical fields\\\\n \\\\tResources: {Requests: {s\\\\\"storage\\\\\": {i: {...}, s: \\\\\"1Gi\\\\\", Format: \\\\\"BinarySI\\\\\"}}},\\\\n \\\\tVolumeName: \\\\\"\\\\\",\\\\n- \\\\tStorageClassName: \\\\u0026\\\\\"azure-file\\\\\",\\\\n+ \\\\tStorageClassName: \\\\u0026\\\\\"managed-csi\\\\\",\\\\n \\\\tVolumeMode: \\\\u0026\\\\\"Filesystem\\\\\",\\\\n \\\\tDataSource: nil,\\\\n \\\\tDataSourceRef: nil,\\\\n }\\\\n\",\"reason\":\"Invalid\",\"details\":{\"name\":\"readiness-readwritemany\",\"kind\":\"PersistentVolumeClaim\",\"causes\":[{\"reason\":\"FieldValueForbidden\",\"message\":\"Forbidden: spec is immutable after creation except resources.requests for bound claims\\\\n core.PersistentVolumeClaimSpec{\\\\n \\\\t... // 2 identical fields\\\\n \\\\tResources: {Requests: {s\\\\\"storage\\\\\": {i: {...}, s: \\\\\"1Gi\\\\\", Format: \\\\\"BinarySI\\\\\"}}},\\\\n \\\\tVolumeName: \\\\\"\\\\\",\\\\n- \\\\tStorageClassName: \\\\u0026\\\\\"azure-file\\\\\",\\\\n+ \\\\tStorageClassName: \\\\u0026\\\\\"managed-csi\\\\\",\\\\n \\\\tVolumeMode: \\\\u0026\\\\\"Filesystem\\\\\",\\\\n \\\\tDataSource: nil,\\\\n \\\\tDataSourceRef: nil,\\\\n }\\\\n\",\"field\":\"spec\"}]},\"code\":422}\\n'", "reason": "Unprocessable Entity", "status": 422}
ok: [localhost] => (item={'name': 'mount-job.yaml.j2'})
PLAY RECAP *********************************************************************
localhost : ok=8 changed=1 unreachable=0 failed=1 skipped=2 rescued=0 ignored=0 Test #3
So then I deployed an ODF Storage cluster specifying managed-premium as the backing storage.
That created the RWO storage class: ocs-storagecluster-ceph-rbd.
Then I reran the tests with:
storageClass_ReadWriteMany: azure-file
storageClass_ReadWriteOnce: ocs-storagecluster-ceph-rbdAnd the RWO tests failed with:
spec: Forbidden: spec is immutable after creation except resources.requests for bound claims\
run_k8s_storage_test
PLAY [localhost] ***************************************************************
TASK [ocp login using creds] ***************************************************
changed: [localhost]
TASK [ocp login using token] ***************************************************
skipping: [localhost]
TASK [debug] *******************************************************************
ok: [localhost] => {
"login_creds.stdout_lines": [
"Login successful.",
"",
"You have access to 70 projects, the list has been suppressed. You can list all projects with 'oc projects'",
"",
"Using project \"default\".",
"Welcome! See 'oc help' to get started."
]
}
TASK [debug] *******************************************************************
skipping: [localhost]
TASK [Storage Readiness] *******************************************************
TASK [storage-readiness : Create namespace k8s-validation if not present] ******
ok: [localhost]
TASK [storage-readiness : Run simple mount test for ReadWriteOnce] *************
included: /opt/ansible/roles/storage-readiness/tasks/mount-test.yaml for localhost
TASK [storage-readiness : Test mount ReadWriteOnce volumes for readiness] ******
failed: [localhost] (item={'name': 'create-volume.yaml.j2'}) => {"ansible_loop_var": "item", "changed": false, "error": 422, "item": {"name": "create-volume.yaml.j2"}, "msg": "Failed to patch object: b'{\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"PersistentVolumeClaim \\\\\"readiness-readwriteonce\\\\\" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests for bound claims\\\\n core.PersistentVolumeClaimSpec{\\\\n \\\\t... // 2 identical fields\\\\n \\\\tResources: {Requests: {s\\\\\"storage\\\\\": {i: {...}, s: \\\\\"1Gi\\\\\", Format: \\\\\"BinarySI\\\\\"}}},\\\\n \\\\tVolumeName: \\\\\"pvc-bb87825e-239a-4f77-a415-1ebfb19a91df\\\\\",\\\\n- \\\\tStorageClassName: \\\\u0026\\\\\"ocs-storagecluster-ceph-rbd\\\\\",\\\\n+ \\\\tStorageClassName: \\\\u0026\\\\\"managed-premium\\\\\",\\\\n \\\\tVolumeMode: \\\\u0026\\\\\"Filesystem\\\\\",\\\\n \\\\tDataSource: nil,\\\\n \\\\tDataSourceRef: nil,\\\\n }\\\\n\",\"reason\":\"Invalid\",\"details\":{\"name\":\"readiness-readwriteonce\",\"kind\":\"PersistentVolumeClaim\",\"causes\":[{\"reason\":\"FieldValueForbidden\",\"message\":\"Forbidden: spec is immutable after creation except resources.requests for bound claims\\\\n core.PersistentVolumeClaimSpec{\\\\n \\\\t... // 2 identical fields\\\\n \\\\tResources: {Requests: {s\\\\\"storage\\\\\": {i: {...}, s: \\\\\"1Gi\\\\\", Format: \\\\\"BinarySI\\\\\"}}},\\\\n \\\\tVolumeName: \\\\\"pvc-bb87825e-239a-4f77-a415-1ebfb19a91df\\\\\",\\\\n- \\\\tStorageClassName: \\\\u0026\\\\\"ocs-storagecluster-ceph-rbd\\\\\",\\\\n+ \\\\tStorageClassName: \\\\u0026\\\\\"managed-premium\\\\\",\\\\n \\\\tVolumeMode: \\\\u0026\\\\\"Filesystem\\\\\",\\\\n \\\\tDataSource: nil,\\\\n \\\\tDataSourceRef: nil,\\\\n }\\\\n\",\"field\":\"spec\"}]},\"code\":422}\\n'", "reason": "Unprocessable Entity", "status": 422}
ok: [localhost] => (item={'name': 'mount-job.yaml.j2'})
PLAY RECAP *********************************************************************
localhost : ok=4 changed=1 unreachable=0 failed=1 skipped=2 rescued=0 ignored=0 Test #4
Just for fun I tried:
storageClass_ReadWriteMany: ocs-storagecluster-cephfs
storageClass_ReadWriteOnce: managed-premiumThe RWO tests passed:
"msg": "######################## MOUNT TESTS PASSED FOR ReadWriteOnce Volume #################################"But the RWX tests with the ODF File storage class ocs-storagecluster-cephfs FAILED with
spec: Forbidden: spec is immutable after creation except resources.requests for bound claims
run_k8s_storage_test
PLAY [localhost] ***************************************************************
TASK [ocp login using creds] ***************************************************
changed: [localhost]
TASK [ocp login using token] ***************************************************
skipping: [localhost]
TASK [debug] *******************************************************************
ok: [localhost] => {
"login_creds.stdout_lines": [
"Login successful.",
"",
"You have access to 70 projects, the list has been suppressed. You can list all projects with 'oc projects'",
"",
"Using project \"default\".",
"Welcome! See 'oc help' to get started."
]
}
TASK [debug] *******************************************************************
skipping: [localhost]
TASK [Storage Readiness] *******************************************************
TASK [storage-readiness : Create namespace k8s-validation if not present] ******
ok: [localhost]
TASK [storage-readiness : Run simple mount test for ReadWriteOnce] *************
included: /opt/ansible/roles/storage-readiness/tasks/mount-test.yaml for localhost
TASK [storage-readiness : Test mount ReadWriteOnce volumes for readiness] ******
ok: [localhost] => (item={'name': 'create-volume.yaml.j2'})
ok: [localhost] => (item={'name': 'mount-job.yaml.j2'})
TASK [storage-readiness : Verify mount completed for ReadWriteOnce] ************
ok: [localhost]
TASK [storage-readiness : debug] ***********************************************
ok: [localhost] => {
"msg": "######################## MOUNT TESTS PASSED FOR ReadWriteOnce Volume #################################"
}
TASK [storage-readiness : Run simple mount test for ReadWriteMany] *************
included: /opt/ansible/roles/storage-readiness/tasks/mount-test.yaml for localhost
TASK [storage-readiness : Test mount ReadWriteMany volumes for readiness] ******
failed: [localhost] (item={'name': 'create-volume.yaml.j2'}) => {"ansible_loop_var": "item", "changed": false, "error": 422, "item": {"name": "create-volume.yaml.j2"}, "msg": "Failed to patch object: b'{\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"PersistentVolumeClaim \\\\\"readiness-readwritemany\\\\\" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests for bound claims\\\\n core.PersistentVolumeClaimSpec{\\\\n \\\\t... // 2 identical fields\\\\n \\\\tResources: {Requests: {s\\\\\"storage\\\\\": {i: {...}, s: \\\\\"1Gi\\\\\", Format: \\\\\"BinarySI\\\\\"}}},\\\\n \\\\tVolumeName: \\\\\"\\\\\",\\\\n- \\\\tStorageClassName: \\\\u0026\\\\\"ocs-storagecluster-cephfs\\\\\",\\\\n+ \\\\tStorageClassName: \\\\u0026\\\\\"managed-csi\\\\\",\\\\n \\\\tVolumeMode: \\\\u0026\\\\\"Filesystem\\\\\",\\\\n \\\\tDataSource: nil,\\\\n \\\\tDataSourceRef: nil,\\\\n }\\\\n\",\"reason\":\"Invalid\",\"details\":{\"name\":\"readiness-readwritemany\",\"kind\":\"PersistentVolumeClaim\",\"causes\":[{\"reason\":\"FieldValueForbidden\",\"message\":\"Forbidden: spec is immutable after creation except resources.requests for bound claims\\\\n core.PersistentVolumeClaimSpec{\\\\n \\\\t... // 2 identical fields\\\\n \\\\tResources: {Requests: {s\\\\\"storage\\\\\": {i: {...}, s: \\\\\"1Gi\\\\\", Format: \\\\\"BinarySI\\\\\"}}},\\\\n \\\\tVolumeName: \\\\\"\\\\\",\\\\n- \\\\tStorageClassName: \\\\u0026\\\\\"ocs-storagecluster-cephfs\\\\\",\\\\n+ \\\\tStorageClassName: \\\\u0026\\\\\"managed-csi\\\\\",\\\\n \\\\tVolumeMode: \\\\u0026\\\\\"Filesystem\\\\\",\\\\n \\\\tDataSource: nil,\\\\n \\\\tDataSourceRef: nil,\\\\n }\\\\n\",\"field\":\"spec\"}]},\"code\":422}\\n'", "reason": "Unprocessable Entity", "status": 422}
ok: [localhost] => (item={'name': 'mount-job.yaml.j2'})
PLAY RECAP *********************************************************************
localhost : ok=8 changed=1 unreachable=0 failed=1 skipped=2 rescued=0 ignored=0 Conclusion:
The RWO storage validation tests pass when the storage class = managed-premium but fail with the ODF RWO storage class = cs-storagecluster-ceph-rbd.
They also fail with the ODF RWX storage class ocs-storagecluster-cephfs
In both cases they fail with the same type of error:
spec: Forbidden: spec is immutable after creation except resources.requests for bound claims
bxu1999
My ARO clusters: pk6-aro url: https://console-openshift-console.apps.oeq8pl8k.eastus.aroapp.io/ Contact me for the credentials
pk7-aro url: https://console-openshift-console.apps.jce5djuj.eastus.aroapp.io/k8s/cluster/storageclasses az aro list-credentials \
--name pk7-aro \ --resource-group cpbu-sdlc-rg Contact me for the credentials
Storage Validation tests for ODF storage classes are failing with:
params.yml params.yml.zip