IBM / k8s-storage-tests

Playbooks for validating a Kubernetes persistent storage for IBM Cloud Paks
Apache License 2.0
16 stars 12 forks source link

Storage validation tests are failing on Openshift 4.10 on ARO with ODF (self-managed) storage #14

Open pamandrejko opened 1 year ago

pamandrejko commented 1 year ago

My ARO clusters: pk6-aro url: https://console-openshift-console.apps.oeq8pl8k.eastus.aroapp.io/ Contact me for the credentials

pk7-aro url: https://console-openshift-console.apps.jce5djuj.eastus.aroapp.io/k8s/cluster/storageclasses az aro list-credentials \
--name pk7-aro \ --resource-group cpbu-sdlc-rg Contact me for the credentials

Storage Validation tests for ODF storage classes are failing with:

 run_k8s_storage_test

PLAY [localhost] ***************************************************************

TASK [ocp login using creds] ***************************************************
changed: [localhost]

TASK [ocp login using token] ***************************************************
skipping: [localhost]

TASK [debug] *******************************************************************
ok: [localhost] => {
    "login_creds.stdout_lines": [
        "Login successful.",
        "",
        "You have access to 70 projects, the list has been suppressed. You can list all projects with 'oc projects'",
        "",
        "Using project \"default\".",
        "Welcome! See 'oc help' to get started."
    ]
}

TASK [debug] *******************************************************************
skipping: [localhost]

TASK [Storage Readiness] *******************************************************

TASK [storage-readiness : Create namespace k8s-validation if not present] ******
ok: [localhost]

TASK [storage-readiness : Run simple mount test for ReadWriteOnce] *************
included: /opt/ansible/roles/storage-readiness/tasks/mount-test.yaml for localhost

TASK [storage-readiness : Test mount ReadWriteOnce volumes for readiness] ******
failed: [localhost] (item={'name': 'create-volume.yaml.j2'}) => {"ansible_loop_var": "item", "changed": false, "error": 422, "item": {"name": "create-volume.yaml.j2"}, "msg": "Failed to patch object: b'{\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"PersistentVolumeClaim \\\\\"readiness-readwriteonce\\\\\" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests for bound claims\\\\n\\xc2\\xa0\\xc2\\xa0core.PersistentVolumeClaimSpec{\\\\n\\xc2\\xa0\\xc2\\xa0\\\\t... // 2 identical fields\\\\n\\xc2\\xa0\\xc2\\xa0\\\\tResources:        {Requests: {s\\\\\"storage\\\\\": {i: {...}, s: \\\\\"1Gi\\\\\", Format: \\\\\"BinarySI\\\\\"}}},\\\\n\\xc2\\xa0\\xc2\\xa0\\\\tVolumeName:       \\\\\"\\\\\",\\\\n-\\xc2\\xa0\\\\tStorageClassName: \\\\u0026\\\\\"ocs-storagecluster-ceph-rbd\\\\\",\\\\n+\\xc2\\xa0\\\\tStorageClassName: \\\\u0026\\\\\"rook-ceph-block\\\\\",\\\\n\\xc2\\xa0\\xc2\\xa0\\\\tVolumeMode:       \\\\u0026\\\\\"Filesystem\\\\\",\\\\n\\xc2\\xa0\\xc2\\xa0\\\\tDataSource:       nil,\\\\n\\xc2\\xa0\\xc2\\xa0\\\\tDataSourceRef:    nil,\\\\n\\xc2\\xa0\\xc2\\xa0}\\\\n\",\"reason\":\"Invalid\",\"details\":{\"name\":\"readiness-readwriteonce\",\"kind\":\"PersistentVolumeClaim\",\"causes\":[{\"reason\":\"FieldValueForbidden\",\"message\":\"Forbidden: spec is immutable after creation except resources.requests for bound claims\\\\n\\xc2\\xa0\\xc2\\xa0core.PersistentVolumeClaimSpec{\\\\n\\xc2\\xa0\\xc2\\xa0\\\\t... // 2 identical fields\\\\n\\xc2\\xa0\\xc2\\xa0\\\\tResources:        {Requests: {s\\\\\"storage\\\\\": {i: {...}, s: \\\\\"1Gi\\\\\", Format: \\\\\"BinarySI\\\\\"}}},\\\\n\\xc2\\xa0\\xc2\\xa0\\\\tVolumeName:       \\\\\"\\\\\",\\\\n-\\xc2\\xa0\\\\tStorageClassName: \\\\u0026\\\\\"ocs-storagecluster-ceph-rbd\\\\\",\\\\n+\\xc2\\xa0\\\\tStorageClassName: \\\\u0026\\\\\"rook-ceph-block\\\\\",\\\\n\\xc2\\xa0\\xc2\\xa0\\\\tVolumeMode:       \\\\u0026\\\\\"Filesystem\\\\\",\\\\n\\xc2\\xa0\\xc2\\xa0\\\\tDataSource:       nil,\\\\n\\xc2\\xa0\\xc2\\xa0\\\\tDataSourceRef:    nil,\\\\n\\xc2\\xa0\\xc2\\xa0}\\\\n\",\"field\":\"spec\"}]},\"code\":422}\\n'", "reason": "Unprocessable Entity", "status": 422}
ok: [localhost] => (item={'name': 'mount-job.yaml.j2'})

PLAY RECAP *********************************************************************
localhost                  : ok=4    changed=1    unreachable=0    failed=1    skipped=2    rescued=0    ignored=0   

params.yml params.yml.zip

pamandrejko commented 1 year ago

These tests were performed on cluster: pk7-aro url: https://console-openshift-console.apps.jce5djuj.eastus.aroapp.io/k8s/cluster/storageclasses az aro list-credentials \ --name pk7-aro --resource-group cpbu-sdlc-rg Contact me for the credentials

When you deploy an ODF storage cluster on ARO, you have to select a "Backing storage type" from the existing storage classes available: Screen Shot 2022-12-06 at 3 21 24 PM

From the screenshot you can see only 2 options are available on ARO:

The ARO ODF documentation says to select managed-premium.

Note: The validation tests always test BOTH RWO and RWX storage classes. It always tests RWO first, followed by RWX. But if the RWO tests FAIL, then the test aborts, and never runs the RWX storage tests.

Test #1

So let's just run the tests with the managed-premium RWO storage class, and remove ODF from the equation:

If I run the storage validation tests against the underlying backing storage managed-premium they PASS,

msg": "######################## MOUNT TESTS PASSED FOR ReadWriteOnce Volume  #################################"

but fail for the RWX storage. (I expect the RWX tests to fail at this point.) In this case, in params.yml I had specified:

storageClass_ReadWriteMany: managed-csi
storageClass_ReadWriteOnce: managed-premium

managed-csi is bogus, but I had to specify something to test the managed-premium and the managed-premium storage class tests passed!

run_k8s_storage_test

PLAY [localhost] ***************************************************************

TASK [ocp login using creds] ***************************************************
changed: [localhost]

TASK [ocp login using token] ***************************************************
skipping: [localhost]

TASK [debug] *******************************************************************
ok: [localhost] => {
    "login_creds.stdout_lines": [
        "Login successful.",
        "",
        "You have access to 69 projects, the list has been suppressed. You can list all projects with 'oc projects'",
        "",
        "Using project \"default\".",
        "Welcome! See 'oc help' to get started."
    ]
}

TASK [debug] *******************************************************************
skipping: [localhost]

TASK [Storage Readiness] *******************************************************

TASK [storage-readiness : Create namespace k8s-validation if not present] ******
changed: [localhost]

TASK [storage-readiness : Run simple mount test for ReadWriteOnce] *************
included: /opt/ansible/roles/storage-readiness/tasks/mount-test.yaml for localhost

TASK [storage-readiness : Test mount ReadWriteOnce volumes for readiness] ******
changed: [localhost] => (item={'name': 'create-volume.yaml.j2'})
changed: [localhost] => (item={'name': 'mount-job.yaml.j2'})

TASK [storage-readiness : Verify mount completed for ReadWriteOnce] ************
FAILED - RETRYING: Verify mount completed for ReadWriteOnce (20 retries left).
FAILED - RETRYING: Verify mount completed for ReadWriteOnce (19 retries left).
ok: [localhost]

TASK [storage-readiness : debug] ***********************************************
ok: [localhost] => {
    "msg": "######################## MOUNT TESTS PASSED FOR ReadWriteOnce Volume  #################################"
}

TASK [storage-readiness : Run simple mount test for ReadWriteMany] *************
included: /opt/ansible/roles/storage-readiness/tasks/mount-test.yaml for localhost

TASK [storage-readiness : Test mount ReadWriteMany volumes for readiness] ******
changed: [localhost] => (item={'name': 'create-volume.yaml.j2'})
changed: [localhost] => (item={'name': 'mount-job.yaml.j2'})

TASK [storage-readiness : Verify mount completed for ReadWriteMany] ************
FAILED - RETRYING: Verify mount completed for ReadWriteMany (20 retries left).
FAILED - RETRYING: Verify mount completed for ReadWriteMany (19 retries left).
FAILED - RETRYING: Verify mount completed for ReadWriteMany (18 retries left).
FAILED - RETRYING: Verify mount completed for ReadWriteMany (17 retries left).
FAILED - RETRYING: Verify mount completed for ReadWriteMany (16 retries left).
FAILED - RETRYING: Verify mount completed for ReadWriteMany (15 retries left).
FAILED - RETRYING: Verify mount completed for ReadWriteMany (14 retries left).
FAILED - RETRYING: Verify mount completed for ReadWriteMany (13 retries left).
FAILED - RETRYING: Verify mount completed for ReadWriteMany (12 retries left).
FAILED - RETRYING: Verify mount completed for ReadWriteMany (11 retries left).
FAILED - RETRYING: Verify mount completed for ReadWriteMany (10 retries left).
FAILED - RETRYING: Verify mount completed for ReadWriteMany (9 retries left).
FAILED - RETRYING: Verify mount completed for ReadWriteMany (8 retries left).
FAILED - RETRYING: Verify mount completed for ReadWriteMany (7 retries left).
FAILED - RETRYING: Verify mount completed for ReadWriteMany (6 retries left).
FAILED - RETRYING: Verify mount completed for ReadWriteMany (5 retries left).
FAILED - RETRYING: Verify mount completed for ReadWriteMany (4 retries left).
FAILED - RETRYING: Verify mount completed for ReadWriteMany (3 retries left).
FAILED - RETRYING: Verify mount completed for ReadWriteMany (2 retries left).
FAILED - RETRYING: Verify mount completed for ReadWriteMany (1 retries left).
fatal: [localhost]: FAILED! => {"attempts": 20, "changed": false, "resources": [{"apiVersion": "batch/v1", "kind": "Job", "metadata": {"creationTimestamp": "2022-12-06T20:20:15Z", "generation": 1, "labels": {"controller-uid": "65ea6529-81ec-4ec4-a40e-c5412ad4d69d", "job-name": "readiness-mount-job-readwritemany"}, "managedFields": [{"apiVersion": "batch/v1", "fieldsType": "FieldsV1", "fieldsV1": {"f:spec": {"f:backoffLimit": {}, "f:completionMode": {}, "f:completions": {}, "f:parallelism": {}, "f:suspend": {}, "f:template": {"f:metadata": {"f:name": {}}, "f:spec": {"f:containers": {"k:{\"name\":\"mount-job\"}": {".": {}, "f:args": {}, "f:command": {}, "f:image": {}, "f:imagePullPolicy": {}, "f:name": {}, "f:resources": {}, "f:terminationMessagePath": {}, "f:terminationMessagePolicy": {}, "f:volumeMounts": {".": {}, "k:{\"mountPath\":\"/mnt/data\"}": {".": {}, "f:mountPath": {}, "f:name": {}}}}}, "f:dnsPolicy": {}, "f:restartPolicy": {}, "f:schedulerName": {}, "f:securityContext": {}, "f:terminationGracePeriodSeconds": {}, "f:volumes": {".": {}, "k:{\"name\":\"tmp-store\"}": {".": {}, "f:name": {}, "f:persistentVolumeClaim": {".": {}, "f:claimName": {}}}}}}}}, "manager": "OpenAPI-Generator", "operation": "Update", "time": "2022-12-06T20:20:15Z"}, {"apiVersion": "batch/v1", "fieldsType": "FieldsV1", "fieldsV1": {"f:status": {"f:active": {}, "f:startTime": {}}}, "manager": "kube-controller-manager", "operation": "Update", "subresource": "status", "time": "2022-12-06T20:20:15Z"}], "name": "readiness-mount-job-readwritemany", "namespace": "k8s-validation", "resourceVersion": "48001", "uid": "65ea6529-81ec-4ec4-a40e-c5412ad4d69d"}, "spec": {"backoffLimit": 6, "completionMode": "NonIndexed", "completions": 1, "parallelism": 1, "selector": {"matchLabels": {"controller-uid": "65ea6529-81ec-4ec4-a40e-c5412ad4d69d"}}, "suspend": false, "template": {"metadata": {"creationTimestamp": null, "labels": {"controller-uid": "65ea6529-81ec-4ec4-a40e-c5412ad4d69d", "job-name": "readiness-mount-job-readwritemany"}, "name": "mount-job"}, "spec": {"containers": [{"args": ["-c", "echo \"volume mounted\";echo \"writing data\" > /mnt/data/readiness"], "command": ["sh"], "image": "quay.io/centos/amd64", "imagePullPolicy": "IfNotPresent", "name": "mount-job", "resources": {}, "terminationMessagePath": "/dev/termination-log", "terminationMessagePolicy": "File", "volumeMounts": [{"mountPath": "/mnt/data", "name": "tmp-store"}]}], "dnsPolicy": "ClusterFirst", "restartPolicy": "Never", "schedulerName": "default-scheduler", "securityContext": {}, "terminationGracePeriodSeconds": 30, "volumes": [{"name": "tmp-store", "persistentVolumeClaim": {"claimName": "readiness-readwritemany"}}]}}}, "status": {"active": 1, "startTime": "2022-12-06T20:20:15Z"}}]}

PLAY RECAP *********************************************************************
localhost                  : ok=9    changed=4    unreachable=0    failed=1    skipped=2    rescued=0    ignored=0 

Test #2

I followed these instructions to deploy a new NFS backed POSIX compliant azure-file storage (currently in Tech Preview).

This time I run the storage validation tests with:

storageClass_ReadWriteMany: azure-file
storageClass_ReadWriteOnce: managed-premium

And this time the RWO tests PASS,

"msg": "######################## MOUNT TESTS PASSED FOR ReadWriteOnce Volume  #################################"

but my RWX tests fail: spec: Forbidden: spec is immutable after creation except resources.requests for bound claims

run_k8s_storage_test

PLAY [localhost] ***************************************************************

TASK [ocp login using creds] ***************************************************
changed: [localhost]

TASK [ocp login using token] ***************************************************
skipping: [localhost]

TASK [debug] *******************************************************************
ok: [localhost] => {
    "login_creds.stdout_lines": [
        "Login successful.",
        "",
        "You have access to 70 projects, the list has been suppressed. You can list all projects with 'oc projects'",
        "",
        "Using project \"default\".",
        "Welcome! See 'oc help' to get started."
    ]
}

TASK [debug] *******************************************************************
skipping: [localhost]

TASK [Storage Readiness] *******************************************************

TASK [storage-readiness : Create namespace k8s-validation if not present] ******
ok: [localhost]

TASK [storage-readiness : Run simple mount test for ReadWriteOnce] *************
included: /opt/ansible/roles/storage-readiness/tasks/mount-test.yaml for localhost

TASK [storage-readiness : Test mount ReadWriteOnce volumes for readiness] ******
ok: [localhost] => (item={'name': 'create-volume.yaml.j2'})
ok: [localhost] => (item={'name': 'mount-job.yaml.j2'})

TASK [storage-readiness : Verify mount completed for ReadWriteOnce] ************
ok: [localhost]

TASK [storage-readiness : debug] ***********************************************
ok: [localhost] => {
    "msg": "######################## MOUNT TESTS PASSED FOR ReadWriteOnce Volume  #################################"
}

TASK [storage-readiness : Run simple mount test for ReadWriteMany] *************
included: /opt/ansible/roles/storage-readiness/tasks/mount-test.yaml for localhost

TASK [storage-readiness : Test mount ReadWriteMany volumes for readiness] ******
failed: [localhost] (item={'name': 'create-volume.yaml.j2'}) => {"ansible_loop_var": "item", "changed": false, "error": 422, "item": {"name": "create-volume.yaml.j2"}, "msg": "Failed to patch object: b'{\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"PersistentVolumeClaim \\\\\"readiness-readwritemany\\\\\" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests for bound claims\\\\n  core.PersistentVolumeClaimSpec{\\\\n  \\\\t... // 2 identical fields\\\\n  \\\\tResources:        {Requests: {s\\\\\"storage\\\\\": {i: {...}, s: \\\\\"1Gi\\\\\", Format: \\\\\"BinarySI\\\\\"}}},\\\\n  \\\\tVolumeName:       \\\\\"\\\\\",\\\\n- \\\\tStorageClassName: \\\\u0026\\\\\"azure-file\\\\\",\\\\n+ \\\\tStorageClassName: \\\\u0026\\\\\"managed-csi\\\\\",\\\\n  \\\\tVolumeMode:       \\\\u0026\\\\\"Filesystem\\\\\",\\\\n  \\\\tDataSource:       nil,\\\\n  \\\\tDataSourceRef:    nil,\\\\n  }\\\\n\",\"reason\":\"Invalid\",\"details\":{\"name\":\"readiness-readwritemany\",\"kind\":\"PersistentVolumeClaim\",\"causes\":[{\"reason\":\"FieldValueForbidden\",\"message\":\"Forbidden: spec is immutable after creation except resources.requests for bound claims\\\\n  core.PersistentVolumeClaimSpec{\\\\n  \\\\t... // 2 identical fields\\\\n  \\\\tResources:        {Requests: {s\\\\\"storage\\\\\": {i: {...}, s: \\\\\"1Gi\\\\\", Format: \\\\\"BinarySI\\\\\"}}},\\\\n  \\\\tVolumeName:       \\\\\"\\\\\",\\\\n- \\\\tStorageClassName: \\\\u0026\\\\\"azure-file\\\\\",\\\\n+ \\\\tStorageClassName: \\\\u0026\\\\\"managed-csi\\\\\",\\\\n  \\\\tVolumeMode:       \\\\u0026\\\\\"Filesystem\\\\\",\\\\n  \\\\tDataSource:       nil,\\\\n  \\\\tDataSourceRef:    nil,\\\\n  }\\\\n\",\"field\":\"spec\"}]},\"code\":422}\\n'", "reason": "Unprocessable Entity", "status": 422}
ok: [localhost] => (item={'name': 'mount-job.yaml.j2'})

PLAY RECAP *********************************************************************
localhost                  : ok=8    changed=1    unreachable=0    failed=1    skipped=2    rescued=0    ignored=0   

Test #3

So then I deployed an ODF Storage cluster specifying managed-premium as the backing storage. That created the RWO storage class: ocs-storagecluster-ceph-rbd.

Then I reran the tests with:

storageClass_ReadWriteMany: azure-file
storageClass_ReadWriteOnce: ocs-storagecluster-ceph-rbd

And the RWO tests failed with: spec: Forbidden: spec is immutable after creation except resources.requests for bound claims\

run_k8s_storage_test

PLAY [localhost] ***************************************************************

TASK [ocp login using creds] ***************************************************
changed: [localhost]

TASK [ocp login using token] ***************************************************
skipping: [localhost]

TASK [debug] *******************************************************************
ok: [localhost] => {
    "login_creds.stdout_lines": [
        "Login successful.",
        "",
        "You have access to 70 projects, the list has been suppressed. You can list all projects with 'oc projects'",
        "",
        "Using project \"default\".",
        "Welcome! See 'oc help' to get started."
    ]
}

TASK [debug] *******************************************************************
skipping: [localhost]

TASK [Storage Readiness] *******************************************************

TASK [storage-readiness : Create namespace k8s-validation if not present] ******
ok: [localhost]

TASK [storage-readiness : Run simple mount test for ReadWriteOnce] *************
included: /opt/ansible/roles/storage-readiness/tasks/mount-test.yaml for localhost

TASK [storage-readiness : Test mount ReadWriteOnce volumes for readiness] ******
failed: [localhost] (item={'name': 'create-volume.yaml.j2'}) => {"ansible_loop_var": "item", "changed": false, "error": 422, "item": {"name": "create-volume.yaml.j2"}, "msg": "Failed to patch object: b'{\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"PersistentVolumeClaim \\\\\"readiness-readwriteonce\\\\\" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests for bound claims\\\\n  core.PersistentVolumeClaimSpec{\\\\n  \\\\t... // 2 identical fields\\\\n  \\\\tResources:        {Requests: {s\\\\\"storage\\\\\": {i: {...}, s: \\\\\"1Gi\\\\\", Format: \\\\\"BinarySI\\\\\"}}},\\\\n  \\\\tVolumeName:       \\\\\"pvc-bb87825e-239a-4f77-a415-1ebfb19a91df\\\\\",\\\\n- \\\\tStorageClassName: \\\\u0026\\\\\"ocs-storagecluster-ceph-rbd\\\\\",\\\\n+ \\\\tStorageClassName: \\\\u0026\\\\\"managed-premium\\\\\",\\\\n  \\\\tVolumeMode:       \\\\u0026\\\\\"Filesystem\\\\\",\\\\n  \\\\tDataSource:       nil,\\\\n  \\\\tDataSourceRef:    nil,\\\\n  }\\\\n\",\"reason\":\"Invalid\",\"details\":{\"name\":\"readiness-readwriteonce\",\"kind\":\"PersistentVolumeClaim\",\"causes\":[{\"reason\":\"FieldValueForbidden\",\"message\":\"Forbidden: spec is immutable after creation except resources.requests for bound claims\\\\n  core.PersistentVolumeClaimSpec{\\\\n  \\\\t... // 2 identical fields\\\\n  \\\\tResources:        {Requests: {s\\\\\"storage\\\\\": {i: {...}, s: \\\\\"1Gi\\\\\", Format: \\\\\"BinarySI\\\\\"}}},\\\\n  \\\\tVolumeName:       \\\\\"pvc-bb87825e-239a-4f77-a415-1ebfb19a91df\\\\\",\\\\n- \\\\tStorageClassName: \\\\u0026\\\\\"ocs-storagecluster-ceph-rbd\\\\\",\\\\n+ \\\\tStorageClassName: \\\\u0026\\\\\"managed-premium\\\\\",\\\\n  \\\\tVolumeMode:       \\\\u0026\\\\\"Filesystem\\\\\",\\\\n  \\\\tDataSource:       nil,\\\\n  \\\\tDataSourceRef:    nil,\\\\n  }\\\\n\",\"field\":\"spec\"}]},\"code\":422}\\n'", "reason": "Unprocessable Entity", "status": 422}
ok: [localhost] => (item={'name': 'mount-job.yaml.j2'})

PLAY RECAP *********************************************************************
localhost                  : ok=4    changed=1    unreachable=0    failed=1    skipped=2    rescued=0    ignored=0   

Test #4

Just for fun I tried:

storageClass_ReadWriteMany: ocs-storagecluster-cephfs
storageClass_ReadWriteOnce: managed-premium

The RWO tests passed:

 "msg": "######################## MOUNT TESTS PASSED FOR ReadWriteOnce Volume  #################################"

But the RWX tests with the ODF File storage class ocs-storagecluster-cephfs FAILED with spec: Forbidden: spec is immutable after creation except resources.requests for bound claims

run_k8s_storage_test

PLAY [localhost] ***************************************************************

TASK [ocp login using creds] ***************************************************
changed: [localhost]

TASK [ocp login using token] ***************************************************
skipping: [localhost]

TASK [debug] *******************************************************************
ok: [localhost] => {
    "login_creds.stdout_lines": [
        "Login successful.",
        "",
        "You have access to 70 projects, the list has been suppressed. You can list all projects with 'oc projects'",
        "",
        "Using project \"default\".",
        "Welcome! See 'oc help' to get started."
    ]
}

TASK [debug] *******************************************************************
skipping: [localhost]

TASK [Storage Readiness] *******************************************************

TASK [storage-readiness : Create namespace k8s-validation if not present] ******
ok: [localhost]

TASK [storage-readiness : Run simple mount test for ReadWriteOnce] *************
included: /opt/ansible/roles/storage-readiness/tasks/mount-test.yaml for localhost

TASK [storage-readiness : Test mount ReadWriteOnce volumes for readiness] ******
ok: [localhost] => (item={'name': 'create-volume.yaml.j2'})
ok: [localhost] => (item={'name': 'mount-job.yaml.j2'})

TASK [storage-readiness : Verify mount completed for ReadWriteOnce] ************
ok: [localhost]

TASK [storage-readiness : debug] ***********************************************
ok: [localhost] => {
    "msg": "######################## MOUNT TESTS PASSED FOR ReadWriteOnce Volume  #################################"
}

TASK [storage-readiness : Run simple mount test for ReadWriteMany] *************
included: /opt/ansible/roles/storage-readiness/tasks/mount-test.yaml for localhost

TASK [storage-readiness : Test mount ReadWriteMany volumes for readiness] ******
failed: [localhost] (item={'name': 'create-volume.yaml.j2'}) => {"ansible_loop_var": "item", "changed": false, "error": 422, "item": {"name": "create-volume.yaml.j2"}, "msg": "Failed to patch object: b'{\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"PersistentVolumeClaim \\\\\"readiness-readwritemany\\\\\" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests for bound claims\\\\n  core.PersistentVolumeClaimSpec{\\\\n  \\\\t... // 2 identical fields\\\\n  \\\\tResources:        {Requests: {s\\\\\"storage\\\\\": {i: {...}, s: \\\\\"1Gi\\\\\", Format: \\\\\"BinarySI\\\\\"}}},\\\\n  \\\\tVolumeName:       \\\\\"\\\\\",\\\\n- \\\\tStorageClassName: \\\\u0026\\\\\"ocs-storagecluster-cephfs\\\\\",\\\\n+ \\\\tStorageClassName: \\\\u0026\\\\\"managed-csi\\\\\",\\\\n  \\\\tVolumeMode:       \\\\u0026\\\\\"Filesystem\\\\\",\\\\n  \\\\tDataSource:       nil,\\\\n  \\\\tDataSourceRef:    nil,\\\\n  }\\\\n\",\"reason\":\"Invalid\",\"details\":{\"name\":\"readiness-readwritemany\",\"kind\":\"PersistentVolumeClaim\",\"causes\":[{\"reason\":\"FieldValueForbidden\",\"message\":\"Forbidden: spec is immutable after creation except resources.requests for bound claims\\\\n  core.PersistentVolumeClaimSpec{\\\\n  \\\\t... // 2 identical fields\\\\n  \\\\tResources:        {Requests: {s\\\\\"storage\\\\\": {i: {...}, s: \\\\\"1Gi\\\\\", Format: \\\\\"BinarySI\\\\\"}}},\\\\n  \\\\tVolumeName:       \\\\\"\\\\\",\\\\n- \\\\tStorageClassName: \\\\u0026\\\\\"ocs-storagecluster-cephfs\\\\\",\\\\n+ \\\\tStorageClassName: \\\\u0026\\\\\"managed-csi\\\\\",\\\\n  \\\\tVolumeMode:       \\\\u0026\\\\\"Filesystem\\\\\",\\\\n  \\\\tDataSource:       nil,\\\\n  \\\\tDataSourceRef:    nil,\\\\n  }\\\\n\",\"field\":\"spec\"}]},\"code\":422}\\n'", "reason": "Unprocessable Entity", "status": 422}
ok: [localhost] => (item={'name': 'mount-job.yaml.j2'})

PLAY RECAP *********************************************************************
localhost                  : ok=8    changed=1    unreachable=0    failed=1    skipped=2    rescued=0    ignored=0  

Conclusion:

The RWO storage validation tests pass when the storage class = managed-premium but fail with the ODF RWO storage class = cs-storagecluster-ceph-rbd. They also fail with the ODF RWX storage class ocs-storagecluster-cephfs In both cases they fail with the same type of error:

spec: Forbidden: spec is immutable after creation except resources.requests for bound claims

bxu1999 commented 1 year ago

Will look into the issues in this git issue. Thanks.