search

IBM / k8s-storage-tests

Playbooks for validating a Kubernetes persistent storage for IBM Cloud Paks
Apache License 2.0
16 stars 12 forks source link

File Lock Tests: flock and fcntl positive lock and negative lock pods scheduled on same node #21

Closed shankarpentyala07 closed 5 months ago

shankarpentyala07 commented 5 months ago
  1. flock across nodes is not supported.
  2. Use podAffinity for create-lock , create-lock2 pods to be scheduled on the same node

Test results:

spectrum storage:


 ansible-playbook main.yml --extra-vars "@./params.yml" | tee output.log
[WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match 'all'

PLAY [localhost] ***************************************************************

TASK [ocp login using creds] ***************************************************
changed: [localhost]

TASK [ocp login using token] ***************************************************
skipping: [localhost]

TASK [debug] *******************************************************************
ok: [localhost] => {
    "login_creds.stdout_lines": [
        "WARNING: Using insecure TLS client config. Setting this option is not supported!",
        "",
        "Login successful.",
        "",
        "You have access to 82 projects, the list has been suppressed. You can list all projects with 'oc projects'",
        "",
        "Using project \"zen\"."
    ]
}

TASK [debug] *******************************************************************
skipping: [localhost]

TASK [Storage Readiness] *******************************************************
[WARNING]: Collection kubernetes.core does not support Ansible version 2.10.17

TASK [storage-readiness : Create namespace stgtest if not present] *************
ok: [localhost]

TASK [storage-readiness : Run simple mount test for ReadWriteOnce] *************
included: /root/storage_test/k8s-storage-tests/roles/storage-readiness/tasks/mount-test.yaml for localhost

TASK [storage-readiness : Test mount ReadWriteOnce volumes for readiness] ******
changed: [localhost] => (item={'name': 'create-volume.yaml.j2'})
ok: [localhost] => (item={'name': 'mount-job.yaml.j2'})

TASK [storage-readiness : Verify mount completed for ReadWriteOnce] ************
FAILED - RETRYING: Verify mount completed for ReadWriteOnce (20 retries left).
FAILED - RETRYING: Verify mount completed for ReadWriteOnce (19 retries left).
ok: [localhost]

TASK [storage-readiness : debug] ***********************************************
ok: [localhost] => {
    "msg": "######################## MOUNT TESTS PASSED FOR ReadWriteOnce Volume  #################################"
}

TASK [storage-readiness : Run simple mount test for ReadWriteMany] *************
included: /root/storage_test/k8s-storage-tests/roles/storage-readiness/tasks/mount-test.yaml for localhost

TASK [storage-readiness : Test mount ReadWriteMany volumes for readiness] ******
changed: [localhost] => (item={'name': 'create-volume.yaml.j2'})
changed: [localhost] => (item={'name': 'mount-job.yaml.j2'})

TASK [storage-readiness : Verify mount completed for ReadWriteMany] ************
FAILED - RETRYING: Verify mount completed for ReadWriteMany (20 retries left).
FAILED - RETRYING: Verify mount completed for ReadWriteMany (19 retries left).
ok: [localhost]

TASK [storage-readiness : debug] ***********************************************
ok: [localhost] => {
    "msg": "######################## MOUNT TESTS PASSED FOR ReadWriteMany Volume  #################################"
}

TASK [storage-readiness : Run sequential RW tests for ReadWriteOnce] ***********
included: /root/storage_test/k8s-storage-tests/roles/storage-readiness/tasks/sequential-rw.yaml for localhost

TASK [storage-readiness : Test mount ReadWriteOnce volumes for readiness] ******
ok: [localhost] => (item={'name': 'create-volume.yaml.j2'})

TASK [storage-readiness : Set config maps] *************************************
changed: [localhost] => (item={'name': 'producer-cm.yaml.j2'})
changed: [localhost] => (item={'name': 'consumer-cm.yaml.j2'})

TASK [storage-readiness : Sequential read/write ReadWriteOnce volumes for readiness] ***
changed: [localhost] => (item={'name': 'producer.yaml.j2'})
changed: [localhost] => (item={'name': 'consumer.yaml.j2'})

TASK [storage-readiness : debug] ***********************************************
ok: [localhost] => {
    "msg": "######################## SEQUENTIAL READ WRITE TEST PASSED FOR ReadWriteOnce Volume #################################"
}

TASK [storage-readiness : Run sequential RW tests for ReadWriteMany] ***********
included: /root/storage_test/k8s-storage-tests/roles/storage-readiness/tasks/sequential-rw.yaml for localhost

TASK [storage-readiness : Test mount ReadWriteMany volumes for readiness] ******
ok: [localhost] => (item={'name': 'create-volume.yaml.j2'})

TASK [storage-readiness : Set config maps] *************************************
ok: [localhost] => (item={'name': 'producer-cm.yaml.j2'})
ok: [localhost] => (item={'name': 'consumer-cm.yaml.j2'})

TASK [storage-readiness : Sequential read/write ReadWriteMany volumes for readiness] ***
changed: [localhost] => (item={'name': 'producer.yaml.j2'})
changed: [localhost] => (item={'name': 'consumer.yaml.j2'})

TASK [storage-readiness : debug] ***********************************************
ok: [localhost] => {
    "msg": "######################## SEQUENTIAL READ WRITE TEST PASSED FOR ReadWriteMany Volume #################################"
}

TASK [storage-readiness : Run parallel rw single thread tests  ReadWriteOnce] ***
included: /root/storage_test/k8s-storage-tests/roles/storage-readiness/tasks/parallel-rw-single-thread.yaml for localhost

TASK [storage-readiness : Test mount ReadWriteOnce volumes for readiness] ******
ok: [localhost] => (item={'name': 'create-volume.yaml.j2'})

TASK [storage-readiness : Set config maps] *************************************
ok: [localhost] => (item={'name': 'producer-cm.yaml.j2'})
changed: [localhost] => (item={'name': 'consumer-nocheck-cm.yaml.j2'})

TASK [storage-readiness : Parallel read/write  ReadWriteOnce volumes for readiness] ***
changed: [localhost] => (item={'name': 'producer.yaml.j2'})

TASK [storage-readiness : Parallel concurrency read while producer is running] ***
changed: [localhost] => (item={'name': 'consumer-nocheck.yaml.j2'})

TASK [storage-readiness : Assert that producer is still running] ***************
ok: [localhost]

TASK [storage-readiness : Warn if producer job is already completed] ***********
skipping: [localhost]

TASK [storage-readiness : Wait until producer job is done] *********************
FAILED - RETRYING: Wait until producer job is done (20 retries left).
ok: [localhost]

TASK [storage-readiness : Parallel read  ReadWriteOnce volumes for readiness] ***
changed: [localhost] => (item={'name': 'consumer.yaml.j2'})

TASK [storage-readiness : debug] ***********************************************
ok: [localhost] => {
    "msg": "######################## SINGLE THREAD PARALLEL READ WRITE TEST PASSED for ReadWriteOnce #################################"
}

TASK [storage-readiness : Run parallel rw single thread tests for ReadWriteMany] ***
included: /root/storage_test/k8s-storage-tests/roles/storage-readiness/tasks/parallel-rw-single-thread.yaml for localhost

TASK [storage-readiness : Test mount ReadWriteMany volumes for readiness] ******
ok: [localhost] => (item={'name': 'create-volume.yaml.j2'})

TASK [storage-readiness : Set config maps] *************************************
ok: [localhost] => (item={'name': 'producer-cm.yaml.j2'})
ok: [localhost] => (item={'name': 'consumer-nocheck-cm.yaml.j2'})

TASK [storage-readiness : Parallel read/write  ReadWriteMany volumes for readiness] ***
changed: [localhost] => (item={'name': 'producer.yaml.j2'})

TASK [storage-readiness : Parallel concurrency read while producer is running] ***
changed: [localhost] => (item={'name': 'consumer-nocheck.yaml.j2'})

TASK [storage-readiness : Assert that producer is still running] ***************
ok: [localhost]

TASK [storage-readiness : Warn if producer job is already completed] ***********
skipping: [localhost]

TASK [storage-readiness : Wait until producer job is done] *********************
FAILED - RETRYING: Wait until producer job is done (20 retries left).
ok: [localhost]

TASK [storage-readiness : Parallel read  ReadWriteMany volumes for readiness] ***
changed: [localhost] => (item={'name': 'consumer.yaml.j2'})

TASK [storage-readiness : debug] ***********************************************
ok: [localhost] => {
    "msg": "######################## SINGLE THREAD PARALLEL READ WRITE TEST PASSED for ReadWriteMany #################################"
}

TASK [storage-readiness : Run parallel rw multi thread tests for ReadWriteOnce] ***
included: /root/storage_test/k8s-storage-tests/roles/storage-readiness/tasks/parallel-rw-multi-thread.yaml for localhost

TASK [storage-readiness : Test mount ReadWriteOnce volumes for readiness] ******
ok: [localhost] => (item={'name': 'create-volume.yaml.j2'})

TASK [storage-readiness : Set config maps] *************************************
ok: [localhost] => (item={'name': 'producer-cm.yaml.j2'})
ok: [localhost] => (item={'name': 'consumer-nocheck-cm.yaml.j2'})

TASK [storage-readiness : Get node count] **************************************
ok: [localhost]

TASK [storage-readiness : set_fact] ********************************************
ok: [localhost]

TASK [storage-readiness : Parallel write from same/separate nodes] *************
changed: [localhost] => (item={'name': 'parallel-producer.yaml.j2'})

TASK [storage-readiness : Parallel write no wait from same/separate nodes] *****
changed: [localhost] => (item={'name': 'parallel-producer.yaml.j2'})

TASK [storage-readiness : Parallel read no wait from same/separate nodes] ******
changed: [localhost] => (item={'name': 'parallel-consumer.yaml.j2'})

TASK [storage-readiness : Wait Until parallel producer job is done] ************
FAILED - RETRYING: Wait Until parallel producer job is done (10 retries left).
ok: [localhost]

TASK [storage-readiness : Wait Until parallel consumer job is done] ************
ok: [localhost]

TASK [storage-readiness : debug] ***********************************************
ok: [localhost] => {
    "msg": "######################## MULTI NODE PARALLEL READ WRTIE TEST PASSED FOR ReadWriteOnce #################################"
}

TASK [storage-readiness : Run parallel rw multi thread tests  for ReadWriteMany] ***
included: /root/storage_test/k8s-storage-tests/roles/storage-readiness/tasks/parallel-rw-multi-thread.yaml for localhost

TASK [storage-readiness : Test mount ReadWriteMany volumes for readiness] ******
ok: [localhost] => (item={'name': 'create-volume.yaml.j2'})

TASK [storage-readiness : Set config maps] *************************************
ok: [localhost] => (item={'name': 'producer-cm.yaml.j2'})
ok: [localhost] => (item={'name': 'consumer-nocheck-cm.yaml.j2'})

TASK [storage-readiness : Get node count] **************************************
ok: [localhost]

TASK [storage-readiness : set_fact] ********************************************
ok: [localhost]

TASK [storage-readiness : Parallel write from same/separate nodes] *************
changed: [localhost] => (item={'name': 'parallel-producer.yaml.j2'})

TASK [storage-readiness : Parallel write no wait from same/separate nodes] *****
changed: [localhost] => (item={'name': 'parallel-producer.yaml.j2'})

TASK [storage-readiness : Parallel read no wait from same/separate nodes] ******
changed: [localhost] => (item={'name': 'parallel-consumer.yaml.j2'})

TASK [storage-readiness : Wait Until parallel producer job is done] ************
FAILED - RETRYING: Wait Until parallel producer job is done (10 retries left).
ok: [localhost]

TASK [storage-readiness : Wait Until parallel consumer job is done] ************
ok: [localhost]

TASK [storage-readiness : debug] ***********************************************
ok: [localhost] => {
    "msg": "######################## MULTI NODE PARALLEL READ WRTIE TEST PASSED FOR ReadWriteMany #################################"
}

TASK [storage-readiness : File UID permission tests for ReadWriteMany] *********
included: /root/storage_test/k8s-storage-tests/roles/storage-readiness/tasks/file-uid-test.yaml for localhost

TASK [storage-readiness : Test mount ReadWriteMany volumes for readiness] ******
ok: [localhost] => (item={'name': 'create-volume.yaml.j2'})

TASK [storage-readiness : Set config maps] *************************************
ok: [localhost] => (item={'name': 'producer-cm.yaml.j2'})
ok: [localhost] => (item={'name': 'consumer-cm.yaml.j2'})

TASK [storage-readiness : create custom scc] ***********************************
changed: [localhost]

TASK [storage-readiness : create custom sa] ************************************
changed: [localhost]

TASK [storage-readiness : add scc to custom sa] ********************************
changed: [localhost]

TASK [storage-readiness : File uid create ReadWriteMany volumes for readiness] ***
changed: [localhost] => (item={'name': 'file-owner-uid-test.yaml.j2'})
changed: [localhost] => (item={'name': 'file-create-uid.yaml.j2'})
changed: [localhost] => (item={'name': 'file-edit-uid.yaml.j2'})

TASK [storage-readiness : debug] ***********************************************
ok: [localhost] => {
    "msg": "######################## FILE UID TEST PASSED FOR ReadWriteMany Volume #################################"
}

TASK [storage-readiness : SupplementalGroup GID permission tests for ReadWriteMany] ***
included: /root/storage_test/k8s-storage-tests/roles/storage-readiness/tasks/file-permissions-test.yaml for localhost

TASK [storage-readiness : Test mount ReadWriteMany volumes for readiness] ******
ok: [localhost] => (item={'name': 'create-volume.yaml.j2'})

TASK [storage-readiness : Set config maps] *************************************
ok: [localhost] => (item={'name': 'producer-cm.yaml.j2'})
ok: [localhost] => (item={'name': 'consumer-cm.yaml.j2'})

TASK [storage-readiness : Get first uid from project range] ********************
changed: [localhost]

TASK [storage-readiness : Set target uid to set for the reader job to be different than the writer] ***
ok: [localhost]

TASK [storage-readiness : debug] ***********************************************
ok: [localhost] => {
    "msg": "The target_uid_reader is 1000810100"
}

TASK [storage-readiness : create custom scc] ***********************************
skipping: [localhost]

TASK [storage-readiness : create custom sa] ************************************
skipping: [localhost]

TASK [storage-readiness : add scc to custom sa] ********************************
skipping: [localhost]

TASK [storage-readiness : File Permissions GID test for ReadWriteMany volumes for readiness] ***
changed: [localhost] => (item={'name': 'create-file-gid.yaml.j2'})
changed: [localhost] => (item={'name': 'read-file-gid.yaml.j2'})
changed: [localhost] => (item={'name': 'edit-file-gid.yaml.j2'})

TASK [storage-readiness : File Permissions Supplemental Group test for ReadWriteMany volumes for readiness] ***
changed: [localhost] => (item={'name': 'create-file-gid.yaml.j2'})
changed: [localhost] => (item={'name': 'read-file-gid.yaml.j2'})
changed: [localhost] => (item={'name': 'edit-file-gid.yaml.j2'})

TASK [storage-readiness : debug] ***********************************************
ok: [localhost] => {
    "msg": "######################## FILE PERMISSIONS TEST PASSED FOR ReadWriteMany Volume #################################"
}

TASK [storage-readiness : FSGroup GID permission tests for ReadWriteOnce] ******
included: /root/storage_test/k8s-storage-tests/roles/storage-readiness/tasks/file-permissions-test.yaml for localhost

TASK [storage-readiness : Test mount ReadWriteOnce volumes for readiness] ******
ok: [localhost] => (item={'name': 'create-volume.yaml.j2'})

TASK [storage-readiness : Set config maps] *************************************
ok: [localhost] => (item={'name': 'producer-cm.yaml.j2'})
ok: [localhost] => (item={'name': 'consumer-cm.yaml.j2'})

TASK [storage-readiness : Get first uid from project range] ********************
changed: [localhost]

TASK [storage-readiness : Set target uid to set for the reader job to be different than the writer] ***
ok: [localhost]

TASK [storage-readiness : debug] ***********************************************
ok: [localhost] => {
    "msg": "The target_uid_reader is 1000810100"
}

TASK [storage-readiness : create custom scc] ***********************************
changed: [localhost]

TASK [storage-readiness : create custom sa] ************************************
changed: [localhost]

TASK [storage-readiness : add scc to custom sa] ********************************
changed: [localhost]

TASK [storage-readiness : File Permissions GID test for ReadWriteOnce volumes for readiness] ***
changed: [localhost] => (item={'name': 'create-file-gid.yaml.j2'})
changed: [localhost] => (item={'name': 'read-file-gid.yaml.j2'})
changed: [localhost] => (item={'name': 'edit-file-gid.yaml.j2'})

TASK [storage-readiness : File Permissions Supplemental Group test for ReadWriteOnce volumes for readiness] ***
changed: [localhost] => (item={'name': 'create-file-gid.yaml.j2'})
changed: [localhost] => (item={'name': 'read-file-gid.yaml.j2'})
changed: [localhost] => (item={'name': 'edit-file-gid.yaml.j2'})

TASK [storage-readiness : debug] ***********************************************
ok: [localhost] => {
    "msg": "######################## FILE PERMISSIONS TEST PASSED FOR ReadWriteOnce Volume #################################"
}

TASK [storage-readiness : Subpath tests ReadWriteMany] *************************
included: /root/storage_test/k8s-storage-tests/roles/storage-readiness/tasks/volume-subpath-test.yaml for localhost

TASK [storage-readiness : Test mount ReadWriteMany volumes for readiness] ******
ok: [localhost] => (item={'name': 'create-volume.yaml.j2'})

TASK [storage-readiness : Set config maps] *************************************
ok: [localhost] => (item={'name': 'producer-cm.yaml.j2'})
ok: [localhost] => (item={'name': 'consumer-cm.yaml.j2'})

TASK [storage-readiness : Subpath tests for ReadWriteMany volumes for readiness] ***
changed: [localhost] => (item={'name': 'producer-subpath.yaml.j2'})
changed: [localhost] => (item={'name': 'consumer-subpath.yaml.j2'})

TASK [storage-readiness : debug] ***********************************************
ok: [localhost] => {
    "msg": "######################## SUB PATH TEST PASSED FOR ReadWriteMany Volume #################################"
}

TASK [storage-readiness : File locks tests] ************************************
included: /root/storage_test/k8s-storage-tests/roles/storage-readiness/tasks/file-lock-test.yaml for localhost

TASK [storage-readiness : Test mount ReadWriteMany volumes for readiness] ******
ok: [localhost] => (item={'name': 'create-volume.yaml.j2'})

TASK [storage-readiness : Set config maps] *************************************
ok: [localhost] => (item={'name': 'producer-cm.yaml.j2'})

TASK [storage-readiness : File lock test for readiness] ************************
changed: [localhost] => (item={'name': 'file-lock.yaml.j2'})

TASK [storage-readiness : Wait few seconds] ************************************
Pausing for 20 seconds
(ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort)
ok: [localhost]

TASK [storage-readiness : Negative file lock test for readiness] ***************
changed: [localhost] => (item={'name': 'file-lock2.yaml.j2'})

TASK [storage-readiness : Wait for lock to be released] ************************
changed: [localhost] => (item={'name': 'file-lock.yaml.j2'})

TASK [storage-readiness : Positive lock test for readiness] ********************

changed: [localhost] => (item={'name': 'file-lock.yaml.j2'})

TASK [storage-readiness : Fcntl test for readiness] ****************************
changed: [localhost] => (item={'name': 'file-lock-fcntl.yaml.j2'})

TASK [storage-readiness : Wait few seconds] ************************************
Pausing for 20 seconds
(ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort)
ok: [localhost]

TASK [storage-readiness : Negative Fcntl test for readiness] *******************
changed: [localhost] => (item={'name': 'file-lock-fcntl2.yaml.j2'})

TASK [storage-readiness : Wait for Fcntl lock to be released] ******************
changed: [localhost] => (item={'name': 'file-lock-fcntl.yaml.j2'})

TASK [storage-readiness : Positive Fcntl test for readiness] *******************
changed: [localhost] => (item={'name': 'file-lock-fcntl.yaml.j2'})

TASK [storage-readiness : debug] ***********************************************
ok: [localhost] => {
    "msg": "######################## FILE LOCK TESTS PASSED FOR ReadWriteMany Volume #################################"
}

PLAY RECAP *********************************************************************
localhost                  : ok=109  changed=41   unreachable=0    failed=0    skipped=7    rescued=0    ignored=0  
flock,fcntl pods :

oc get po -o wide  | grep readiness-file-lock
readiness-file-lock-fcntl-job-create-lock-r22jb                   0/1     Completed   0          5m17s   10.254.41.137   worker5.sre-chs-3m15w-ss-c627.cp.fyre.ibm.com    <none>           <none>
readiness-file-lock-fcntl-job-create-lock2-rktnn                  0/1     Error       0          4m45s   10.254.41.140   worker5.sre-chs-3m15w-ss-c627.cp.fyre.ibm.com    <none>           <none>
readiness-file-lock-fcntl-job-create-lock2-x9d28                  0/1     Error       0          4m56s   10.254.41.139   worker5.sre-chs-3m15w-ss-c627.cp.fyre.ibm.com    <none>           <none>
readiness-file-lock-fcntl-job-create-lock3-ll2fz                  0/1     Completed   0          3m59s   10.254.41.143   worker5.sre-chs-3m15w-ss-c627.cp.fyre.ibm.com    <none>           <none>
readiness-file-lock-job-create-lock-pj5gr                         0/1     Completed   0          7m45s   10.254.41.121   worker5.sre-chs-3m15w-ss-c627.cp.fyre.ibm.com    <none>           <none>
readiness-file-lock-job-create-lock2-6wmnq                        0/1     Error       0          7m14s   10.254.41.125   worker5.sre-chs-3m15w-ss-c627.cp.fyre.ibm.com    <none>           <none>
readiness-file-lock-job-create-lock2-kldhw                        0/1     Error       0          7m24s   10.254.41.124   worker5.sre-chs-3m15w-ss-c627.cp.fyre.ibm.com    <none>           <none>
readiness-file-lock-job-create-lock3-hdx2k                        0/1     Completed   0          6m28s   10.254.41.130   worker5.sre-chs-3m15w-ss-c627.cp.fyre.ibm.com    <none>           <none>
``

Tested on NFS, ODF and pods create-lock,create-lock2 are getting scheduled on same node as expected.
shankarpentyala07 commented 5 months ago

Merging Changes