Add automated test case for OCP-24688

hu-weihua commented 4 years ago

This pull request is used to automate test case OCP-24688

fbladilo commented 4 years ago

@hu-weihua Also could you briefly describe the test logic on the PR, I don't seem to have access to the reference...

hu-weihua commented 4 years ago

@fbladilo Thank you for all you comments.

I have replaced all k8s_info by k8s_facts. Could you review again?
The main purpose of test case OCP-24688 are to test that the controllers will discover what storageclasses are available on the destination side. So this automated test does below things
- Create a bogus storageclasses named test-storageclass in migration source cluster, and check if creation is successfully.
- To check if the migcluster definition of source cluster contains the information of the bogus test-storageclass and other storageclasses information existed before.
- To check if the status of source cluster has been changed to The cluster is ready
- Create a bogus storageclasses named test-storageclass in migration target cluster, and check if creation is successfully.
- To check if the migcluster definition of target cluster contains the information of the bogus test-storageclass and other storageclasses information existed before.
- To check if the status of target cluster has been changed to The cluster is ready

If you have any other question or comment, feel free to let me know, Thanks again.

fbladilo commented 4 years ago

@hu-weihua Thanks for the revision, I added some other minor comments since last review.

I tested the PR on a 3.11 source and 4.2 target (using CAM v1.0.1) , the first run was successful on both source and destination. The second time, it failed retrying on the destination cluster , see below :

[fbladilo@fbladilo:/tmp/mig-e2e] (24688 %=)$ ansible-playbook e2e_mig_samples.yml -t ocp-24688-storageclasses -e with_deploy=false
 [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'

PLAY [localhost] **************************************************************************************************************************************************************************************************

TASK [Gathering Facts] ********************************************************************************************************************************************************************************************
ok: [localhost]
included: /tmp/mig-e2e/roles/migration_prepare/tasks/prepare_repos.yml for localhost

TASK [migration_prepare : set_fact] *******************************************************************************************************************************************************************************
ok: [localhost]

TASK [migration_prepare : Check if mig-demo-apps exists] **********************************************************************************************************************************************************
ok: [localhost]

TASK [migration_prepare : Include required vars] ******************************************************************************************************************************************************************
ok: [localhost]

TASK [migration_prepare : Create source migration cluster definition] *********************************************************************************************************************************************
ok: [localhost]

TASK [migration_prepare : Create SA token on destination cluster] *************************************************************************************************************************************************
ok: [localhost]

TASK [migration_prepare : Create remote migration cluster definition] *********************************************************************************************************************************************
ok: [localhost]

TASK [migration_prepare : Create migration storage creds] *********************************************************************************************************************************************************
ok: [localhost]

TASK [migration_prepare : Create migration storage] ***************************************************************************************************************************************************************
ok: [localhost]

TASK [migration_cleanup : Ensure namespace ocp-24688-storageclasses is absent before continuing...] ***************************************************************************************************************
ok: [localhost] => (item=ocp-24688-storageclasses)

TASK [migration_cleanup : Check existing migrations for ocp-24688-storageclasses] *********************************************************************************************************************************
ok: [localhost]

TASK [migration_cleanup : Check existing migplans for ocp-24688-storageclasses] ***********************************************************************************************************************************
ok: [localhost]

TASK [ocp-24688-storageclasses : Check existing StorageClass for test-storageclass in migration target cluster migcluster-local-mod] ******************************************************************************
ok: [localhost]

TASK [ocp-24688-storageclasses : Extract StorageClass for test-storageclass in migration target cluster migcluster-local-mod] *************************************************************************************
ok: [localhost]

TASK [ocp-24688-storageclasses : debug] ***************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": [
        "These existing StorageClass will be removed :", 
        [
            "test-storageclass"
        ]
    ]
}

TASK [ocp-24688-storageclasses : Remove existing storageclass for test-storageclass from migration target cluster migcluster-local-mod] ***************************************************************************
changed: [localhost] => (item=test-storageclass)

TASK [ocp-24688-storageclasses : Create bogus storageclasses in migration target cluster migcluster-local-mod] ****************************************************************************************************
changed: [localhost]

TASK [ocp-24688-storageclasses : get all storageclasses infornamtion of migration target cluster] *****************************************************************************************************************
ok: [localhost]

TASK [ocp-24688-storageclasses : get all storageclasses name list of migration target cluster] ********************************************************************************************************************
ok: [localhost]
FAILED - RETRYING: verify migcluster definition of target cluster migcluster-local-mod (30 retries left).
FAILED - RETRYING: verify migcluster definition of target cluster migcluster-local-mod (29 retries left).
FAILED - RETRYING: verify migcluster definition of target cluster migcluster-local-mod (28 retries left).
FAILED - RETRYING: verify migcluster definition of target cluster migcluster-local-mod (27 retries left).
FAILED - RETRYING: verify migcluster definition of target cluster migcluster-local-mod (26 retries left).
FAILED - RETRYING: verify migcluster definition of target cluster migcluster-local-mod (25 retries left).
FAILED - RETRYING: verify migcluster definition of target cluster migcluster-local-mod (24 retries left).
FAILED - RETRYING: verify migcluster definition of target cluster migcluster-local-mod (23 retries left).
FAILED - RETRYING: verify migcluster definition of target cluster migcluster-local-mod (22 retries left).
FAILED - RETRYING: verify migcluster definition of target cluster migcluster-local-mod (21 retries left).
FAILED - RETRYING: verify migcluster definition of target cluster migcluster-local-mod (20 retries left).
FAILED - RETRYING: verify migcluster definition of target cluster migcluster-local-mod (19 retries left).
FAILED - RETRYING: verify migcluster definition of target cluster migcluster-local-mod (18 retries left).
FAILED - RETRYING: verify migcluster definition of target cluster migcluster-local-mod (17 retries left).
FAILED - RETRYING: verify migcluster definition of target cluster migcluster-local-mod (16 retries left).
FAILED - RETRYING: verify migcluster definition of target cluster migcluster-local-mod (15 retries left).
FAILED - RETRYING: verify migcluster definition of target cluster migcluster-local-mod (14 retries left).
FAILED - RETRYING: verify migcluster definition of target cluster migcluster-local-mod (13 retries left).
FAILED - RETRYING: verify migcluster definition of target cluster migcluster-local-mod (12 retries left).

Something is probably not cleaned up correctly on the target deploy/validation, it exhibits the issue when it is run the second consecutive time, you should be able to reproduce just by running twice on the destination/target cluster :

ansible-playbook e2e_mig_samples.yml -t ocp-24688-storageclasses -e with_deploy=false

fbladilo commented 4 years ago

@hu-weihua Let me know how to proceed with this one, I made another revision back in Feb with some comments/experience with it.

hu-weihua commented 4 years ago

@fbladilo Thanks a lot for your valuable comments and sorry to reply later.

I have updated the typos and replaced sleep by pause. I also updated a logic error.

But I did not hit your problem pasted above. I rerun the test in same test environment many times without below error.

TASK [ocp-24688-storageclasses : get all storageclasses name list of migration target cluster] ********************************************************************************************************************
ok: [localhost]
FAILED - RETRYING: verify migcluster definition of target cluster migcluster-local-mod (30 retries left).
...........
FAILED - RETRYING: verify migcluster definition of target cluster migcluster-local-mod (13 retries left)

My test result in target cluster

$ ansible-playbook e2e_mig_samples.yml -e with_deploy=false  --tags ocp-24688-storageclasses
[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'

[DEPRECATION WARNING]: k8s_facts is kept for backwards compatibility but usage is discouraged. The module documentation 
details page may explain more about this rationale.. This feature will be removed in a future release. Deprecation warnings 
can be disabled by setting deprecation_warnings=False in ansible.cfg.

PLAY [localhost] **************************************************************************************************************

TASK [Gathering Facts] ********************************************************************************************************
ok: [localhost]
included: /home/fedora/hwh/github/konveyor/mig-e2e-branch-24688/mig-e2e/roles/migration_prepare/tasks/prepare_repos.yml for localhost

TASK [migration_prepare : set_fact] *******************************************************************************************
ok: [localhost]

TASK [migration_prepare : Check if mig-demo-apps exists] **********************************************************************
ok: [localhost]

TASK [migration_prepare : Include required vars] ******************************************************************************
ok: [localhost]

TASK [migration_prepare : Create source migration cluster definition] *********************************************************
ok: [localhost]

TASK [migration_prepare : Create SA token on destination cluster] *************************************************************
changed: [localhost]

TASK [migration_prepare : Create remote migration cluster definition] *********************************************************
ok: [localhost]

TASK [migration_prepare : Create migration storage creds] *********************************************************************
ok: [localhost]

TASK [migration_prepare : Create migration storage] ***************************************************************************
ok: [localhost]

TASK [migration_cleanup : Ensure namespace ocp-24688-storageclasses is absent before continuing...] ***************************
ok: [localhost] => (item=ocp-24688-storageclasses)

TASK [migration_cleanup : Check existing migrations for ocp-24688-storageclasses] *********************************************
[DEPRECATION WARNING]: The 'k8s_facts' module has been renamed to 'k8s_info'. This feature will be removed in version 2.13. 
Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
ok: [localhost]

TASK [migration_cleanup : Check existing migplans for ocp-24688-storageclasses] ***********************************************
ok: [localhost]

TASK [ocp-24688-storageclasses : Check existing StorageClass for test-storageclass in migration target cluster host] **********
ok: [localhost]

TASK [ocp-24688-storageclasses : Extract StorageClass for test-storageclass in migration target cluster host] *****************
ok: [localhost]

TASK [ocp-24688-storageclasses : debug] ***************************************************************************************
ok: [localhost] => {
    "msg": [
        "These existing StorageClass will be removed :", 
        [
            "test-storageclass"
        ]
    ]
}

TASK [ocp-24688-storageclasses : Remove existing storageclass for test-storageclass from migration target cluster host] *******
changed: [localhost] => (item=test-storageclass)

TASK [ocp-24688-storageclasses : Create bogus storageclasses in migration target cluster host] ********************************
changed: [localhost]

TASK [ocp-24688-storageclasses : get all storageclasses information of migration target cluster] ******************************
ok: [localhost]

TASK [ocp-24688-storageclasses : get all storageclasses name list of migration target cluster] ********************************
ok: [localhost]

TASK [ocp-24688-storageclasses : verify migcluster definition of target cluster host] *****************************************
ok: [localhost]

TASK [ocp-24688-storageclasses : set fact for string] *************************************************************************
ok: [localhost]

TASK [ocp-24688-storageclasses : Verity migcluster definition of source cluster source-cluster] *******************************
ok: [localhost]

TASK [ocp-24688-storageclasses : save the source cluster sc name list in migcluster definition to tmp file] *******************
ok: [localhost]

TASK [ocp-24688-storageclasses : compare the 2 tmp files] *********************************************************************
changed: [localhost]

PLAY RECAP ********************************************************************************************************************
localhost                  : ok=25   changed=4    unreachable=0    failed=0    skipped=27   rescued=0    ignored=0   

$ ansible-playbook e2e_mig_samples.yml -e with_deploy=false  --tags ocp-24688-storageclasses
[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'

[DEPRECATION WARNING]: k8s_facts is kept for backwards compatibility but usage is discouraged. The module documentation 
details page may explain more about this rationale.. This feature will be removed in a future release. Deprecation warnings 
can be disabled by setting deprecation_warnings=False in ansible.cfg.

PLAY [localhost] **************************************************************************************************************

TASK [Gathering Facts] ********************************************************************************************************
ok: [localhost]
included: /home/fedora/hwh/github/konveyor/mig-e2e-branch-24688/mig-e2e/roles/migration_prepare/tasks/prepare_repos.yml for localhost

TASK [migration_prepare : set_fact] *******************************************************************************************
ok: [localhost]

TASK [migration_prepare : Check if mig-demo-apps exists] **********************************************************************
ok: [localhost]

TASK [migration_prepare : Include required vars] ******************************************************************************
ok: [localhost]

TASK [migration_prepare : Create source migration cluster definition] *********************************************************
ok: [localhost]

TASK [migration_prepare : Create SA token on destination cluster] *************************************************************
ok: [localhost]

TASK [migration_prepare : Create remote migration cluster definition] *********************************************************
ok: [localhost]

TASK [migration_prepare : Create migration storage creds] *********************************************************************
ok: [localhost]

TASK [migration_prepare : Create migration storage] ***************************************************************************
ok: [localhost]

TASK [migration_cleanup : Ensure namespace ocp-24688-storageclasses is absent before continuing...] ***************************
ok: [localhost] => (item=ocp-24688-storageclasses)

TASK [migration_cleanup : Check existing migrations for ocp-24688-storageclasses] *********************************************
[DEPRECATION WARNING]: The 'k8s_facts' module has been renamed to 'k8s_info'. This feature will be removed in version 2.13. 
Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
ok: [localhost]

TASK [migration_cleanup : Check existing migplans for ocp-24688-storageclasses] ***********************************************
ok: [localhost]

TASK [ocp-24688-storageclasses : Check existing StorageClass for test-storageclass in migration target cluster host] **********
ok: [localhost]

TASK [ocp-24688-storageclasses : Extract StorageClass for test-storageclass in migration target cluster host] *****************
ok: [localhost]

TASK [ocp-24688-storageclasses : debug] ***************************************************************************************
ok: [localhost] => {
    "msg": [
        "These existing StorageClass will be removed :", 
        [
            "test-storageclass"
        ]
    ]
}

TASK [ocp-24688-storageclasses : Remove existing storageclass for test-storageclass from migration target cluster host] *******
changed: [localhost] => (item=test-storageclass)

TASK [ocp-24688-storageclasses : Create bogus storageclasses in migration target cluster host] ********************************
changed: [localhost]

TASK [ocp-24688-storageclasses : get all storageclasses information of migration target cluster] ******************************
ok: [localhost]

TASK [ocp-24688-storageclasses : get all storageclasses name list of migration target cluster] ********************************
ok: [localhost]

TASK [ocp-24688-storageclasses : verify migcluster definition of target cluster host] *****************************************
ok: [localhost]

TASK [ocp-24688-storageclasses : set fact for string] *************************************************************************
ok: [localhost]

TASK [ocp-24688-storageclasses : Verity migcluster definition of source cluster source-cluster] *******************************
ok: [localhost]

TASK [ocp-24688-storageclasses : save the source cluster sc name list in migcluster definition to tmp file] *******************
ok: [localhost]

TASK [ocp-24688-storageclasses : compare the 2 tmp files] *********************************************************************
changed: [localhost]

PLAY RECAP ********************************************************************************************************************
localhost                  : ok=25   changed=3    unreachable=0    failed=0    skipped=27   rescued=0    ignored=0

migtools / mig-e2e

Add automated test case for OCP-24688 #68