Docker CI 3.3:Remote copy replication and FC_ISCSI_Multi_array test failing due to error during mounting.

[vishal-rane]

Below tests are failing both the ubuntu and centos clients(managed plugin): hpe_3par_api_multi_array_test.py::MultiArrayTest::test_multi_array_for_both_ISCSI_and_FC hpe_3par_api_remoteCopy_test.py::RemoteCopyTest::test_active_passive_replication Note: Tests working fine on same setup and configuration for "3.2.1-prerelease" tag.

error_message.txt

replication_log.txt

[nilangekarss]

@vishal-rane did you try performing these tests manually(not individual CI test case run)? Did it pass or fail?

[vishal-rane]

Hi Swapnil, Test is creating volume and trying to mount it and failing when trying to mount the volume. I tried to mount the volume manually which is created by the script which is failing with the same error. mount in other tests working fine but failing for this 2 tests. seems to be a specific case. Please look at the latest CI runs logs for more details: http://16.184.15.6:8080/job/docker-3par-iscsi-plugin-ubuntu/779/consoleFull LEt me know if any other details are required.

[vishal-rane]

Hi Swapnil, Please find 3pardcv logs for ISCSI_FC test failure: ISCSI_FC_Failure_log.txt

[nilangekarss]

@vishal-rane Did you use multiple backend to created volumes and try mounting it? If it is failing to do so, i can consider this as a plugin bug. If it is failing for only CI test case, this test case needs to be debugged to find the root cause and needs to be fixed accordingly. Can you confirm that in case of multiple backends volume mount is failing for created volumes?

[imran-ansari]

Exception seen in logs

2019-10-10 03:57:20,749 [ERROR] hpedockerplugin.fileutil [139805717601000] PoolThread-twisted.internet.reactor-0 ('Make directory failed exception is : %s', 'list index out of range')

Explanation

Logs indicate that instead of picking up multipath device, the plugin is picking up single device "/dev/sdd". When this path is passed to fileutil.makedir_for_mounting(), it expects this to be a multipath device and tries to parse the string according to that format:

        data = path.split("/")
        # TODO: Investigate what triggers OS Brick to return a
        # /dev/mapper vs. /dev/disk/by-path path
        if 'mapper' in data:
            uuid = data[3]
        else:
            uuid = data[4]

Since splitting "/dev/sdd" results in just two tokens whereas the plugin tries to access non-existing tokens at 3rd or 4th index, it results in the above exception.

Requires investigation to find out why plugin rather os-brick ends up picking single device when multipath is specified.

[vishal-rane]

Both test got passed after adding FC Zoning between 3par and client, hence issue can be closed. hpe_3par_api_multi_array_test.py::MultiArrayTest::test_multi_array_for_both_ISCSI_and_FC Passed hpe_3par_api_remoteCopy_test.py::RemoteCopyTest::test_active_passive_replication Passed