The test creates (256 * 2) + 1 data objects and then starts waiting for them to be moved to another tier. This works, but close observation shows there are failures which lead to stale replicas existing on the original tier. I've noticed at least 3 replicas in this state following test completion. The test fails because it finds replicas on the original tier, even though it moved 400+ replicas.
Decreasing the number of replicas involved (by 100 or so) resulted in the test passing. However, reducing the number of replicas isn't a real fix. We need to figure out WHY data movement fails for some replicas.
Below is the output of the failed test.
<testcase classname="irods.test.test_plugin_unified_storage_tiering.TestStorageTieringContinueInxMigration" name="test_put_multi_fetch_page" time="287.555" timestamp="2024-08-13T21:11:55" file="scripts/irods/test/test_plugin_unified_storage_tiering.py" line="1503">
<failure type="AssertionError" message=""><![CDATA[Traceback (most recent call last):
File "/var/lib/irods/scripts/irods/test/test_plugin_unified_storage_tiering.py", line 180, in delay_assert
out, err, rc = function()
^^^^^^^^^^
File "/var/lib/irods/scripts/irods/test/test_plugin_unified_storage_tiering.py", line 1520, in <lambda>
delay_assert(lambda: admin_session.assert_icommand_fail(['ils', '-l', dirname], 'STDOUT_SINGLELINE', 'ufs0'))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/lib/irods/scripts/irods/test/session.py", line 166, in assert_icommand_fail
return assert_command_fail(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/lib/irods/scripts/irods/test/command.py", line 79, in assert_command_fail
return _assert_helper(*args, should_fail=True, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/lib/irods/scripts/irods/test/command.py", line 104, in _assert_helper
assert result
^^^^^^
AssertionError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/irods/scripts/irods/test/test_plugin_unified_storage_tiering.py", line 1520, in test_put_multi_fetch_page
delay_assert(lambda: admin_session.assert_icommand_fail(['ils', '-l', dirname], 'STDOUT_SINGLELINE', 'ufs0'))
File "/var/lib/irods/scripts/irods/test/test_plugin_unified_storage_tiering.py", line 187, in delay_assert
assert(False)
^^^^^
AssertionError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/irods/scripts/irods/test/test_plugin_unified_storage_tiering.py", line 1526, in test_put_multi_fetch_page
admin_session.assert_icommand('irm -r ' + dirname)
File "/var/lib/irods/scripts/irods/test/session.py", line 162, in assert_icommand
return assert_command(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/lib/irods/scripts/irods/test/command.py", line 76, in assert_command
return _assert_helper(*args, should_fail=False, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/lib/irods/scripts/irods/test/command.py", line 104, in _assert_helper
assert result
^^^^^^
AssertionError
]]></failure>
Here is the final listing before the assertion fails.
[ ] 4-3-stable
Bug Report
Encountered during testing of what will be iRODS 4.3.3. Platform is Ubuntu 24.04. Database is MySQL 8.4.
The test fails due to the following assertion.
https://github.com/irods/irods_capability_storage_tiering/blob/822347c9dc05612e53e6eb2e1da473283ee675f1/packaging/test_plugin_unified_storage_tiering.py#L1520
The test creates (256 * 2) + 1 data objects and then starts waiting for them to be moved to another tier. This works, but close observation shows there are failures which lead to stale replicas existing on the original tier. I've noticed at least 3 replicas in this state following test completion. The test fails because it finds replicas on the original tier, even though it moved 400+ replicas.
Decreasing the number of replicas involved (by 100 or so) resulted in the test passing. However, reducing the number of replicas isn't a real fix. We need to figure out WHY data movement fails for some replicas.
Below is the output of the failed test.
Here is the final listing before the assertion fails.