implement preserve.replicas when restaging files

[cookie33]

Implement fix for: https://github.com/irods/irods_capability_storage_tiering/issues/233

[cookie33]

We have to think about it. It works as follows for a restage:

• get replica for the resource where to stage from. fi: 7

  $ ils -l
  /igor/home/rods:
  rods              7 eudatPnfs      1931831 2023-12-11.13:44 & test_20231208_60.txt

• get group name by replica number. But this might be an other number. 8 if a previous restage has been done and the file has been removed from the upper tier again.

  $ imeta ls -d test_20231208_60.txt
  AVUs defined for dataObj /igor/home/rods/test_20231208_60.txt:
  attribute: irods::access_time
  value: 1702298856
  units:
  ----
  attribute: irods::storage_tiering::group
  value: eudat
  units: 8

  And in this stage it fails because the replica and group version do not match. See if there is an other function to get the group.

[cookie33]

Changed one query:

-            boost::format("SELECT META_DATA_ATTR_VALUE WHERE DATA_NAME = '%s' AND COLL_NAME = '%s' AND META_DATA_ATTR_NAME = '%s' AND META_DATA_ATTR_UNITS = '%s'")
+            boost::format("SELECT META_DATA_ATTR_VALUE WHERE DATA_NAME = '%s' AND COLL_NAME = '%s' AND META_DATA_ATTR_NAME = '%s' AND META_DATA_ATTR_UNITS >= '%s'")

This makes sure that the function get_group_name_by_replica_number will work even if the replica has a lower number than the unit in irods::storage_tiering::group

[alanking]

I left a comment in #232 regarding the query update: https://github.com/irods/irods_capability_storage_tiering/issues/232#issuecomment-1852808480

[cookie33]

I made it a draft pull request. I will make some tests in bash and when that is OK I will convert it back to a real pull request again. The test will be similar like the automated tests from the irods consortium itself.

[korydraughn]

packaging/test_plugin_unified_storage_tiering.py contains all of our tests. All tests are expected to be implemented using Python.

We can also translate your bash tests into python if that helps.

[cookie33]

I tried running the python tests on my system. But the first thing it does is stop the iRODS server :(

I will try to get it running in a similar fashion as the iRODS tests. This will make it easier to incorporate. After that I will update the official test_plugin_unified_storage_tiering.py with my updates after testing as much as possible.

But it will take some time to get some proper tests together. So please bear with me.

[korydraughn]

That's correct. The run_tests.py script restarts the server before launching the tests to make sure the environment is ready.

Tests which depend on the delay server (like this project) normally restart the server too.

We're happy to help you cross the finish line :-)

[korydraughn]

Forgot to mention that you can use the iRODS Development Environment to run the tests easier. See https://github.com/irods/irods_development_environment.

With that, you can use the plugin builder and plugin test runners to build and test.

[cookie33]

Updated test_plugin_unified_storage_tiering_test .py in the class TestStorageTieringPluginPreserveReplica

Renamed one test to test_put because it did not do a get. Added 4 other tests where preserve_replicas is set to true. With some tests it now also checks if the replication number is the one which is expected.

tested:

$ time python3 run_tests.py --run_specific_test test_plugin_unified_storage_tiering > /tmp/test.txt
irods.test.test_plugin_unified_storage_tiering_test.TestStorageTieringPluginPreserveReplica.test_put ... ok
irods.test.test_plugin_unified_storage_tiering_test.TestStorageTieringPluginPreserveReplica.test_put_and_get_01 ... ok
irods.test.test_plugin_unified_storage_tiering_test.TestStorageTieringPluginPreserveReplica.test_put_and_get_02 ... ok
irods.test.test_plugin_unified_storage_tiering_test.TestStorageTieringPluginPreserveReplica.test_put_and_get_03 ... ok
irods.test.test_plugin_unified_storage_tiering_test.TestStorageTieringPluginPreserveReplica.test_put_and_get_04 ... ok

----------------------------------------------------------------------
Ran 5 tests in 308.318s

OK

real    5m8.492s
user    0m4.896s
sys     0m1.739s

[alanking]

Confirmed that tests are passing. Added #s. Merging.