search

irods / irods_capability_storage_tiering

BSD 3-Clause "New" or "Revised" License
5 stars 10 forks source link

Tiering out fails when replica exists on next tier #235

Closed alanking closed 7 months ago

alanking commented 7 months ago

Bug Report

iRODS Version, OS and Version

iRODS server: main Storage tiering plugin: main OS: ubuntu 22.04

What did you try to do?

I discovered this while writing an automated test for #234. The test suite was test_plugin_unified_storage_tiering.TestStorageTieringPluginPreserveReplica. This test had the following setup:

It then does this:

  1. Put some data on ufs0
  2. Tier out the data to ufs2 from ufs0 and then ufs1, and confirm that ufs1 and ufs2 have replicas, per the configured policy
  3. Get the data and confirm that a restage occurs, resulting in replicas on ufs0, ufs1, and ufs2
  4. Tier out the data from ufs0 again

Expected behavior

I expected good replicas on ufs1 and ufs2 and no replica on ufs0.

Observed behavior (including steps to reproduce, if applicable)

An error occurs and the replica remains on ufs0. Here are the messages from the log (filtered through jq and formatted):

"irods::storage_tiering migrating [/tempZone/home/rods/test_put_file] from [ufs0] to [ufs1]"
"use default query for [ufs1]"
"rsDataObjRepl - Failed to replicate data object. status:[-169000]"
"[-]    /irods_plugin_source/libirods_rule_engine_plugin-unified_storage_tiering.cpp:761:irods::error exec_rule_expression(irods::default_re_ctx &, const std::string &, msParamArray_t *, irods::callback) :  status [SYS_NOT_ALLOWED]  errno [] -- message [iRODS Exception:
    file: /irods_plugin_source/libirods_rule_engine_plugin-unified_storage_tiering.cpp
    function: void (anonymous namespace)::replicate_object_to_resource(rcComm_t *, const std::string &, const std::string &, const std::string &, const std::string &)
    line: 110
    code: -169000 (SYS_NOT_ALLOWED)
    message:
        failed to migrate [/tempZone/home/rods/test_put_file] to [ufs1]
stack trace:
--------------
 0# irods::stacktrace::dump() const in /lib/libirods_common.so.4.3.1
 1# irods::exception::assemble_full_display_what() const in /lib/libirods_common.so.4.3.1
 2# irods::exception::what() const in /lib/libirods_common.so.4.3.1
 3# exec_rule_expression(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback) in /usr/lib/irods/plugins/rule_engines/libirods_rule_engine_plugin-unified_storage_tiering.so
 4# std::__1::__function::__func<irods::error (*)(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback), std::__1::allocator<irods::error (*)(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback)>, irods::error (std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback)>::operator()(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*&&, irods::callback&&) in /usr/lib/irods/plugins/rule_engines/libirods_rule_engine_plugin-unified_storage_tiering.so
 5# 0x00007F3AD5F0D45C in /lib/libirods_server.so.4.3.1
 6# std::__1::function<irods::error (std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback)>::operator()(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback) const in /lib/libirods_server.so.4.3.1
 7# irods::pluggable_rule_engine<std::__1::tuple<> >::exec_rule_expression(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback) in /lib/libirods_server.so.4.3.1
 8# irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule_expression(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*) in /lib/libirods_server.so.4.3.1
 9# rsExecRuleExpression(RsComm*, ExecRuleExpression*) in /lib/libirods_server.so.4.3.1
10# 0x00007F3AD663DF3B in /lib/libirods_server.so.4.3.1
11# int std::__1::__invoke_void_return_wrapper<int, false>::__call<int (*&)(RsComm*, ExecRuleExpression*), RsComm*, ExecRuleExpression*>(int (*&)(RsComm*, ExecRuleExpression*), RsComm*&&, ExecRuleExpression*&&) in /lib/libirods_server.so.4.3.1
12# 0x00007F3AD663DE97 in /lib/libirods_server.so.4.3.1
13# std::__1::__function::__func<int (*)(RsComm*, ExecRuleExpression*), std::__1::allocator<int (*)(RsComm*, ExecRuleExpression*)>, int (RsComm*, ExecRuleExpression*)>::operator()(RsComm*&&, ExecRuleExpression*&&) in /lib/libirods_server.so.4.3.1
14# 0x00007F3AD6536E2F in /lib/libirods_server.so.4.3.1
15# std::__1::function<int (RsComm*, ExecRuleExpression*)>::operator()(RsComm*, ExecRuleExpression*) const in /lib/libirods_server.so.4.3.1
16# irods::api_call_adaptor<ExecRuleExpression*>::operator()(irods::plugin_context&, RsComm*, ExecRuleExpression*) in /lib/libirods_server.so.4.3.1
17# 0x00007F3AD6536A91 in /lib/libirods_server.so.4.3.1
18# irods::error std::__1::__invoke_void_return_wrapper<irods::error, false>::__call<irods::api_call_adaptor<ExecRuleExpression*>&, irods::plugin_context&, RsComm*, ExecRuleExpression*>(irods::api_call_adaptor<ExecRuleExpression*>&, irods::plugin_context&, RsComm*&&, ExecRuleExpression*&&) in /lib/libirods_server.so.4.3.1
19# 0x00007F3AD653699F in /lib/libirods_server.so.4.3.1
20# std::__1::__function::__func<irods::api_call_adaptor<ExecRuleExpression*>, std::__1::allocator<irods::api_call_adaptor<ExecRuleExpression*> >, irods::error (irods::plugin_context&, RsComm*, ExecRuleExpression*)>::operator()(irods::plugin_context&, RsComm*&&, ExecRuleExpression*&&) in /lib/libirods_server.so.4.3.1
21# 0x00007F3AD6536F47 in /lib/libirods_server.so.4.3.1
22# std::__1::function<irods::error (irods::plugin_context&, RsComm*, ExecRuleExpression*)>::operator()(irods::plugin_context&, RsComm*, ExecRuleExpression*) const in /lib/libirods_server.so.4.3.1
23# int irods::api_entry::call_handler<ExecRuleExpression*>(RsComm*, ExecRuleExpression*) in /lib/libirods_server.so.4.3.1
24# call_execRuleExpressionInp(irods::api_entry*, RsComm*, ExecRuleExpression*) in /lib/libirods_server.so.4.3.1
25# rsApiHandler(RsComm*, int, BytesBuf*, BytesBuf*) in /lib/libirods_server.so.4.3.1
26# readAndProcClientMsg(RsComm*, int) in /lib/libirods_server.so.4.3.1
27# agentMain(RsComm*) in /lib/libirods_server.so.4.3.1
28# runIrodsAgentFactory(sockaddr_un) in /lib/libirods_server.so.4.3.1
29# operator() at /irods_source/server/main_server/src/rodsServer.cpp:1320
30# main at /irods_source/server/main_server/src/rodsServer.cpp:1387
31# 0x00007F3AD35EBD90 in /lib/x86_64-linux-gnu/libc.so.6
32# __libc_start_main in /lib/x86_64-linux-gnu/libc.so.6
33# _start in /usr/sbin/irodsServer

]

"
"rsDataObjRepl - Failed to replicate data object. status:[-169000]"
"[-]    /irods_plugin_source/libirods_rule_engine_plugin-unified_storage_tiering.cpp:761:irods::error exec_rule_expression(irods::default_re_ctx &, const std::string &, msParamArray_t *, irods::callback) :  status [SYS_NOT_ALLOWED]  errno [] -- message [iRODS Exception:
    file: /irods_plugin_source/libirods_rule_engine_plugin-unified_storage_tiering.cpp
    function: void (anonymous namespace)::replicate_object_to_resource(rcComm_t *, const std::string &, const std::string &, const std::string &, const std::string &)
    line: 110
    code: -169000 (SYS_NOT_ALLOWED)
    message:
        failed to migrate [/tempZone/home/rods/test_put_file] to [ufs1]
stack trace:
--------------
 0# irods::stacktrace::dump() const in /lib/libirods_common.so.4.3.1
 1# irods::exception::assemble_full_display_what() const in /lib/libirods_common.so.4.3.1
 2# irods::exception::what() const in /lib/libirods_common.so.4.3.1
 3# exec_rule_expression(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback) in /usr/lib/irods/plugins/rule_engines/libirods_rule_engine_plugin-unified_storage_tiering.so
 4# std::__1::__function::__func<irods::error (*)(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback), std::__1::allocator<irods::error (*)(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback)>, irods::error (std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback)>::operator()(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*&&, irods::callback&&) in /usr/lib/irods/plugins/rule_engines/libirods_rule_engine_plugin-unified_storage_tiering.so
 5# 0x00007F3AD5F0D45C in /lib/libirods_server.so.4.3.1
 6# std::__1::function<irods::error (std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback)>::operator()(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback) const in /lib/libirods_server.so.4.3.1
 7# irods::pluggable_rule_engine<std::__1::tuple<> >::exec_rule_expression(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback) in /lib/libirods_server.so.4.3.1
 8# irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule_expression(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*) in /lib/libirods_server.so.4.3.1
 9# rsExecRuleExpression(RsComm*, ExecRuleExpression*) in /lib/libirods_server.so.4.3.1
10# 0x00007F3AD663DF3B in /lib/libirods_server.so.4.3.1
11# int std::__1::__invoke_void_return_wrapper<int, false>::__call<int (*&)(RsComm*, ExecRuleExpression*), RsComm*, ExecRuleExpression*>(int (*&)(RsComm*, ExecRuleExpression*), RsComm*&&, ExecRuleExpression*&&) in /lib/libirods_server.so.4.3.1
12# 0x00007F3AD663DE97 in /lib/libirods_server.so.4.3.1
13# std::__1::__function::__func<int (*)(RsComm*, ExecRuleExpression*), std::__1::allocator<int (*)(RsComm*, ExecRuleExpression*)>, int (RsComm*, ExecRuleExpression*)>::operator()(RsComm*&&, ExecRuleExpression*&&) in /lib/libirods_server.so.4.3.1
14# 0x00007F3AD6536E2F in /lib/libirods_server.so.4.3.1
15# std::__1::function<int (RsComm*, ExecRuleExpression*)>::operator()(RsComm*, ExecRuleExpression*) const in /lib/libirods_server.so.4.3.1
16# irods::api_call_adaptor<ExecRuleExpression*>::operator()(irods::plugin_context&, RsComm*, ExecRuleExpression*) in /lib/libirods_server.so.4.3.1
17# 0x00007F3AD6536A91 in /lib/libirods_server.so.4.3.1
18# irods::error std::__1::__invoke_void_return_wrapper<irods::error, false>::__call<irods::api_call_adaptor<ExecRuleExpression*>&, irods::plugin_context&, RsComm*, ExecRuleExpression*>(irods::api_call_adaptor<ExecRuleExpression*>&, irods::plugin_context&, RsComm*&&, ExecRuleExpression*&&) in /lib/libirods_server.so.4.3.1
19# 0x00007F3AD653699F in /lib/libirods_server.so.4.3.1
20# std::__1::__function::__func<irods::api_call_adaptor<ExecRuleExpression*>, std::__1::allocator<irods::api_call_adaptor<ExecRuleExpression*> >, irods::error (irods::plugin_context&, RsComm*, ExecRuleExpression*)>::operator()(irods::plugin_context&, RsComm*&&, ExecRuleExpression*&&) in /lib/libirods_server.so.4.3.1
21# 0x00007F3AD6536F47 in /lib/libirods_server.so.4.3.1
22# std::__1::function<irods::error (irods::plugin_context&, RsComm*, ExecRuleExpression*)>::operator()(irods::plugin_context&, RsComm*, ExecRuleExpression*) const in /lib/libirods_server.so.4.3.1
23# int irods::api_entry::call_handler<ExecRuleExpression*>(RsComm*, ExecRuleExpression*) in /lib/libirods_server.so.4.3.1
24# call_execRuleExpressionInp(irods::api_entry*, RsComm*, ExecRuleExpression*) in /lib/libirods_server.so.4.3.1
25# rsApiHandler(RsComm*, int, BytesBuf*, BytesBuf*) in /lib/libirods_server.so.4.3.1
26# readAndProcClientMsg(RsComm*, int) in /lib/libirods_server.so.4.3.1
27# agentMain(RsComm*) in /lib/libirods_server.so.4.3.1
28# runIrodsAgentFactory(sockaddr_un) in /lib/libirods_server.so.4.3.1
29# operator() at /irods_source/server/main_server/src/rodsServer.cpp:1320
30# main at /irods_source/server/main_server/src/rodsServer.cpp:1387
31# 0x00007F3AD35EBD90 in /lib/x86_64-linux-gnu/libc.so.6
32# __libc_start_main in /lib/x86_64-linux-gnu/libc.so.6
33# _start in /usr/sbin/irodsServer

]

"

This is a direct result of the new rules for replication established in 4.2.9. A good replica is not allowed to be overwritten through replication: https://docs.irods.org/4.3.1/system_overview/data_objects/#replicate The replication API returns SYS_NOT_ALLOWED in this case, and the trim does not occur as a result.

I see a few ways to address this which are by no means the only ways or even the best way:

  1. This plugin should check for the existence of a marked-good replica on the destination resource before attempting the replication and skip the replication step in the event that a good replica already exists on the destination resource (and then continue to the trim step).
  2. The plugin should see SYS_NOT_ALLOWED and take some action to determine whether tiering out or trimming should occur.
  3. The replication API should have an option to not treat this situation as an error.
alanking commented 7 months ago

Another consideration is what happens to the replica on ufs1 if/when the pseudo-tier-out occurs. For the default time-based policy, the access time metadata would need to be updated regardless of whether a replication occurs so that tiering out from ufs1 could happen appropriately. Otherwise, the data on ufs1 would immediately be in violation (since it hasn't been touched since it was last tiered out to ufs2) and it could potentially attempt to tier it out from ufs1 prematurely.

trel commented 7 months ago

Upside to Option 1, no error handling, just ask for whether replication is necessary, then (as per your follow up comment) update its access time, then trim the source. Downside, a minimal TOCTOU window to consider/weigh for consequences.

Upside to Option 2, no TOCTOU. Downside, using errors as 'normal flow'... and potentially masking real errors when they occur.

Upside to Option 3, perhaps the cleanest... Downside, seemingly significant more work before we know it's worth it and lots of other things could depend on changes/additions we might make.

Voting for Option 1. Willing to be swayed.

korydraughn commented 7 months ago

If a write occurs and there are preserved replicas on ufs1 and ufs2, those replicas would become stale and the access time is updated, correct?

alanking commented 7 months ago

If a write occurs and there are preserved replicas on ufs1 and ufs2, those replicas would become stale and the access time is updated, correct?

I think that is true.

alanking commented 7 months ago

After some discussion, I think we are in agreement that Option 1 is the way to go.