search

irods / irods_capability_storage_tiering

BSD 3-Clause "New" or "Revised" License
5 stars 10 forks source link

storage tiering does not work for objects from remote users/zones #215

Closed cookie33 closed 9 months ago

cookie33 commented 1 year ago

BUG

VERSIONS

iRODS 4.2.12

Expected BEHAVIOUR

When putting a file on a remote site in a home directory in a tiered storage resource the object should be relicated to a second tier.

Observed behaviour

The action causes errors in the logfiles and it is not replicated to the second tier.

Replication steps

We have two zones:

The user robertv is known at both sides:

When putting files from igor to frank in the home directory on frank the file is not replicated from TIER 0 to TIER 1

put file on remote zone frank in home directory (from igor to frank):

$ iput eudatCreatePid_2.r /frank/home/robertv#igor/test_data_org_igor_20230704_01.txt -R eudatCache

The file is present:

$ ils -l /frank/home/robertv#igor/
/frank/home/robertv#igor:
...
 robertv           0 eudatCache         1195 2023-07-04.14:26 & test_data_org_igor_20230704_01.txt
...

The metadata of the file which was automaticly created with the put action:

$ imeta ls -d /frank/home/robertv#igor/test_data_org_igor_20230704_01.txt
AVUs defined for dataObj /frank/home/robertv#igor/test_data_org_igor_20230704_01.txt:
attribute: irods::access_time
value: 1688473568
units:

Errors in the logfile. It will not move from TIER 0 to TIER 1

Jul  4 14:26:00 pid:25201 NOTICE: use default query for [eudatCache]
Jul  4 14:26:08 pid:25218 NOTICE: writeLine: inString = Entering pep_api_data_obj_put_post for /frank/home/robertv#igor/test_data_org_igor_20230704_01.txt
Jul  4 14:27:30 pid:25346 NOTICE: use default query for [eudatCache]
Jul  4 14:28:30 pid:25429 NOTICE: use default query for [eudatCache]
Jul  4 14:28:30 pid:25429 NOTICE: found 4 objects for resc [eudatCache] with query [SELECT DATA_NAME, COLL_NAME, USER_NAME, DATA_REPL_NUM WHERE META_DATA_ATTR_NAME = 'irods::access_time' AND META_DATA_ATTR_VALUE < '1688473590' AND META_DATA_ATTR_UNITS <> 'irods::storage_tiering::migration_scheduled' AND DATA_RESC_ID IN ('10002',)] type [0]
Jul  4 14:28:30 pid:25429 NOTICE: irods::storage_tiering :: delay params for [eudatCache] - [<INST_NAME>irods_rule_engine_plugin-unified_storage_tiering-instance</INST_NAME><EF>1h DOUBLE UNTIL SUCCESS OR 6 TIMES</EF><PLUSET>26s</PLUSET>]
Jul  4 14:28:30 pid:25429 remote addresses: 127.0.0.1, x.y.z.n ERROR: data movement scheduling failed - [-808000]::[iRODS Exception:
    file: /irods_plugin_source/storage_tiering.cpp
    function: std::string irods::storage_tiering::get_metadata_for_data_object(rcComm_t *, const std::string &, const std::string &)
    line: 91
    code: -808000 (CAT_NO_ROWS_FOUND)
    message:
        no results found for object [/frank/home/robertv#igor/test_data_org_igor_20230704_01.txt] with attribute [irods::access_time]
stack trace:
--------------

Dumping stack trace
<0>     Offset: 0x6e    Address: 0x7fed6b592b0e irods::exception::exception(long, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, unsigned int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)
<1>     Offset: 0x42    Address: 0x7fed6b592cd2 irods::exception::exception(long, boost::basic_format<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, unsigned int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)
<2>     Offset: 0x491   Address: 0x7fed66a5e951 irods::storage_tiering::get_metadata_for_data_object(RcComm*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)
<3>     Offset: 0x34    Address: 0x7fed66a66bb4 irods::storage_tiering::set_migration_metadata_flag_for_object(RcComm*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)
<4>     Offset: 0x65    Address: 0x7fed66a64c55 irods::storage_tiering::queue_data_movement(RcComm*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)
<5>     Offset:         Address: 0x7fed66a6942c /usr/lib/irods/plugins/rule_engines/libirods_rule_engine_plugin-unified_storage_tiering.so(+0x8242c) [0x7fed66a6942c]
<6>     Offset: 0x26    Address: 0x7fed66a85236 irods::query_processor<RcComm>::execute(irods::thread_pool&, RcComm&)::'lambda'()::operator()()
<7>     Offset: 0x49    Address: 0x7fed66a85179 void boost::asio::system_executor::dispatch<irods::query_processor<RcComm>::execute(irods::thread_pool&, RcComm&)::'lambda'(), std::__1::allocator<void> >(irods::query_processor<RcComm>::execute(irods::thread_pool&, RcComm&)::'lambda'()&&, std::__1::allocator<void> const&) const
<8>     Offset: 0x19    Address: 0x7fed66a85119 void boost::asio::asio_handler_invoke<boost::asio::detail::work_dispatcher<irods::query_processor<RcComm>::execute(irods::thread_pool&, RcComm&)::'lambda'()> >(boost::asio::detail::work_dispatcher<irods::query_processor<RcComm>::execute(irods::thread_pool&, RcComm&)::'lambda'()>&, ...)
<9>     Offset: 0x87    Address: 0x7fed66a84ec7 boost::asio::detail::executor_op<boost::asio::detail::work_dispatcher<irods::query_processor<RcComm>::execute(irods::thread_pool&, RcComm&)::'lambda'()>, std::__1::allocator<void>, boost::asio::detail::scheduler_operation>::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long)
<10>    Offset: 0x307   Address: 0x7fed6dbb8227 boost::asio::detail::scheduler::do_run_one(boost::asio::detail::conditionally_enabled_mutex::scoped_lock&, boost::asio::detail::scheduler_thread_info&, boost::system::error_code const&)
<11>    Offset: 0x171   Address: 0x7fed6dbb7d31 boost::asio::detail::scheduler::run(boost::system::error_code&)
<12>    Offset: 0x25    Address: 0x7fed6dbb7bb5 boost::asio::detail::posix_thread::func<boost::asio::thread_pool::thread_function>::run()
<13>    Offset:         Address: 0x7fed6dbb7b3d /lib/libirods_server.so.4.2.12(boost_asio_detail_posix_thread_function+0xd) [0x7fed6dbb7b3d]
<14>    Offset:         Address: 0x7fed6ab94ea5 /lib64/libpthread.so.0(+0x7ea5) [0x7fed6ab94ea5]
<15>    Offset:         Address: 0x7fed6a8bdb0d /lib64/libc.so.6(clone+0x6d) [0x7fed6a8bdb0d]

]
Jul  4 14:28:30 pid:25429 remote addresses: 127.0.0.1, x.y.z.n ERROR: iRODS Exception:
    file: /irods_plugin_source/storage_tiering.cpp
    function: void irods::storage_tiering::migrate_violating_data_objects(rcComm_t *, const std::string &, const std::string &, const std::string &, const std::string &)
    line: 614
    code: -35000 (SYS_INVALID_OPR_TYPE)
    message:
        scheduling failed for [1] objects for query [SELECT DATA_NAME, COLL_NAME, USER_NAME, DATA_REPL_NUM WHERE META_DATA_ATTR_NAME = 'irods::access_time' AND META_DATA_ATTR_VALUE < '1688473590' AND META_DATA_ATTR_UNITS <> 'irods::storage_tiering::migration_scheduled' AND DATA_RESC_ID IN ('10002',)]
stack trace:
--------------

Dumping stack trace
<0>     Offset: 0x6e    Address: 0x7fed6b592b0e irods::exception::exception(long, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, unsigned int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)
<1>     Offset: 0x42    Address: 0x7fed6b592cd2 irods::exception::exception(long, boost::basic_format<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, unsigned int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)
<2>     Offset: 0x5ca   Address: 0x7fed66a642da irods::storage_tiering::migrate_violating_data_objects(RcComm*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)
<3>     Offset: 0xc3    Address: 0x7fed66a68493 irods::storage_tiering::apply_policy_for_tier_group(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)
<4>     Offset: 0x99c   Address: 0x7fed66a3a05c exec_rule_expression(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback)
<5>     Offset: 0x2f    Address: 0x7fed66a5e31f std::__1::__function::__func<irods::error (*)(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback), std::__1::allocator<irods::error (*)(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback)>, irods::error (std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback)>::operator()(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*&&, irods::callback&&)
<6>     Offset: 0xd8    Address: 0x7fed6d5c90d8 irods::pluggable_rule_engine<std::__1::tuple<> >::exec_rule_expression(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback)
<7>     Offset: 0x455   Address: 0x7fed6d5c8ca5 irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule_expression(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*)
<8>     Offset: 0x282   Address: 0x7fed6d5c86b2 rsExecRuleExpression(RsComm*, ExecRuleExpression*)
<9>     Offset: 0x42    Address: 0x7fed6d9046f2 irods::api_call_adaptor<ExecRuleExpression*>::operator()(irods::plugin_context&, RsComm*, ExecRuleExpression*)
<10>    Offset: 0x13    Address: 0x7fed6d904673 std::__1::__function::__func<irods::api_call_adaptor<ExecRuleExpression*>, std::__1::allocator<irods::api_call_adaptor<ExecRuleExpression*> >, irods::error (irods::plugin_context&, RsComm*, ExecRuleExpression*)>::operator()(irods::plugin_context&, RsComm*&&, ExecRuleExpression*&&)
<11>    Offset: 0x13d2  Address: 0x7fed6d7ae332 int irods::api_entry::call_handler<ExecRuleExpression*>(RsComm*, ExecRuleExpression*)
<12>    Offset: 0x763   Address: 0x7fed6d9b8063 rsApiHandler(RsComm*, int, BytesBuf*, BytesBuf*)
<13>    Offset: 0xa4b   Address: 0x7fed6d9ba10b readAndProcClientMsg(RsComm*, int)
<14>    Offset: 0xd08   Address: 0x7fed6d9ab038 agentMain(RsComm*)
<15>    Offset: 0x2172  Address: 0x7fed6d9a90c2 runIrodsAgentFactory(sockaddr_un)
<16>    Offset:         Address: 0x4341d8       /usr/sbin/irodsServer(main+0x16b8) [0x4341d8]
<17>    Offset:         Address: 0x7fed6a7e1555 /lib64/libc.so.6(__libc_start_main+0xf5) [0x7fed6a7e1555]
<18>    Offset:         Address: 0x432932       /usr/sbin/irodsServer() [0x432932]

Jul  4 14:29:30 pid:25525 NOTICE: use default query for [eudatCache]
Jul  4 14:29:30 pid:25525 NOTICE: found 4 objects for resc [eudatCache] with query [SELECT DATA_NAME, COLL_NAME, USER_NAME, DATA_REPL_NUM WHERE META_DATA_ATTR_NAME = 'irods::access_time' AND META_DATA_ATTR_VALUE < '1688473650' AND META_DATA_ATTR_UNITS <> 'irods::storage_tiering::migration_scheduled' AND DATA_RESC_ID IN ('10002',)] type [0]
Jul  4 14:29:30 pid:25525 NOTICE: irods::storage_tiering :: delay params for [eudatCache] - [<INST_NAME>irods_rule_engine_plugin-unified_storage_tiering-instance</INST_NAME><EF>1h DOUBLE UNTIL SUCCESS OR 6 TIMES</EF><PLUSET>28s</PLUSET>]
Jul  4 14:29:30 pid:25525 remote addresses: 127.0.0.1, x.y.z.n ERROR: data movement scheduling failed - [-808000]::[iRODS Exception:
    file: /irods_plugin_source/storage_tiering.cpp
    function: std::string irods::storage_tiering::get_metadata_for_data_object(rcComm_t *, const std::string &, const std::string &)
    line: 91
    code: -808000 (CAT_NO_ROWS_FOUND)
    message:
        no results found for object [/frank/home/robertv#igor/test_data_org_igor_20230704_01.txt] with attribute [irods::access_time]
stack trace:
--------------

Dumping stack trace
<0>     Offset: 0x6e    Address: 0x7fed6b592b0e irods::exception::exception(long, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, unsigned int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)
<1>     Offset: 0x42    Address: 0x7fed6b592cd2 irods::exception::exception(long, boost::basic_format<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, unsigned int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)
<2>     Offset: 0x491   Address: 0x7fed66a5e951 irods::storage_tiering::get_metadata_for_data_object(RcComm*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)
<3>     Offset: 0x34    Address: 0x7fed66a66bb4 irods::storage_tiering::set_migration_metadata_flag_for_object(RcComm*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)
<4>     Offset: 0x65    Address: 0x7fed66a64c55 irods::storage_tiering::queue_data_movement(RcComm*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)
<5>     Offset:         Address: 0x7fed66a6942c /usr/lib/irods/plugins/rule_engines/libirods_rule_engine_plugin-unified_storage_tiering.so(+0x8242c) [0x7fed66a6942c]
<6>     Offset: 0x26    Address: 0x7fed66a85236 irods::query_processor<RcComm>::execute(irods::thread_pool&, RcComm&)::'lambda'()::operator()()
<7>     Offset: 0x49    Address: 0x7fed66a85179 void boost::asio::system_executor::dispatch<irods::query_processor<RcComm>::execute(irods::thread_pool&, RcComm&)::'lambda'(), std::__1::allocator<void> >(irods::query_processor<RcComm>::execute(irods::thread_pool&, RcComm&)::'lambda'()&&, std::__1::allocator<void> const&) const
<8>     Offset: 0x19    Address: 0x7fed66a85119 void boost::asio::asio_handler_invoke<boost::asio::detail::work_dispatcher<irods::query_processor<RcComm>::execute(irods::thread_pool&, RcComm&)::'lambda'()> >(boost::asio::detail::work_dispatcher<irods::query_processor<RcComm>::execute(irods::thread_pool&, RcComm&)::'lambda'()>&, ...)
<9>     Offset: 0x87    Address: 0x7fed66a84ec7 boost::asio::detail::executor_op<boost::asio::detail::work_dispatcher<irods::query_processor<RcComm>::execute(irods::thread_pool&, RcComm&)::'lambda'()>, std::__1::allocator<void>, boost::asio::detail::scheduler_operation>::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long)
<10>    Offset: 0x307   Address: 0x7fed6dbb8227 boost::asio::detail::scheduler::do_run_one(boost::asio::detail::conditionally_enabled_mutex::scoped_lock&, boost::asio::detail::scheduler_thread_info&, boost::system::error_code const&)
<11>    Offset: 0x171   Address: 0x7fed6dbb7d31 boost::asio::detail::scheduler::run(boost::system::error_code&)
<12>    Offset: 0x25    Address: 0x7fed6dbb7bb5 boost::asio::detail::posix_thread::func<boost::asio::thread_pool::thread_function>::run()
<13>    Offset:         Address: 0x7fed6dbb7b3d /lib/libirods_server.so.4.2.12(boost_asio_detail_posix_thread_function+0xd) [0x7fed6dbb7b3d]
<14>    Offset:         Address: 0x7fed6ab94ea5 /lib64/libpthread.so.0(+0x7ea5) [0x7fed6ab94ea5]
<15>    Offset:         Address: 0x7fed6a8bdb0d /lib64/libc.so.6(clone+0x6d) [0x7fed6a8bdb0d]

]
Jul  4 14:29:30 pid:25525 remote addresses: 127.0.0.1, x.y.z.n ERROR: iRODS Exception:
    file: /irods_plugin_source/storage_tiering.cpp
    function: void irods::storage_tiering::migrate_violating_data_objects(rcComm_t *, const std::string &, const std::string &, const std::string &, const std::string &)
    line: 614
    code: -35000 (SYS_INVALID_OPR_TYPE)
    message:
        scheduling failed for [1] objects for query [SELECT DATA_NAME, COLL_NAME, USER_NAME, DATA_REPL_NUM WHERE META_DATA_ATTR_NAME = 'irods::access_time' AND META_DATA_ATTR_VALUE < '1688473650' AND META_DATA_ATTR_UNITS <> 'irods::storage_tiering::migration_scheduled' AND DATA_RESC_ID IN ('10002',)]
stack trace:
--------------

Dumping stack trace
<0>     Offset: 0x6e    Address: 0x7fed6b592b0e irods::exception::exception(long, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, unsigned int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)
<1>     Offset: 0x42    Address: 0x7fed6b592cd2 irods::exception::exception(long, boost::basic_format<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, unsigned int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)
<2>     Offset: 0x5ca   Address: 0x7fed66a642da irods::storage_tiering::migrate_violating_data_objects(RcComm*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)
<3>     Offset: 0xc3    Address: 0x7fed66a68493 irods::storage_tiering::apply_policy_for_tier_group(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)
<4>     Offset: 0x99c   Address: 0x7fed66a3a05c exec_rule_expression(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback)
<5>     Offset: 0x2f    Address: 0x7fed66a5e31f std::__1::__function::__func<irods::error (*)(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback), std::__1::allocator<irods::error (*)(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback)>, irods::error (std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback)>::operator()(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*&&, irods::callback&&)
<6>     Offset: 0xd8    Address: 0x7fed6d5c90d8 irods::pluggable_rule_engine<std::__1::tuple<> >::exec_rule_expression(std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*, irods::callback)
<7>     Offset: 0x455   Address: 0x7fed6d5c8ca5 irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule_expression(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MsParamArray*)
<8>     Offset: 0x282   Address: 0x7fed6d5c86b2 rsExecRuleExpression(RsComm*, ExecRuleExpression*)
<9>     Offset: 0x42    Address: 0x7fed6d9046f2 irods::api_call_adaptor<ExecRuleExpression*>::operator()(irods::plugin_context&, RsComm*, ExecRuleExpression*)
<10>    Offset: 0x13    Address: 0x7fed6d904673 std::__1::__function::__func<irods::api_call_adaptor<ExecRuleExpression*>, std::__1::allocator<irods::api_call_adaptor<ExecRuleExpression*> >, irods::error (irods::plugin_context&, RsComm*, ExecRuleExpression*)>::operator()(irods::plugin_context&, RsComm*&&, ExecRuleExpression*&&)
<11>    Offset: 0x13d2  Address: 0x7fed6d7ae332 int irods::api_entry::call_handler<ExecRuleExpression*>(RsComm*, ExecRuleExpression*)
<12>    Offset: 0x763   Address: 0x7fed6d9b8063 rsApiHandler(RsComm*, int, BytesBuf*, BytesBuf*)
<13>    Offset: 0xa4b   Address: 0x7fed6d9ba10b readAndProcClientMsg(RsComm*, int)
<14>    Offset: 0xd08   Address: 0x7fed6d9ab038 agentMain(RsComm*)
<15>    Offset: 0x2172  Address: 0x7fed6d9a90c2 runIrodsAgentFactory(sockaddr_un)
<16>    Offset:         Address: 0x4341d8       /usr/sbin/irodsServer(main+0x16b8) [0x4341d8]
<17>    Offset:         Address: 0x7fed6a7e1555 /lib64/libc.so.6(__libc_start_main+0xf5) [0x7fed6a7e1555]
<18>    Offset:         Address: 0x432932       /usr/sbin/irodsServer() [0x432932]

I suspect that it fails because there is a pound sign in the collection/path to the object. Or the user is from a remote zone. That is why it can not find the file and can not do the replication.

trel commented 1 year ago

I think it's probably that our query assumed that a username would be sufficient, rather than username AND zonename.

I think this assumption is the bug. This should be easy enough to reproduce. Thank you.

cookie33 commented 1 year ago

It is also not possible to get data back which is already in the TIER1 storage:

$ imeta ls -d /frank/home/robertv#igor/test_data_org_igor_20210818_10.txt AVUs defined for dataObj /frank/home/robertv#igor/test_data_org_igor_20210818_10.txt: attribute: irods::access_time value: 1629293237 units:

attribute: irods::storage_tiering::group value: eudat units: 1


* The remote resource and metadata on the resource:

$ ilsresc -l eudatPnfs -z frank resource name: eudatPnfs id: 10003 zone: frank type: unixfilesystem location: x.y.z.n vault: /data/eudatPnfs free space: free space time: : Never status: info: comment: create time: 01541080006: 2018-11-01.14:46:46 modify time: 01688109261: 2023-06-30.09:14:21 context: parent: parent context:

$ imeta ls -R eudatPnfs -z frank AVUs defined for resource eudatPnfs: attribute: irods::storage_tiering::group value: eudat units: 1

attribute: irods::storage_tiering::minimum_restage_tier value: true units:

attribute: irods::storage_tiering::verification value: filesystem units:


* Trying to restore a file with `iget`

$ sleep 60 ; iget /frank/home/robertv#igor/test_data_org_igor_20210818_10.txt /tmp/test.txt remote addresses: x.y.z.n ERROR: getUtil: get error for /tmp/test.txt status = -130000 SYS_INVALID_INPUT_PARAM Level 0: resc hier is null


* The error in the logfile:

Jul 5 13:46:37 pid:31003 NOTICE: use default query for [eudatCache] Jul 5 13:47:04 pid:31038 NOTICE: Failed to restage data object [/frank/home/robertv#igor/test_data_org_igor_20210818_10.txt] for resource [eudatPnfs] Exception: [iRODS Exception: file: /irods_plugin_source/storage_tiering.cpp function: std::string irods::storage_tiering::get_replica_number_for_resource(rcComm_t *, const std::string &, const std::string &) line: 745 code: -808000 (CAT_NO_ROWS_FOUND) message: failed to fetch user name and replica number stack trace:

Dumping stack trace

<0> Offset: 0x6e Address: 0x7fed6b592b0e irods::exception::exception(long, std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator > const&, unsigned int, std::__1::basic_string, std::__1::allocator > const&) <1> Offset: 0x4c5 Address: 0x7fed66a67505 irods::storage_tiering::get_replica_number_for_resource(RcComm*, std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator > const&) <2> Offset: 0x31 Address: 0x7fed66a67ed1 irods::storage_tiering::migrate_object_to_minimum_restage_tier(std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator > const&) <3> Offset: 0x1257 Address: 0x7fed66a352a7 exec_rule(std::__1::tuple<>&, std::__1::basic_string, std::__1::allocator > const&, std::__1::list >&, irods::callback) <4> Offset: 0x2e Address: 0x7fed66a5de7e std::__1::__function::__func&, std::__1::basic_string, std::__1::allocator > const&, std::__1::list >&, irods::callback), std::__1::allocator&, std::__1::basic_string, std::__1::allocator > const&, std::__1::list >&, irods::callback)>, irods::error (std::__1::tuple<>&, std::__1::basic_string, std::__1::allocator > const&, std::__1::list >&, irods::callback)>::operator()(std::__1::tuple<>&, std::__1::basic_string, std::__1::allocator > const&, std::__1::list >&, irods::callback&&) <5> Offset: 0xf6 Address: 0x7fed6d80aca6 irods::error irods::pluggable_rule_engine >::exec_rule, std::__1::allocator >&, irods::plugin_context&, DataObjInp*, portalOprOut**, BytesBuf*>(std::__1::basic_string, std::__1::allocator > const&, std::__1::tuple<>&, std::__1::basic_string, std::__1::allocator >&&&, irods::plugin_context&&&, DataObjInp*&&, portalOprOut**&&, BytesBuf*&&, irods::callback) <6> Offset: 0x85 Address: 0x7fed6d80ab35 std::__1::__function::__func, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule, std::__1::allocator >&, irods::plugin_context&, DataObjInp*, portalOprOut**, BytesBuf*>(std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator >&&&, irods::plugin_context&&&, DataObjInp*&&, portalOprOut**&&, BytesBuf*&&)::'lambda'(irods::re_pack_inp >&, std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator >&, irods::plugin_context&, DataObjInp*&&, portalOprOut**&&, BytesBuf*&&)::operator()(irods::re_pack_inp >&, std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator >&, irods::plugin_context&, DataObjInp*&&, portalOprOut**&&, BytesBuf*&&) const::'lambda'(std::__1::basic_string, std::__1::allocator > const&, irods::re_pack_inp >&, std::__1::basic_string, std::__1::allocator >&, irods::plugin_context&, DataObjInp*&&, portalOprOut**&&, BytesBuf*&&), std::__1::allocator, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule, std::__1::allocator >&, irods::plugin_context&, DataObjInp*, portalOprOut**, BytesBuf*>(std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator >&&&, irods::plugin_context&&&, DataObjInp*&&, portalOprOut**&&, BytesBuf*&&)::'lambda'(irods::re_pack_inp >&, std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator >&, irods::plugin_context&, DataObjInp*&&, portalOprOut**&&, BytesBuf*&&)::operator()(irods::re_pack_inp >&, std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator >&, irods::plugin_context&, DataObjInp*&&, portalOprOut**&&, BytesBuf*&&) const::'lambda'(std::__1::basic_string, std::__1::allocator > const&, irods::re_pack_inp >&, std::__1::basic_string, std::__1::allocator >&, irods::plugin_context&, DataObjInp*&&, portalOprOut**&&, BytesBuf*&&)>, irods::error (std::__1::basic_string, std::__1::allocator > const&, irods::re_pack_inp >&, std::__1::basic_string, std::__1::allocator >&, irods::plugin_context&, DataObjInp*&&, portalOprOut**&&, BytesBuf*&&)>::operator()(std::__1::basic_string, std::__1::allocator > const&, irods::re_pack_inp >&, std::__1::basic_string, std::__1::allocator >&, irods::plugin_context&, DataObjInp*&&, portalOprOut**&&, BytesBuf*&&) <7> Offset: 0x193 Address: 0x7fed6d80b693 irods::error irods::dynamic_operation_execution_manager, RuleExecInfo*, (irods::rule_execution_manager_pack)1>::call, std::__1::allocator > const&, irods::re_pack_inp >&, std::__1::basic_string, std::__1::allocator >&, irods::plugin_context&, DataObjInp*&&, portalOprOut**&&, BytesBuf*&&)>, std::__1::basic_string, std::__1::allocator > const&, irods::re_pack_inp >&, std::__1::basic_string, std::__1::allocator >&, irods::plugin_context&, DataObjInp*, portalOprOut**, BytesBuf*>(std::__1::basic_string, std::__1::allocator >, std::__1::basic_string, std::__1::allocator >, std::__1::function, std::__1::allocator > const&, irods::re_pack_inp >&, std::__1::basic_string, std::__1::allocator >&, irods::plugin_context&, DataObjInp*&&, portalOprOut**&&, BytesBuf*&&)>, std::__1::basic_string, std::__1::allocator > const&&&, irods::re_pack_inp >&&&, std::__1::basic_string, std::__1::allocator >&&&, irods::plugin_context&&&, DataObjInp*&&, portalOprOut**&&, BytesBuf*&&)::'lambda'()::operator()() const <8> Offset: 0xce Address: 0x7fed6d80a7de irods::error irods::dynamic_operation_execution_manager, RuleExecInfo*, (irods::rule_execution_manager_pack)1>::call, std::__1::allocator > const&, irods::re_pack_inp >&, std::__1::basic_string, std::__1::allocator >&, irods::plugin_context&, DataObjInp*&&, portalOprOut**&&, BytesBuf*&&)>, std::__1::basic_string, std::__1::allocator > const&, irods::re_pack_inp >&, std::__1::basic_string, std::__1::allocator >&, irods::plugin_context&, DataObjInp*, portalOprOut**, BytesBuf*>(std::__1::basic_string, std::__1::allocator >, std::__1::basic_string, std::__1::allocator >, std::__1::function, std::__1::allocator > const&, irods::re_pack_inp >&, std::__1::basic_string, std::__1::allocator >&, irods::plugin_context&, DataObjInp*&&, portalOprOut**&&, BytesBuf*&&)>, std::__1::basic_string, std::__1::allocator > const&&&, irods::re_pack_inp >&&&, std::__1::basic_string, std::__1::allocator >&&&, irods::plugin_context&&&, DataObjInp*&&, portalOprOut**&&, BytesBuf*&&) <9> Offset: 0x103 Address: 0x7fed6d80a3b3 irods::error irods::rule_engine_context_manager, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule, std::__1::allocator >&, irods::plugin_context&, DataObjInp*, portalOprOut**, BytesBuf*>(std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator >&&&, irods::plugin_context&&&, DataObjInp*&&, portalOprOut**&&, BytesBuf*&&)::'lambda'(irods::re_pack_inp >&, std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator >&, irods::plugin_context&, DataObjInp*&&, portalOprOut**&&, BytesBuf*&&)::operator()(irods::re_pack_inp >&, std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator >&, irods::plugin_context&, DataObjInp*&&, portalOprOut**&&, BytesBuf*&&) const <10> Offset: 0x100 Address: 0x7fed6d80a030 irods::error irods::control, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule, std::__1::allocator >&, irods::plugin_context&, DataObjInp*, portalOprOut**, BytesBuf*>(std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator >&&&, irods::plugin_context&&&, DataObjInp*&&, portalOprOut**&&, BytesBuf*&&)::'lambda'(irods::re_pack_inp >&, std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator >&, irods::plugin_context&, DataObjInp*&&, portalOprOut**&&, BytesBuf*&&), irods::error irods::rule_engine_context_manager, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule, std::__1::allocator >&, irods::plugin_context&, DataObjInp*, portalOprOut**, BytesBuf*>(std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator >&&&, irods::plugin_context&&&, DataObjInp*&&, portalOprOut**&&, BytesBuf*&&)::'lambda'(std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator >&, irods::plugin_context&, DataObjInp*&&, portalOprOut**&&, BytesBuf*&&), std::__1::tuple<>, std::__1::basic_string, std::__1::allocator >&, irods::plugin_context&, DataObjInp*, portalOprOut**, BytesBuf*>(std::__1::list >, std::__1::allocator > > >&, irods::error irods::rule_engine_context_manager, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule, std::__1::allocator >&, irods::plugin_context&, DataObjInp*, portalOprOut**, BytesBuf*>(std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator >&&&, irods::plugin_context&&&, DataObjInp*&&, portalOprOut**&&, BytesBuf*&&)::'lambda'(irods::re_pack_inp >&, std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator >&, irods::plugin_context&, DataObjInp*&&, portalOprOut**&&, BytesBuf*&&), irods::error irods::rule_engine_context_manager, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule, std::__1::allocator >&, irods::plugin_context&, DataObjInp*, portalOprOut**, BytesBuf*>(std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator >&&&, irods::plugin_context&&&, DataObjInp*&&, portalOprOut**&&, BytesBuf*&&)::'lambda'(std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator >&, irods::plugin_context&, DataObjInp*&&, portalOprOut**&&, BytesBuf*&&), std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator >&&&, irods::plugin_context&&&, DataObjInp*&&, portalOprOut**&&, BytesBuf*&&) <11> Offset: 0x629 Address: 0x7fed6d809ac9 irods::error irods::api_entry::invoke_policy_enforcement_point(irods::rule_engine_context_manager, RuleExecInfo*, (irods::rule_execution_manager_pack)0>, irods::plugin_context, std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator > const&, DataObjInp*, portalOprOut**, BytesBuf*) <12> Offset: 0x7ea Address: 0x7fed6d6fa97a int irods::api_entry::call_handler(RsComm*, DataObjInp*, portalOprOut**, BytesBuf*) <13> Offset: 0x746 Address: 0x7fed6d9b8046 rsApiHandler(RsComm*, int, BytesBuf*, BytesBuf*) <14> Offset: 0xa4b Address: 0x7fed6d9ba10b readAndProcClientMsg(RsComm*, int) <15> Offset: 0xd08 Address: 0x7fed6d9ab038 agentMain(RsComm*) <16> Offset: 0x2172 Address: 0x7fed6d9a90c2 runIrodsAgentFactory(sockaddr_un) <17> Offset: Address: 0x4341d8 /usr/sbin/irodsServer(main+0x16b8) [0x4341d8] <18> Offset: Address: 0x7fed6a7e1555 /lib64/libc.so.6(__libc_start_main+0xf5) [0x7fed6a7e1555] <19> Offset: Address: 0x432932 /usr/sbin/irodsServer() [0x432932] ```
cookie33 commented 9 months ago

A fix was tried at #228 using iRODS 4.3.1. But it still does not execute as the corrrect user/remote zone it seems. Can you give us a pointer on how to fix it?

cookie33 commented 9 months ago

fixed in #228.

alanking commented 9 months ago

Thanks, @cookie33!