`pep_database_open_*` cause SYS_HEADER_READ_LEN_ERR

[mstfdkmn]

Hi,

We are trying/testing/using a forked version (not changed much in the core part) of this plugin at this moment. I encountered something not normal. Looks like it might be related to #124. That is;

If pep_database_open_* are defined in the "pep_regex_to_match" of the plugin, the client gets [irods@dev-u0137480 irods]$ ils remote addresses: 10.0.2.100 ERROR: _rcConnect: connectToRhost error, server on dev-u0137480.irods.t.icts.kuleuven.be:1247 is probably down status = -4000 SYS_HEADER_READ_LEN_ERR and the logs prints the followings:

irods stdout | {"log_category":"agent_factory","log_level":"error","log_message":"Agent process [2774] exited with status [11].","server_host":"dev-u0137480.irods.t.icts.kuleuven.be","server_pid":155,"server_timestamp":"2024-01-09T08:41:45.789Z","server_type":"agent_factory","server_zone":"u0137480"}
irods stdout | {"log_category":"server","log_level":"critical","log_message":" 0# stacktrace_signal_handler in /lib/libirods_server.so.4.3.1\n 1# 0x00007FBA54212CF0 in /lib64/libpthread.so.0\n 2# 0x00007FBA53ECBE25 in /lib64/libc.so.6\n 3# void nlohmann::detail::external_constructor<(nlohmann::detail::value_t)3>::construct<nlohmann::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long, unsigned long, double, std::__1::allocator, nlohmann::adl_serializer, std::__1::vector<unsigned char, std::__1::allocator<unsigned char> > >, char [64], 0>(nlohmann::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long, unsigned long, double, std::__1::allocator, nlohmann::adl_serializer, std::__1::vector<unsigned char, std::__1::allocator<unsigned char> > >&, char const (&) [64]) in /usr/sbin/irodsServer\n 4# 0x00007FBA4DE3DF0A\n 5# 0x00007FBA4DE7589E\n 6# irods::error irods::pluggable_rule_engine<std::__1::tuple<> >::exec_rule<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*&>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::tuple<>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*&, irods::callback) in /lib/libirods_server.so.4.3.1\n 7# std::__1::__function::__func<irods::error irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*&>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*&)::'lambda'(irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*&)::operator()(irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*&) const::'lambda'(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*&), std::__1::allocator<irods::error irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*&>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*&)::'lambda'(irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*&)::operator()(irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*&) const::'lambda'(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*&)>, irods::error (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*&)>::operator()(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*&) in /lib/libirods_server.so.4.3.1\n 8# irods::error irods::dynamic_operation_execution_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)1>::call<std::__1::function<irods::error (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*&)>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*&>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::function<irods::error (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*&)>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*&)::'lambda'()::operator()() const in /lib/libirods_server.so.4.3.1\n 9# irods::error irods::dynamic_operation_execution_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)1>::call<std::__1::function<irods::error (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*&)>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*&>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::function<irods::error (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*&)>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*&) in /lib/libirods_server.so.4.3.1\n10# irods::error irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*&>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*&)::'lambda'(irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*&)::operator()(irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*&) const in /lib/libirods_server.so.4.3.1\n11# irods::error irods::control<irods::error irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*&>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*&)::'lambda'(irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*&), irods::error irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*&>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*&)::'lambda'(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*&), std::__1::tuple<>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*&>(std::__1::list<irods::re_pack_inp<std::__1::tuple<> >, std::__1::allocator<irods::re_pack_inp<std::__1::tuple<> > > >&, irods::error irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*&>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*&)::'lambda'(irods::re_pack_inp<std::__1::tuple<> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*&), irods::error irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>::exec_rule<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*&>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*&)::'lambda'(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*&), std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, irods::plugin_context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*&) in /lib/libirods_server.so.4.3.1\n12# irods::error irods::plugin_base::invoke_policy_enforcement_point<>(irods::rule_engine_context_manager<std::__1::tuple<>, RuleExecInfo*, (irods::rule_execution_manager_pack)0>, irods::plugin_context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) in /lib/libirods_server.so.4.3.1\n13# irods::error irods::plugin_base::call<>(RsComm*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, boost::shared_ptr<irods::first_class_object>) in /lib/libirods_server.so.4.3.1\n14# chlOpen() in /lib/libirods_server.so.4.3.1\n15# connectRcat() in /lib/libirods_server.so.4.3.1\n16# initServerInfo(int, RsComm*) in /lib/libirods_server.so.4.3.1\n17# initAgent(int, RsComm*) in /lib/libirods_server.so.4.3.1\n18# runIrodsAgentFactory(sockaddr_un) in /lib/libirods_server.so.4.3.1\n19# main::$_5::operator()() const in /usr/sbin/irodsServer\n20# main in /usr/sbin/irodsServer\n21# __libc_start_main in /lib64/libc.so.6\n22# _start in /usr/sbin/irodsServer\n","server_host":"dev-u0137480.irods.t.icts.kuleuven.be","server_pid":146,"server_timestamp":"2024-01-09T08:41:46.191Z","server_type":"server","server_zone":"u0137480","stacktrace_agent_pid":"2774","stacktrace_timestamp_epoch_milliseconds":"779","stacktrace_timestamp_epoch_seconds":"1704789705","stacktrace_timestamp_utc":"2024-01-09T08:41:45.779Z"}
irods stdout | {"log_category":"legacy","log_level":"error","log_message":"_rcConnect: connectToRhost error, server on dev-u0137480.irods.t.icts.kuleuven.be:1247 is probably down status = -4000 SYS_HEADER_READ_LEN_ERR","server_host":"dev-u0137480.irods.t.icts.kuleuven.be","server_pid":146,"server_timestamp":"2024-01-09T08:41:46.742Z","server_type":"server","server_zone":"u0137480"}
irods stdout | {"log_category":"agent_factory","log_level":"error","log_message":"Agent process [2808] exited with status [11].","server_host":"dev-u0137480.irods.t.icts.kuleuven.be","server_pid":155,"server_timestamp":"2024-01-09T08:41:46.759Z","server_type":"agent_factory","server_zone":"u0137480"}
irods stdout | {"log_category":"server","log_level":"error","log_message":"Caught exception in migrate_delay_server(): iRODS Exception:\n    file: /irods_source/lib/core/src/client_connection.cpp\n    function: void irods::experimental::client_connection::only_connect(const std::string &, const int, const irods::experimental::fully_qualified_username &)\n    line: 188\n    code: -305000 (USER_SOCK_CONNECT_ERR)\n    message:\n        Connect error\nstack trace:\n--------------\n 0# irods::stacktrace::dump() const in /lib/libirods_common.so.4.3.1\n 1# irods::exception::assemble_full_display_what() const in /lib/libirods_common.so.4.3.1\n 2# irods::exception::what() const in /lib/libirods_common.so.4.3.1\n 3# std::__1::__function::__func<initServerMain(RsComm*, bool, bool)::$_13, std::__1::allocator<initServerMain(RsComm*, bool, bool)::$_13>, void ()>::operator()() in /usr/sbin/irodsServer\n 4# irods::experimental::cron::cron_task::operator()() in /usr/sbin/irodsServer\n 5# irods::experimental::cron::cron::run() in /usr/sbin/irodsServer\n 6# void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, main::$_10> >(void*) in /usr/sbin/irodsServer\n 7# 0x00007FBA542081CA in /lib64/libpthread.so.0\n 8# clone in /lib64/libc.so.6\n\n","server_host":"dev-u0137480.irods.t.icts.kuleuven.be","server_pid":146,"server_timestamp":"2024-01-09T08:41:46.795Z","server_type":"server","server_zone":"u0137480"}

I am wondering whether someone else has encountered the same with the original plugin. If you can test this when you have time, that would be great. Thanks.

Add "pep_regex_to_match": "pep_database_open_post" in the rule engine part in the server_config.json file.

[korydraughn]

That regex looks a little questionable. Try using pep_database_open_.+ or pep_database_open_.* instead and let us know if that results in the same error.

[mstfdkmn]

Yes, they resulted in the same error. Even a general pattern such as pep_.+ and pep_.* for regex fail.

[korydraughn]

Okay. We will investigate.

[korydraughn]

I have not been able to reproduce this using the 4.3.1.0 or 4.3.2.0 plugins. All reported PEPs are being captured as expected. The icommands are not experiencing errors.

Everything is working as expected. Closing.

[mstfdkmn]

In our case (on my end hope everything correct) pep_databaseopen* looks not capturing anything (but yes doenst cause the server down).