CAT_NO_ACCESS_PERMISSION and USER_PACKSTRUCT_INPUT_ERR on open/read/close activity

iRODS: 4.2.7, Linux Plugin: 2.7.0

A local user on a (federated) zone, with read-only access to data on that zone used baton-get to read a file. baton-get uses the iRODS open/read/close API, which triggers irods::access_time updates in this version of the plugin. The same user also uses iget (which does not trigger irods::access_time updates in this version of the plugin)

Expected:

The file to be downloaded successfully with both clients, with only baton-get triggering an access time update.

Observed:

iget successfully downloads the file and irods::access_time is not set.

baton-get fails when the update of irods::acces_time is attempted. Setting access time fails with CAT_NO_ACCESS_PERMISSION, but an underlying USER_PACKSTRUCT_INPUT_ERR.

xxxxxx@xxxxxxx:~/xxxxx/genotype_data_check$ baton-get --avu < qc_plex_list_060522_oneplate.txt 
remote addresses: xxx.xxx.xxx.xxx ERROR: unpackNonpointerItem: strlen of msg > dim size, content: iRODS Exception:
    file: /irods_plugin/libirods_rule_engine_plugin-apply_access_time.cpp
    function: void (anonymous namespace)::update_access_time_for_data_object(rsComm_t *, const std::string &, const std::string &)
    line: 36
    code: -818000 (CAT_NO_ACCESS_PERMISSION)
    message:
        failed to set access time for [xxxxxx/xxxxxx.csv]
        : [-]   /irods/server/re/include/irods_re_plugin.hpp:325:irods::error irods::dynamic_operation_execution_manager<std::__1::tuple<>, RuleExecInfo *, irods::rule_execution_manager_pack::DONT_AUDIT_RULE>::call(std::string, std::string, OP, As &&...) [T = std::__1::tuple<>, C = RuleExecInfo *, Audit = irods::rule_execution_manager_pack::DONT_AUDIT_RULE, OP = std::__1::function<irods::error (const std::__1::basic_string<char> &, irods::re_pack_inp<std::__1::tuple<> > &, irods::unpack &&)>, As = <const std::__1::basic_string<char> &, irods::re_pack_inp<std::__1::tuple<> > &, irods::unpack>] :  status [CAT_NO_ACCESS_PERM 
remote addresses: 10.160.2.194 ERROR: readAndProcApiReply:unpackStruct error. status = -308000 status = -308000 USER_PACKSTRUCT_INPUT_ERR
{"collection": "/xxxxxx, "data_object": "xxxxxx.csv", "error": {"message": "Failed to close data object: '/xxxxxx/xxxxxx.csv' error -818000 CAT_NO_ACCESS_PERMISSION", "code": -818000}}
2022-05-06T10:34:06Z WARN Processed 1 items with 1 errors

In this case is CAT_NO_ACCESS_PERMISSION genuine, or an artefact of failure due to USER_PACKSTRUCT_INPUT_ERR?

N.B. (sorry for the pedantry)

The file to be downloaded successfully with both clients, with only baton-get triggering an access time update.

Is only expected because of https://github.com/irods/irods_capability_storage_tiering/issues/200 - we would normally expect iget to also trigger an access time update.

While this error was reproducible multiple times on the day, this is no longer the case. These all now succeed:

python-irodsclient session.data_objects.get()
baton-get --avu
baton-get (i.e. no metadata)

Trying to manually remove irods::access_time first does not change the outcome. However, I'm not sure that I can effectively remove it with imeta rm -d because when I look with imeta ls -d, it is immediately restored.

Maybe the error occurs only on the initial absence of irods::access_time?

First, that seems like great news.

Second, imeta ls -d adds irods::access_time?! That feels... surprising.

My error. It had just got a new access time, so my imeta rm -d had no effect. Confirmed and double-checked that I'm removing it this time. I also made the error of calling session.data_objects.get() with only its first argument when I should have used the two argument version.

Now things are a little different:

As rodsadmin on the federated zone:

baton-get --avu succeeds and updates irods::access_time
session.data_object_get(remote, local) was not tried (because I don't have a convenient install at the moment)

As a regular user on my home zone (federated to the above):

baton-get --avu succeeds but does not update irods::access_time

session.data_object_get(remote, local) fails with:

File "/software/team117/kdj/lib/miniconda/envs/irods-4.2.7/lib/python3.9/site-packages/irods/manager/data_object_manager.py", line 122, in get
self._download(path, local_path, num_threads = num_threads, **options)
File "/software/team117/kdj/lib/miniconda/envs/irods-4.2.7/lib/python3.9/site-packages/irods/manager/data_object_manager.py", line 99, in _download
with open(local_file, 'wb') as f, self.open(obj, 'r', **options) as o:
File "/software/team117/kdj/lib/miniconda/envs/irods-4.2.7/lib/python3.9/site-packages/irods/manager/data_object_manager.py", line 326, in open
desc = conn.recv().int_info
File "/software/team117/kdj/lib/miniconda/envs/irods-4.2.7/lib/python3.9/site-packages/irods/connection.py", line 128, in recv
raise get_exception_by_code(msg.int_info, err_msg)
irods.exception.SYS_RESC_DOES_NOT_EXIST: None

which I guess might be because it's looking at the wrong zone. I'm calling it so:

    try:
        env_file = os.environ['IRODS_ENVIRONMENT_FILE']
    except KeyError:
        env_file = os.path.expanduser('~/.irods/irods_environment.json')

    ssl_context = ssl.create_default_context(purpose=ssl.Purpose.SERVER_AUTH,
                                             cafile=None, capath=None, cadata=None)
    ssl_settings = {'ssl_context': ssl_context}

    with iRODSSession(irods_env_file=env_file, zone="seq", **ssl_settings) as session:
        obj = session.data_objects.get("/seq/fluidigm/00/13/aa/1662457010/02/4b/ab/S091_1662457010.csv",
                                       "foo.csv")

I've tried both with and without the zone="seq" argument. I'm using python-irodsclient 1.1.3 from pip.

I can now reproduce this 100% of the time, including with the Pyton API. Both a plain baton-get and baton-get --avu give the same error:

remote addresses: xxx.xxx.xxx.xxx ERROR: unpackNonpointerItem: strlen of msg > dim size, content: iRODS Exception:
    file: /irods_plugin/libirods_rule_engine_plugin-apply_access_time.cpp
    function: void (anonymous namespace)::update_access_time_for_data_object(rsComm_t *, const std::string &, const std::string &)
    line: 36
    code: -818000 (CAT_NO_ACCESS_PERMISSION)
    message:
        failed to set access time for [/seq/fluidigm/00/13/aa/1662457010/02/4b/ab/S091_1662457010.csv]
        : [-]   /irods/server/re/include/irods_re_plugin.hpp:325:irods::error irods::dynamic_operation_execution_manager<std::__1::tuple<>, RuleExecInfo *, irods::rule_execution_manager_pack::DONT_AUDIT_RULE>::call(std::string, std::string, OP, As &&...) [T = std::__1::tuple<>, C = RuleExecInfo *, Audit = irods::rule_execution_manager_pack::DONT_AUDIT_RULE, OP = std::__1::function<irods::error (const std::__1::basic_string<char> &, irods::re_pack_inp<std::__1::tuple<> > &, irods::unpack &&)>, As = <const std::__1::basic_string<char> &, irods::re_pack_inp<std::__1::tuple<> > &, irods::unpack>] :  status [CAT_NO_ACCESS_PERM
remote addresses: xxx.xxx.xxx.xxx ERROR: readAndProcApiReply:unpackStruct error. status = -308000 status = -308000 USER_PACKSTRUCT_INPUT_ERR
{"coll": "/seq/fluidigm/00/13/aa/1662457010/02/4b/ab/", "obj": "S091_1662457010.csv", "error": {"message": "Failed to close data object: '/seq/fluidigm/00/13/aa/1662457010/02/4b/ab/S091_1662457010.csv' error -818000 CAT_NO_ACCESS_PERMISSION", "code": -818000}}
2022-05-12T12:13:25Z WARN Processed 1 items with 1 errors

Note that in this case (connecting to the same zone as before, but from a different host) we no longer see the packStruct error.

With the Python API:

Traceback (most recent call last):
  File "/home/xxx/kdj_tmp/./test.py", line 24, in <module>
    main()
  File "/home/xxx/kdj_tmp/./test.py", line 19, in main
    obj = session.data_objects.get("/seq/fluidigm/00/13/aa/1662457010/02/4b/ab/S091_1662457010.csv",
  File "/home/xxx/kdj_tmp/miniconda/envs/irods_test/lib/python3.9/site-packages/irods/manager/data_object_manager.py", line 122, in get
    self._download(path, local_path, num_threads = num_threads, **options)
  File "/home/xxx/kdj_tmp/miniconda/envs/irods_test/lib/python3.9/site-packages/irods/manager/data_object_manager.py", line 108, in _download
    f.write(chunk)
  File "/home/xxx/kdj_tmp/miniconda/envs/irods_test/lib/python3.9/site-packages/irods/data_object.py", line
174, in close
    self.conn.close_file(self.desc, **self.options)
  File "/home/xxx/kdj_tmp/miniconda/envs/irods_test/lib/python3.9/site-packages/irods/connection.py", line 605, in close_file
    self.recv()
  File "/home/xxx/kdj_tmp/miniconda/envs/irods_test/lib/python3.9/site-packages/irods/connection.py", line 128, in recv
    raise get_exception_by_code(msg.int_info, err_msg)
irods.exception.CAT_NO_ACCESS_PERMISSION: iRODS Exception:
    file: /irods_plugin/libirods_rule_engine_plugin-apply_access_time.cpp
    function: void (anonymous namespace)::update_access_time_for_data_object(rsComm_t *, const std::string &, const
std::string &)
    line: 36
    code: -818000 (CAT_NO_ACCESS_PERMISSION)
    message:
        failed to set access time for [/seq/fluidigm/00/13/aa/1662457010/02/4b/ab/S091_1662457010.csv]
        : [-]   /irods/server/re/include/irods_re_plugin.hpp:325:irods::error irods::dynamic_operation_execution_manager<std::__1::tuple<>, RuleExecInfo *, irods::rule_execution_manager_pack::DONT_AUDIT_RULE>::call(std::string, std::string, OP, As &&...) [T = std::__1::tuple<>, C = RuleExecInfo *, Audit = irods::rule_execution_manager_pack::DONT_AUDIT_RULE, OP = std::__1::function<irods::error (const std::__1::basic_string<char> &, irods::re_pack_inp<std::__1::tuple<> > &, irods::unpack &&)>, As = <const std::__1::basic_string<char> &, irods::re_pack_inp<std::__1::tuple<> > &, irods::unpack>] :  status [CAT_NO_ACCESS_PERM

So the core issue looking more like it relates to permissions and the packStruct is separate. The iRODS user these clients are running as, is a read-only account because of a policy of least privilege.

Excellent. Thanks for the detective work.

If this is due to simply matching the wrong username (b/c federation), then this is a symptom/duplicate of https://github.com/irods/irods_capability_storage_tiering/issues/187.

If this is more subtle and due to the read-only nature of the connected user, then we will need to think slightly differently about how to address it.

irods / irods_capability_storage_tiering

CAT_NO_ACCESS_PERMISSION and USER_PACKSTRUCT_INPUT_ERR on open/read/close activity #203