add microservice libmsiobjjpeg

[DOC-MEX]

General question. I see that it is recommended to install metalnx-msi. However, it seems that that repo is not actively develop (it last commit 2018). I have installed metalnx-web and it seems to work fine with IRODS 4.3 in an Ubuntu 20 machine. I want to capture simple metadata using the python rule engine and for that I would need to add the metalnx-msi. I just want to be sure that it would work with Irods 4.3 and Ubuntu 20 (The documentation mentions Ubuntu 14! )

[trel]

No, the metalnx-msi repository is no longer maintained. We are waiting for metalnx 3.0 before we mark the 3-4 other packages as archived. If you need to add metadata to iRODS, please consider any of the other maintained tools/libraries to do that.

for that I would need to add the metalnx-msi

No - this is not needed. With the Python Rule Engine Plugin, you can write rules for the server in Python, and then have them fire when you need them.

[DOC-MEX]

I am using the python rule engine to extract metadata automatically from images immediately after using "iput". This works fine on the terminal. However, when I upload an image through Metalnx, the rule is not applied, no metada is extracted. The irods logs complain about a missing file libmsiobjjpeg_extract.so in lib/irods/plugins/microservices. libmsiobjjpeg_extract.so is generated by metalnx-msi, but it that is no longer the way to do it, how can I keep the rule extracting the metadata in Metalnx after uploading a photo?

[trel]

Metalnx is just a client. If the server is successfully doing work when an iput occurs, then we should be able to see the same behavior when Metalnx uploads a file. Can you please share the entire error message you see in the logs? That will help determine why that microservice is being requested.

[DOC-MEX]

When I upload a test image using iput on the terminal, the logs just show this:

{"log_category":"legacy","log_facility":"local0","log_level":"info","log_message":"writeLine: inString = PYTHON - pep_api_data_obj_put_post() complete\n","request_api_name":"DATA_OBJ_PUT_AN","request_api_number":606,"request_api_version":"d","request_client_user":"rods","request_host":"127.0.0.1","request_proxy_user":"rods","request_release_version":"rods4.3.0","server_host":"grassroots-development","server_pid":880305,"server_timestamp":"2023-09-18T14:30:35.509Z","server_type":"agent"}

And the metadata is extracted.

But when I upload it through metalnx I get these error:

{"log_category":"legacy","log_facility":"local0","log_level":"error","log_message":"[exec_rule_text:695] Improperly formatted rule text in Python rule engine plugin","request_api_name":"EXEC_MY_RULE_AN","request_api_number":625,"request_api_version":"d","request_client_user":"rods","request_host":"172.18.0.2","request_proxy_user":"rods","request_release_version":"rods3.2","server_host":"grassroots-development","server_pid":881221,"server_timestamp":"2023-09-18T14:37:19.064Z","server_type":"agent"} {"log_category":"legacy","log_facility":"local0","log_level":"error","log_message":"[-]\t/irods_source/server/re/src/irods_re_structs.cpp:54:int actionTableLookUp(irods::ms_table_entry &, char ) : status [PLUGIN_ERROR_MISSING_SHARED_OBJECT] errno [] -- message [Failed to create ms plugin entry.]\n\t[-]\t/irods_source/server/re/src/irods_ms_plugin.cpp:254:irods::error irods::load_microservice_plugin(irods::ms_table &, const std::string &) : status [PLUGIN_ERROR_MISSING_SHARED_OBJECT] errno [] -- message [Failed to create ms plugin entry.]\n\t\t[-]\t/irods_source/lib/core/include/irods/irods_load_plugin.hpp:157:irods::error irods::load_plugin(PluginType &, const std::string &, const std::string &, const std::string &, const Ts &...) [PluginType = irods::ms_table_entry, Ts = <char [4]>] : status [PLUGIN_ERROR_MISSING_SHARED_OBJECT] errno [] -- message [shared library does not exist [/usr/lib/irods/plugins/microservices/libmsiobjjpeg_extract.so]]\n\n","request_api_name":"EXEC_MY_RULE_AN","request_api_number":625,"request_api_version":"d","request_client_user":"rods","request_host":"172.18.0.2","request_proxy_user":"rods","request_release_version":"rods3.2","server_host":"grassroots-development","server_pid":881221,"server_timestamp":"2023-09-18T14:37:19.064Z","server_type":"agent"} {"log_category":"legacy","log_facility":"local0","log_level":"error","log_message":"executeRuleAction Failed for msiobjjpeg_extract status = -1102000 NO_MICROSERVICE_FOUND_ERR","request_api_name":"EXEC_MY_RULE_AN","request_api_number":625,"request_api_version":"d","request_client_user":"rods","request_host":"172.18.0.2","request_proxy_user":"rods","request_release_version":"rods3.2","server_host":"grassroots-development","server_pid":881221,"server_timestamp":"2023-09-18T14:37:19.064Z","server_type":"agent"}

[korydraughn]

The error message indicates you're calling a MSI that doesn't exist.

Can you share the policy that is doing the extraction?

[DOC-MEX]

I am using the example from the irods training repo (_https://github.com/irods/irods_training/tree/main/beginner_): Core.py and training.re https://raw.githubusercontent.com/irods/irods_training/ugm2022/beginner/training.re https://raw.githubusercontent.com/irods/irods_training/ugm2022/beginner/core.py

How can I generate the missing file(s) that Metalnx is asking without using metalnx-msi which is no longer maintained? As I mention, when I upload directly in the terminal, the metadata is extracted from the image with no problem, so the pyrhon rule engine is working OK.

[korydraughn]

The rule code you shared is triggered by PUT operations only.

Metalnx most likely uses the streaming API rather than the PUT API to upload data. You'll have to use different PEPs to make it work.

# We'll store the logical path of the opened data object in this variable.
obj_path = ''

# Triggered if Metalnx is using the original open API call.
# Notice the "_post" suffix. That means it is only triggered if
# the open was successful.
pep_api_data_obj_open_post(args, cb, rei):
    # This will write a message to the server log if the PEP is triggered.
    cb.writeLine('serverLog', '>>>>>> In [pep_api_data_obj_open_post] <<<<<<')

    data_obj_inp = rule_args[2]

    # Capture the logical path of the data object that's being opened.
    # This is one way to share state across PEP boundaries.
    global obj_path
    obj_path = str(data_obj_inp.objPath)

# Triggered if Metalnx is using the original close API call.
pep_api_data_obj_close_post(args, cb, rei):
    # This will write a message to the server log if the PEP is triggered.
    cb.writeLine('serverLog', '>>>>>> In [pep_api_data_obj_close_post] <<<<<<')

    global obj_path
    # Use "obj_path" to do work.

If those don't work, try looking at the following:

• pep_api_replica_open_post
• pep_api_replica_close_post

You can find all the PEPs I've mentioned and more at https://docs.irods.org/4.3.0/plugins/dynamic_policy_enforcement_points/.

[DOC-MEX]

Many thanks!

The logs errors I showed before were caused because I set to true this parameter:

 # disable automatic detection and running of rules on upload
 metalnx.enable.upload.rules=false

in the metalnx.properties file, it was not caused by the policy itself.

Now I can clearly verify that pep_api_data_obj_put_post is not triggered when uploading a file through metalnx, but, for example, the call "pep_api_data_obj_close_post" is triggered. I will use that one to implement the extraction of metadata.

[korydraughn]

Excellent.