two ways omero / nfsrods

[glyg]

Hi,

Thank you for the sandbox example, which runs smoothly! I have a complementary use-case:

Importing data in OMERO can be done inplace meaning that the ManagedRepository directory only contains a symbolic link (or a hard link but I'm not sure it's possible here) to the imported data. We plan to use that feature and use iRODS to transport data from the microscope to the data center and import it only then into OMERO without extra copies.

While I think this looks feasible, I have questions:

• How would the mount points be precisely managed? Should there be one mount point per user? Can this be automated?
• On the OMERO server, the /DATA directory needs to be accessible as read only by the omero-server user — this is manageable through group permission, right?
• Only the omero-server user accesses both /DATA and /OMERO on the OMERO server, do we need to have the other ones (user1, user2, etc) created there?

I am a complete newbie with iRODS, so a lot of all that is a bit obscure, sorry if I am missing obvious stuff, and I am happy to clarify if my sketch is too obscure!

Best

Guillaume

[trel]

Ah, I see. This is the equivalent of ireg in the iRODS world. Just registering an existing file in-place, with no data movement or additional disk space.

To attempt to answer your questions...

How would the mount points be precisely managed? Should there be one mount point per user? Can this be automated?

A single NFSRODS mountpoint can handle multiple users without trouble. The linux user who steps into the mountpoint must also exist in the iRODS namespace, and the NFSv4 ACLs are honored/mapped to iRODS permissions.

So, more investigation / discussion is needed to understand the 'normal' workflow of an inplace OMERO import - and whether any user can perform this, or the admin can perform this on behalf of a user, or whether it can be automated with a button in the OMERO interface in some way...

On the OMERO server, the /DATA directory needs to be accessible as read only by the omero-server user — this is manageable through group permission, right?

I think so - we should be able to manage permissions across the mountpoint as desired. It's not yet clear to me what that desire is, exactly. That is the discussion to be had it seems.

Only the omero-server user accesses both /DATA and /OMERO on the OMERO server, do we need to have the other ones (user1, user2, etc) created there?

I think this is part of the same discussion. The omero-server linux account must have a one-to-one account in the iRODS namespace. If no other linux users are walking into the mountpoint, then no - we should be good without creating all the other users. But, we haven't written down any use cases yet - that appears to be the next step(s).

[glyg]

H @trel thank you for the detailed answer. So to be more precise on the import inplace workflow:

1. "user 1" creates a dataset at the microscope, Usually user authentication at the microscope isn't related to omero and irods.

2. the user logs into an iRODS client as user1 to upload the dataset (maybe through an NFS share on the microscope machine?)

3. The dataset is ingested (?) by iRODS, which owns a local disk space acting as a buffer and is transported to the datacenter. In iRODS, it is stored at /omeroZone/home/user1/investigation1/study1/dataset0

4. Once upload is complete, the user can trigger the import of its data to OMERO (either through some UI or automatically based on metadata). The same credentials are used to log into OMERO and see the dataset there.

This latter step is done by a script on the omero server machine. A single linux user (omero-server) of this machine performs the import inplace. This linux user logs into OMERO as an admin user and performs the import on behalf of the OMERO user user1. For that, /omeroZone/home/user1/ must be mounted on the omero server and accessible in read only by omero-server

So to summarize:

• All data producing users have both an iRODS and OMERO account, but no linux account on the omero server machine, e.g because they share a common LDAP.
• The omero-server linux account on the omero server machine has also an account on iRODS to use an nfsrods mount as /OMERO (the part you already demonstrated)

I hope it is a bit clearer this way,

Thanks again!

[trel]

Yes, this seems very reasonable. Just have to grant omero-server iRODS user read access to the others users' iRODS collections (which could also be granted by omero-server itself, since it's a rodsadmin).

This should prove a very useful pattern if we can iron out any surprises.

--

One note... if the microscope user has access to the NFSRODS mountpoint... saving files 'in' there is already 'in' iRODS... physically located over in the datacenter. If so, then steps 2 and 3 above are a single step.

[glyg]

Thanks, I'll try to advance a docker-compose version of that.

One note... if the microscope user has access to the NFSRODS mountpoint... saving files 'in' there is already 'in' iRODS... physically located over in the datacenter. If so, then steps 2 and 3 above are a single step.

Yes this would be great. Am I right in assuming it is not too complicated to have a NFS drive mounted on a Windows OS, that would require authentication to mount, so we are in the correct user space from the get go?

Another question: When importing "inplace" in OMERO it is possible to create hard links instead of symbolic ones. I saw issues on nfsrods mentioning hard links, is it possible? I am not sure it is desirable, but adds a level of redundancy as deleting in /DATA would not result in a deletion in /OMERO.

[trel]

The design of NFSRODS is that it is NFSv4.1, so I believe mounting on Windows should work as expected. I'm not sure the extent this has been tested in the wild, as of yet. Please let us know if it is not behaving for some use case(s).

The hard links in iRODS effort has slowed - there were a few edge cases that were not yet resolved in the design... we should probably document those a bit better... https://github.com/irods/irods_rule_engine_plugin_hard_links

I'm sure about the implications of the symlink vs hardlink distinction... in our case, the entire 'inplace' is already a kind of link-only approach... but yes, we should experiment a bit and see what is best.

[glyg]

Hi @trel I hit a problem today: It does not seem possible to create a symbolic link on the mounted NFS drive and as a consequence omero import inplace --transfer=ln_s fails.

To reproduce, I guess the original setup here is sufficient. Bellow, the /OMERO directory on the omero-server machine is the nfsrods mount

docker exec -it -u omero-server omeroserver bash
bash-4.2$
bash-4.2$
bash-4.2$
bash-4.2$ pwd
/opt/setup
bash-4.2$ cd /OMERO/
bash-4.2$ ls
BioFormatsCache  FullText  ManagedRepository  certs
bash-4.2$ touch test
bash-4.2$ ln -s test test2
ln: failed to create symbolic link 'test2': Remote I/O error

Here is the volume definition in docker-compose.yml

  omero_rods:
    name: omero_rods
    driver: local
    driver_opts:
      type: nfs
      o: "addr=$NFSRODS_IPADDRESS,rw,noatime,tcp,timeo=14,nolock,soft,rw,nfsvers=4"
      device: ":/home/omero-server"

It is declared like so:

  omeroserver:
    # This container uses the tag for the latest server release of OMERO 5
    # To upgrade to the next major release, increment the major version number
    image: "openmicroscopy/omero-server:5"
    container_name: omeroserver
    restart: unless-stopped
    env_file:
      - .env
    networks:
      - omero
    ports:
      - "4063:4063"
      - "4064:4064"
    volumes:
      - type: volume
        source: omero_rods
        target: /OMERO
        volume:
          nocopy: true

Reading around, I do not think it is an intrinsic issue with NFS, but am not sure at all.

You can look at the whole project here:

https://gitlab.in2p3.fr/fbi-data/fbi-omero/-/blob/combo

Do you know if this a configuration problem or something else?

Thanks a lot for any hint,

Best

Guillaume

[korydraughn]

NFSRODS does not support symbolic links. Are symlinks a requirement for omero import inplace to work?

[glyg]

Thanks for the rapid answer @korydraughn .

Inplace import can use either symbolic or hard links. The alternative is to perform a copy (with or without subsequent deletion of the original file).

see: https://omero.readthedocs.io/en/stable/sysadmins/in-place-import.html#getting-started

A work around my colleague just mentioned is to write the symlinks target in a 'normal' hard drive, although now the "direct import" route would need to be managed differently. I'm testing this now.

Eventually, it would maybe make sense to write a dedicated importer, class, as I think @joshmoore mentioned, and use:

 $ CLASSPATH=mycode.jar ./importer-cli --transfer=com.example.MyTransfer baz.tiff

[glyg]

Allright, so having /OMERO as a "native" drive on the omero server and linking from an NFSRODS mounted /DATA/ works, the docker-compose has been updated.

Now, in order to get closer to the sketch at the top of this thread, one needs to move 'directly imported' files from the native drive to the NFSRODS drive. I see how to do this with a python script listening to inotify but it would surely be better if iRODS was taking care of that, no?

For now, I am focusing on a multiple users scenario

[glyg]

Hi, Quick update and a question,

Provided the omero-server user has the correct acls, import inplace of any user's data seems to be working fine, though a lot of hardening / tests and so on are missing obviously — see https://gitlab.in2p3.fr/fbi-data/fbi-omero/-/blob/combo/scripts/ldap_irods_sync.py :sparkles:

Now for the question:

When a new user is created in iRODS, the directory tree in /DATA/, i.e. the NFSRODS volume exposing /tempZone/ for the omeroserver container, does not show the new directory.

It appears if I restart the container, but this is not super elegant. I guess mounting / un-mounting the volume in the container would also do the trick, but is there another way to update the NFS mount with the new directory structure in the zone?

Best

Guillaume

[trel]

When a new user is created in iRODS ... the directory tree ... does not show the new directory

I believe if the client side just checks/stats/lists again, it would be there?

@korydraughn this would be client driven, right?

[korydraughn]

Yes. It is client driven. However, I'd expect any system accessing the mount point to cause an update to the tree. Definitely something we need to confirm.

[glyg]

Thanks for the feedback — I only tried ls /DATA/, is there something else I should try? Could this be docker specific? I can open an issue on the nfsrods repo if this is more efficient

[korydraughn]

Try listing the directory of interest directly and seeing if NFSRODS reports it.

I wonder if iRODS is updating the mtime of the parent collection when a collection is created inside of it. If not, then that is likely part of the problem here.

[glyg]

Oh OK I get it, I'll try this tomorrow, I am away from the machine today

[glyg]

Hi, you were right, I can ls the newly created collection directly

[trel]

Most excellent.

[glyg]

Hi @trel, I have some difficulties using a derived version of the docker-compose you provided. I changed the Dockerfile so that the catalog provider is setup with a json configuration file.

Bellow is entrypoint.sh

```sh echo "Running iRODS setup" python3 /var/lib/irods/scripts/setup_irods.py --json_configuration_file /server_config.json echo "Starting server" cd /usr/sbin su irods -c 'bash -c "./irodsServer -u"' ```

Here is server_config.json template (filled through ansible)

```json { "host_system_information": { "service_account_group_name": "irods", "service_account_user_name": "irods" }, "hosts_config": { "host_entries": [], "schema_name": "hosts_config", "schema_version": "v3" }, "host_access_control_config": { "access_entries": [], "schema_name": "host_access_control_config", "schema_version": "v3" }, "server_config": { "advanced_settings": { "default_log_rotation_in_days": 5, "default_number_of_transfer_threads": 4, "default_temporary_password_lifetime_in_seconds": 120, "dns_cache": { "eviction_age_in_seconds": 3600, "shared_memory_size_in_bytes": 5000000 }, "hostname_cache": { "eviction_age_in_seconds": 3600, "shared_memory_size_in_bytes": 2500000 }, "maximum_number_of_concurrent_rule_engine_server_processes": 4, "maximum_size_for_single_buffer_in_megabytes": 32, "maximum_size_of_delay_queue_in_bytes": 0, "maximum_temporary_password_lifetime_in_seconds": 1000, "rule_engine_server_execution_time_in_seconds": 120, "rule_engine_server_sleep_time_in_seconds": 30, "transfer_buffer_size_for_parallel_transfer_in_megabytes": 4, "transfer_chunk_size_for_parallel_transfer_in_megabytes": 40 }, "catalog_provider_hosts": [ "{{ irods_fqdn }}" ], "host_resolution": { "host_entries": [ { "address_type": "local", "addresses": [ "{{ irods_fqdn }}" ] }, { "address_type": "remote", "addresses": [ "{{ nfsrods_ipaddress }}", "omero_irods" ] } ] }, "catalog_service_role": "provider", "client_api_whitelist_policy": "enforce", "default_dir_mode": "0750", "default_file_mode": "0600", "default_hash_scheme": "SHA256", "default_resource_name": "{{ irods_zone_res_name }}", "environment_variables": { "IRODS_DATABASE_USER_PASSWORD_SALT": "{{ irods_db_salt }}" }, "federation": [], "match_hash_policy": "compatible", "negotiation_key": "{{ irods_neg_key }}", "plugin_configuration": { "authentication": {}, "database": { "postgres": { "db_host": "{{ irods_db_host }}", "db_name": "ICAT", "db_odbc_driver": "PostgreSQL Unicode", "db_password": "{{ irods_db_pass }}", "db_port": 5432, "db_username": "irods" } }, "network": {}, "resource": {}, "rule_engines": [ { "instance_name": "irods_rule_engine_plugin-irods_rule_language-instance", "plugin_name": "irods_rule_engine_plugin-irods_rule_language", "plugin_specific_configuration": { "re_data_variable_mapping_set": [ "core" ], "re_function_name_mapping_set": [ "core" ], "re_rulebase_set": [ "core" ], "regexes_for_supported_peps": [ "ac[^ ]*", "msi[^ ]*", "[^ ]*pep_[^ ]*_(pre|post|except|finally)" ] }, "shared_memory_instance": "irods_rule_language_rule_engine" }, { "instance_name": "irods_rule_engine_plugin-cpp_default_policy-instance", "plugin_name": "irods_rule_engine_plugin-cpp_default_policy", "plugin_specific_configuration": {} } ] }, "rule_engine_namespaces": [ "" ], "schema_name": "server_config", "schema_validation_base_uri": "off", "schema_version": "v3", "server_control_plane_encryption_algorithm": "AES-256-CBC", "server_control_plane_encryption_num_hash_rounds": 16, "server_control_plane_key": "{{ irods_cp_key }}", "server_control_plane_port": 1248, "server_control_plane_timeout_milliseconds": 10000, "server_port_range_end": 20199, "server_port_range_start": 20000, "xmsg_port": 1279, "zone_auth_scheme": "native", "zone_key": "{{ irods_zone_key }}", "zone_name": "{{ irods_zone_name }}", "zone_port": 1247, "zone_user": "rods" }, "service_account_environment": { "irods_client_server_negotiation": "request_server_negotiation", "irods_client_server_policy": "CS_NEG_REFUSE", "irods_connection_pool_refresh_time_in_seconds": 300, "irods_cwd": "/{{ irods_zone_name }}/home/rods", "irods_default_hash_scheme": "SHA256", "irods_default_number_of_transfer_threads": 4, "irods_default_resource": "{{ irods_zone_res_name }}", "irods_encryption_algorithm": "AES-256-CBC", "irods_encryption_key_size": 32, "irods_encryption_num_hash_rounds": 16, "irods_encryption_salt_size": 8, "irods_home": "/{{ irods_zone_name }}/home/rods", "irods_match_hash_policy": "compatible", "irods_maximum_size_for_single_buffer_in_megabytes": 32, "irods_port": 1247, "irods_host": "{{ irods_fqdn }}", "irods_server_control_plane_encryption_algorithm": "AES-256-CBC", "irods_server_control_plane_encryption_num_hash_rounds": 16, "irods_server_control_plane_key": "{{ irods_cp_key }}", "irods_server_control_plane_port": 1248, "irods_transfer_buffer_size_for_parallel_transfer_in_megabytes": 4, "irods_user_name": "rods", "irods_zone_name": "{{ irods_zone_name }}", "schema_name": "service_account_environment", "schema_version": "v3" }, "admin_password": "{{ irods_admin_pass }}", "default_resource_directory": "{{ irods_zone_path }}", "default_resource_name": "{{ irods_zone_res_name }}" } ``` `irods_fqdn` is the docker container hostname (it is correctly resolved), `nfsrods_ipaddress` is the IP of the host, for remote access (I have not tried that yet). `irods_admin_pass` is set to `"rods"` to avoid complicating thing

For completeness docker-compose.yml

```yml version: "3.3" networks: omero: volumes: backups: name: backups driver_opts: type: none device: /home/debian/backups o: bind database: name: database driver_opts: type: none device: /home/debian/database o: bind services: irods-catalog: image: "postgres:12" container_name: irods-catalog hostname: irods-catalog # restart: unless-stopped env_file: - .env networks: - omero volumes: - database:/var/lib/postgresql/data irods-catalog-provider: container_name: irods-catalog-provider hostname: irods-catalog-provider # restart: unless-stopped build: context: "./irods_catalog_provider" ports: - 1247:1247 networks: - omero depends_on: - irods-catalog irods-client-nfsrods: image: irods/nfsrods:2.1.0 # restart: unless-stopped container_name: irods-client-nfsrods hostname: irods-client-nfsrods ports: - "2049:2049" networks: - omero volumes: - "./irods_client_nfsrods/nfsrods_config:/nfsrods_config:ro" - "./irods_client_nfsrods/passwd:/etc/passwd:ro" depends_on: - irods-catalog-provider ```

Here is the error I get from up.sh:

./up.sh
irods-catalog uses an image, skipping
irods-client-nfsrods uses an image, skipping
Building irods-catalog-provider
[+] Building 0.8s (14/14) FINISHED
 => [internal] load build definition from Dockerfile                                                              0.0s
 => => transferring dockerfile: 1.47kB                                                                            0.0s
 => [internal] load .dockerignore                                                                                 0.0s
 => => transferring context: 2B                                                                                   0.0s
 => [internal] load metadata for docker.io/library/ubuntu:20.04                                                   0.7s
 => [1/9] FROM docker.io/library/ubuntu:20.04@sha256:db8bf6f4fb351aa7a26e27ba2686cf35a6a409f65603e59d4c203e58387  0.0s
 => [internal] load build context                                                                                 0.0s
 => => transferring context: 164B                                                                                 0.0s
 => CACHED [2/9] RUN apt-get update &&     apt-get install -y         apt-transport-https         gnupg           0.0s
 => CACHED [3/9] RUN wget -qO - https://packages.irods.org/irods-signing-key.asc | apt-key add - &&     echo "de  0.0s
 => CACHED [4/9] RUN apt-get update &&     apt-get install -y         libcurl4-gnutls-dev         python3         0.0s
 => CACHED [5/9] RUN apt-get update &&     apt-get install -y         irods-database-plugin-postgres=4.3.0-1~foc  0.0s
 => CACHED [6/9] COPY irods_environment.json /root/.irods/irods_environment.json                                  0.0s
 => CACHED [7/9] COPY irods_environment_server.json /var/lib/irods/.irods/irods_environment.json                  0.0s
 => CACHED [8/9] COPY server_config.json /server_config.json                                                      0.0s
 => CACHED [9/9] COPY entrypoint.sh /                                                                             0.0s
 => exporting to image                                                                                            0.0s
 => => exporting layers                                                                                           0.0s
 => => writing image sha256:4867fd861c5f6da99c77672c4771a0241cae5bc24bdc9eeaa141f8e1395960d5                      0.0s
 => => naming to docker.io/library/compose_irods-catalog-provider                                                 0.0s
irods-catalog is up-to-date
Starting irods-catalog-provider ... done
Error occurred while authenticating user [rods] [CAT_INVALID_AUTHENTICATION: failed to perform request

] [ec=-826000] failed with error -826000 CAT_INVALID_AUTHENTICATION
irods-catalog is up-to-date
irods-catalog-provider is up-to-date
Starting irods-client-nfsrods ... done

The irods server is up:

irods@irods-catalog-provider:~$ ./irodsctl -v --test --stdout status
Calling status on IrodsController
irodsServer :
  Process 84
  Process 85
  Process 1690

There are some CLOSE_WAIT processes and the server fails to gracefully shutdown.

root@irods-catalog-provider:/# ss -t
State           Recv-Q      Send-Q           Local Address:Port              Peer Address:Port            Process
ESTAB           0           0                   172.26.0.3:38084               172.26.0.2:postgresql
ESTAB           0           0                   172.26.0.3:38074               172.26.0.2:postgresql
CLOSE-WAIT      1           0                   172.26.0.3:1247                172.26.0.3:49030
ESTAB           0           0                   172.26.0.3:49016               172.26.0.3:1247
ESTAB           0           0                   172.26.0.3:1247                172.26.0.3:49016
CLOSE-WAIT      1           0                   172.26.0.3:1247                172.26.0.3:47680

Any hint on how to debug this greatly appreciated.