Is timestamp preservation possible for transfers to iRODS?

[jefscheepers]

A user has noticed that timestamp preservation doesn't work on transfers to our iRODS servers. This is the error in question:

Error (transfer) Endpoint: Server: 134.58.8.4:443 File: Message: This server version does not support timestamp preservation

Details: Sanity check did not pass - MFMT failed

My question is: is timestamp preservation something that the iRODS-Globus-connector takes into account? Or does this feature not exist? I'm curious because unlike other filesystems, iRODS stores the timestamps in the iCAT database.

If this functionality exists, the reason may very well lay outside of the iRODS-Globus connector. But we want to avoid digging into this if we aren't sure it actually exists. In the past, we also had this issue with one of our non-iRODS endpoints, but since that server is gone, we cannot reproduce anymore.

Best regards,

Jef

[trel]

some context for posterity...

https://docs.globus.org/faq/globus-connect-endpoints/#what_can_i_do_if_my_endpoint_does_not_support_timestamp_preservation

and for transfers normally...

https://docs.globus.org/cli/reference/transfer/

--preserve-timestamp, --preserve-mtime
   Preserve file and directory modification times.

[JustinKyleJames]

I've looked through the developer guide and do not see examples on how to determine that this flag is requested and how to get the mtime when transferring files. I have asked my Globus contacts about this and will report back what they say.

I do not believe this feature has currently been implemented in the iRODS DSI.

[JustinKyleJames]

I talked to Globus and they provided me with examples on how to implement this functionality. We will have to makes changes and test it out.

[jefscheepers]

That's great news! Thanks for looking into this.

[JustinKyleJames]

Here is the way to detect a request to preserve the timestamp.

An attribute that you check on a recv() (globus_l_gfs_iRODS_recv()),
which gives you the same unix timestamp.   You can apply it to the
new/replaced file however it makes sense.  You might need this if the
storage has to know the mtime at file creation like some of the cloud
connectors, but it also saves a call on the control channel, so it's
useful even if the storage is compatible with 1).

A timestamp of -1 means the client didn't include an mtime with the
request so it should be ignored.

See also:

https://github.com/globus/globus-toolkit/blob/globus_6_branch/gridftp/server/src/modules/file/globus_gridftp_server_file.c#L2984

https://github.com/globus/globus-toolkit/blob/globus_6_branch/gridftp/server/src/modules/file/globus_gridftp_server_file.c#L508

[JustinKyleJames]

I have implemented this change. However, I am not sure how to test it with the tools I have available so I have a question out for Globus on that.

@jefscheepers Would you be willing to run a test using modify time preservation?

They also stated that this functionality needs to be enabled on their side as well. I'm not sure exactly what that means so I have that question outstanding as well.

[jefscheepers]

Dear @JustinKyleJames,

Apologies for the late response. I'm willing to test, but unfortunately, I'm not able to set up a test environment for this.

On 23/5/2024, our admins have built the latest version of the iRODS-globus-connector (I assume without cherrypicking, should check this). They did this because we needed some of the latest bugfixes. This version is now being used for our production environment.

Currently, when I do a transfer with the option 'preserve source file modification times', I get the following error in the interface:

5/29/2024, 12:11 PM Uncategorized Error

Error (transfer) Endpoint: VSC iRODS set.irods.icts.kuleuven.be (56dd39e2-ffae-4075-8354-ba8ed078dd59) Server: 134.58.8.37:443 File: /set/home/datateam_set/prc_scripts.tar Command: MFMT 20230830093833 /set/home/datateam_set/prc_scripts.tar Message: Fatal FTP response

Details: 500 iRODS: Command (GLOBUS_GFS_CMD_SITE_UTIME) is not implemented.\r\n

[alanking]

They also stated that this functionality needs to be enabled on their side as well. I'm not sure exactly what that means so I have that question outstanding as well.

@JustinKyleJames - Did Globus ever get back to you on this issue?