When rsync upload begins, send a message to Dataverse.

[pdurbin]

Last week when discussing https://trello.com/c/Nbte37k1/9-rsync-file-upload-%26-download-(4.8) with @mheppler @TaniaSchlatter @dlmurphy we agreed that we'd all like to see the Data Capture Module (DCM) POST some JSON to Dataverse when upload has begun. That is to say, the DCM will recognize that the user has started executing the rsync script and inform Dataverse of this fact. When Dataverse receives the "upload has begun" message (or "uploadHasBegun"?), Dataverse will take some actions, possibly sending a notification to the user, preventing the dataset from being deleted, etc. It would be awfully nice if the DCM could send the number of bytes Dataverse should expect, but this is not a hard requirement.

I believe the issue on the Dataverse side is https://github.com/IQSS/dataverse/issues/3348

[pameyer]

Non-trivial change; two possible initial approaches:

• DCM recognizes transfer initialization and informs Dataverse
• script uses API token on user's behalf to make call to Dataverse

First approach has intersections w\ security model (transfer user shell not a login shell; but "transfer initialization" is equivalent to receiving successful login for transfer). Second approach might require more infrastructure, but probably won't have these impacts.

[pdurbin]

Bummer that it's a non-trivial change. I'm not particularly interested in the rsync script talking to Dataverse. It's the rsync script's job to talk to the DCM and I think we should keep it that way. No rush on any of this. It would have been a nice to have for #3942 but we can live without it, I'd say.

[pameyer]

A few notes on initial investigations:

• DCM recognizing transfer initialization. Can't use login triggers, since a non-login shell is used for transfer accounts (intentionally, for security).
• No triggers would mean either wrapping rsync with a script containing a trigger (which is unlikely to work in the context of the non-login shell for the transfer user), or a periodic process checking for transfers that have been started but not yet completed.
• Periodic process would probably run into the impedance mismatch problems previously encountered w\ cron and DV; and potential timing issues (small datasets may miss be fully processed before the cron job notices and notifies DV).

[pameyer]

Obsolete from discussions of https://github.com/IQSS/dataverse/issues/3348 today.

[pdurbin]

I still think the cron job would be better than nuthin'. 😄