HPC mirroring: remember moved directories when retrying failed transfers.

[petersilva]

when there are many copies in flight, and the order of operations is unclear, there will be times when the following sequence of events occurs at source:

echo hoho_content >dir1/hoho
mv dir1 therealdir

the first line will result in a notification message for the dir1/hoho file... the second line will result in a directory rename event.

If the consumer receives the notification after the second line has occurred, it will try to retrieve the file dir1/hoho from the source. The source file will no longer be there, because the directory has been renamed.

It would be good if the retry logic had a record of recent directory renames around, and could identify whether one of the directory renames was relevent, and so adjust the fetch url to reflect the new source.

Existing Test Case

We have an easy canned way to reproduce this issue. In the sarrac ( https://github.com/MetPX/sarrac ) repository, there is a shim_copy_post.sh script, which has a sleep before a directory is renamed to prevent this problem. Commenting out the sleep provokes the problem. so people can try things out easily. then do make test_shim_copy, and the test will fail with some helpful error messages.

[petersilva]

complexities:

• with winnowing, the process that receives the directory rename may not be the same one that received the file message. If the cache is per instance, no guarantee the same instance will see the rename message AND the corresponding download.

• renames are done locally (on the download side.) but need to understand how to modify the upstream url to it's pre-rename value, which is never used in the normal renames... need to figure out the retrieval url that would correspond to undoing a rename.

[petersilva]

• how to account for retPath vs. relPath in directory renaming? file renmaes are done based on where it resolves to on download.

[petersilva]

option 1: subscribers "undo" directory rename on download/retry

implementation is all at the downstream side... (subscribers) with no change to posting side...

• make a mvdir cache listing all directories that have been renamed. (after a rename succeeds store the corresponding message in the cache.)
• when a download fails, have a look in the mvdir cache... see if some the download url contains a directory that has been renamed.
• construct an updated download Url, and try that.

concerns/challenges:

• life cycle of mvdir cache... aging entries out, etc...
• persistence of mvdir cache. shutdown/restart etc...
• looks remarkably like the retry_cache (DiskQueue) ...
• different instances can get file downloads vs. directory renames. This particular problem requires global state, where all downloading instances are aware of all directory renames. Might want to modify global algorithm to post directory renames so that they fanout to all exchanges in a post_exchangeSplit case... making certain messages global... complicated...

Option 2: posters add a trailing symbolic link at the source

This is is entirely on the poster side. The poster just creates a link so that the original download path still works for a while a rename.

• When mv is called on a directory, The poster does the mv, but also does an ln -s of the old value to the new.
• store a note about "temporary links" created...
• after some amount of time, remove the temporary links.
• since the shim library only lasts as long as a process, it cannot do the removals. so generate an additional post for special handling... something like a symlink message with a custom field "mvDirBuffer":"" in the fileOp field?
• need a source-side subscriber to process those links, and remove them later... message q takes care of persistence.

concerns/challenges:

• users will notice and perhaps be annoyed by these temporary links showing up.
• Added complexity of another shovel on the "source" side. Possible to include it at the winnow level to avoid another configuration.
• the source side processing could be done with a plugin.
• destination needs to ignore these messages: another plugin... or if done in the winnow, remove the messages from the stream.
• need a tunable lifetime for these messages... what is it? (something auto ajusting based on lag metrics?)

Existing Test Case

In the C package, there is a shim_copy_post.sh script, which has a sleep before a directory is renamed to prevent this problem. Commenting out the sleep provokes the problem. so people can try things out easily.