race condition while copying is happenning.

MetPX / sarracenia

https://MetPX.github.io/sarracenia

GNU General Public License v2.0

44 stars 22 forks source link

race condition while copying is happenning. #349

Open petersilva opened 4 years ago

petersilva commented 4 years ago

when there is a race-condition and two processes are downloading... mtime will of the file will be current, so "later" file will be rejected as too old, even though the older mtime will be restored, once the file is completely written.

petersilva commented 4 years ago

This is a case that has occurred in the HPC mirroring use case. Clarification:

receive a message with file mtime = x
start downloading.
receive a message with file mtime = x + 1
compare message mtime (x+1) with current mtime of file being written (x+2) conclusion, new file is only at x+1 which is older than x+2, so the newer file version is rejected.
it is only after the copy is complete that the correct mtime (x) will be restored.

So how to have the file in progress marked to indicate the correct mtime when it will be complete. Options:

create an extended attribute that stores the mtime... have the downloaders compare against that.
create a lock file that stores the mtime...

complications:

use of accellerators (third party C programs will not respect either protocol above.)

petersilva commented 4 years ago

perhaps write a plugin that just consolidates multiple updates into a single one by delaying for only a few seconds. the plugin solution would be a lot easier to do with version 3 plugins, which deal with groups of messages, rather than individual ones.

petersilva commented 4 years ago

discusse the delay/consolidation plugin with client. they seem ok with it in principle. the plugin needs to be applied to the winnow components.

petersilva commented 3 years ago

Idea is a mini-cache ... lasting oh... 30 seconds or so.

for each file received, store in a python dictionary?
- index is file name (path)
- value is ... I guess the whole message?
if you receive an updated version of the same file, replace the cache entry with the new one.
once the cache entry is > 30 seconds old, publish it.

make 30 seconds a programmable value.

For the HPC user case, this has to happen in the shovels, because there is no way to ensure a single subscriber will get all versions of the file. This may also solve the problem with the citypages. suppressing a lot of noise.

msg_suppress_excessive_writes

msg_suppress_excessive_write_timeout 30