Closed jeromelecoq closed 1 day ago
This is a good idea. We will prioritize to get this in
@jeromelecoq
I don't think Robocopy / subprocessing here is such a huge concern.
For context, when lims scheduler implemented this, it was because shutil used to be incredibly slow. While shutil is not incredibly slow anymore, robocopy is still providing certain features that should not be overlooked.
Robocopy does have certain failing though.
Fixed but not deployed.
If file is not found, current code will not throw a useful error.
Is your feature request related to a problem? Please describe. Currently we have silent failures associated with watchdog. This is because the slow and key part of the process is handled by a subprocess which essentially isolate the main program from the potentially failing subprocess. This is visible here : https://github.com/AllenNeuralDynamics/aind-watchdog-service/blob/4020d4406149e7a06cd77ef4cf04f364f341edce/src/aind_watchdog_service/run_job.py#L117
Describe the solution you'd like I propose to forward up the errors so that we know what is going on.
Describe alternatives you've considered Removing the subprocess entirely is an alternative solution. Do we really need it? It seems like shutil.copytree is actually faster so far in test implemented by Marton Rosza.