When running an ED2 model, start_model_runs() sometimes fails to copy some ensembles in the run directory to a remote host (HPC). One possible reason for this this is that rsync is currently being run inside of a for-loop and maybe there are some limits to how many connections to the server are open or how often connections can be made. It'll be more efficient to just rsync all the ensemble files over at once outside of a for-loop anyways, even if it doesn't fix this bug.
Bug Description
When running an ED2 model,
start_model_runs()sometimes fails to copy some ensembles in the run directory to a remote host (HPC). One possible reason for this this is thatrsyncis currently being run inside of a for-loop and maybe there are some limits to how many connections to the server are open or how often connections can be made. It'll be more efficient to justrsyncall the ensemble files over at once outside of a for-loop anyways, even if it doesn't fix this bug.It's either happening here: https://github.com/PecanProject/pecan/blob/f5194f84b26a85bbfcb279cc4665d7560205efc7/base/workflow/R/start_model_runs.R#L97-L102
Or maybe here (can't remember) https://github.com/PecanProject/pecan/blob/f5194f84b26a85bbfcb279cc4665d7560205efc7/base/workflow/R/start_model_runs.R#L132-L141
To Reproduce
difficult to reproduce, sorry.
Expected behavior
All files for ensemble runs should be copied over and if they can't be, there should be an informative warning or error.
Additional context
Add any other context about the problem here.