Commit b1560fb introduced retries in the rail-slurm script when rail-estimate returned with an error status, with the goal of recovering after temporary failures. A call to sleep(), to sleep for 1 second when the failure happened allowed the rest of the system some time to recover. Sleeping for 1 second in case of permanent failures is too costly, though. For example, with 5000 tasks and 2 retries before detecting a permanent failure, this results in waiting for almost 3 hours until the permanent failure is detected. This patch reduces the sleep time to 200 ms, which causes a permanent failure to be detected in most of the cases in less than an hour.
Commit b1560fb introduced retries in the rail-slurm script when rail-estimate returned with an error status, with the goal of recovering after temporary failures. A call to sleep(), to sleep for 1 second when the failure happened allowed the rest of the system some time to recover. Sleeping for 1 second in case of permanent failures is too costly, though. For example, with 5000 tasks and 2 retries before detecting a permanent failure, this results in waiting for almost 3 hours until the permanent failure is detected. This patch reduces the sleep time to 200 ms, which causes a permanent failure to be detected in most of the cases in less than an hour.