Open dotsdl opened 1 year ago
Actually, it's very useful to have two separate max times that can be set, with different meanings:
max_time_soft
: amount of time to run, beyond which the service will shut off after it has finished a cyclemax_time_hard
: amount of time to run, beyond which the service will shut off no matter what it is doingThese can of course be used together, with max_time_soft
used to set the time beyond which no new tasks should be taken on (perhaps because the service is running in an HPC job, and there wouldn't be enough time to finish anything), and max_time_hard
could be set to just before the job absolutely must finish, ensuring clean shutdown even if a task isn't completed.
Currently,
max_time
is checked alongsidemax_tasks
at the end of each task execution. For long-running tasks (e.g. hours), this could makemax_time
a bit useless. Instead, we'd like formax_time
to be checked periodically via a separate thread in the same way we are doing heartbeats.