OpenFreeEnergy / alchemiscale

a high-throughput alchemical free energy execution system for use with HPC, cloud, bare metal, and Folding@Home
http://alchemiscale.org
MIT License
23 stars 8 forks source link

Make `max_time` in `SynchronousComputeService.start` check occur in separate thread #124

Open dotsdl opened 1 year ago

dotsdl commented 1 year ago

Currently, max_time is checked alongside max_tasks at the end of each task execution. For long-running tasks (e.g. hours), this could make max_time a bit useless. Instead, we'd like for max_time to be checked periodically via a separate thread in the same way we are doing heartbeats.

dotsdl commented 1 year ago

Actually, it's very useful to have two separate max times that can be set, with different meanings:

These can of course be used together, with max_time_soft used to set the time beyond which no new tasks should be taken on (perhaps because the service is running in an HPC job, and there wouldn't be enough time to finish anything), and max_time_hard could be set to just before the job absolutely must finish, ensuring clean shutdown even if a task isn't completed.