search

HenrikBengtsson / future.batchtools

:rocket: R package future.batchtools: A Future API for Parallel and Distributed Processing using batchtools
https://future.batchtools.futureverse.org
84 stars 9 forks source link

WISH: Polling frequency #7

Closed HenrikBengtsson closed 7 years ago

HenrikBengtsson commented 7 years ago

Background

Currently, the package uses batchtools::waitForJobs() every second (getOption("future.wait.delay", 1.0)) to poll the scheduler to see when jobs are finished. Polling the scheduler to check when jobs are ready can but a heavy burden on the scheduler if there are lots of simultaneous requests.

Suggestion

The future.BatchJobs package had a similar approach, but it slowly increased the waiting time between each poll such that for very long running jobs we didn't keep checking every minute. This should be implemented in future.batchjobs too. This requires to implement our own batchtools::waitForJobs() version (or fix it upstream).

mllg commented 7 years ago

Note: There already is a slowly increasing sleep time (up to 2 mins) between requests:

If you do not like the defaults I picked in default.sleep you can easily provide your own function.

HenrikBengtsson commented 7 years ago

This is awesome; I missed that feature - it looks like exactly what I need.