Avoid shutdown jobs

[wlandau]

Prework

• [x] Read and agree to the Contributor Code of Conduct and contributing guidelines.
• [x] If there is already a relevant issue, whether open or closed, comment on the existing thread instead of posting a new issue.
• [n/a] For any problems you identify, post a minimal reproducible example like this one so the maintainer can troubleshoot. A reproducible example is:
  • [n/a] Runnable: post enough R code and data so any onlooker can create the error on their own computer.
  • [n/a] Minimal: reduce runtime wherever possible and remove complicated details that are irrelevant to the issue at hand.
  • [n/a] Readable: format your code according to the tidyverse style guide.

Description

My early attempts at task queues explored the idea of using a special job to shut down a worker. This shutdown method is proving to be unreliable, and it is too important to fail. The technique for shutting down workers will need to be backend-specific, and this is not always available. For batchtools futures, in the absence of https://github.com/HenrikBengtsson/future/issues/93, those workers will need to be transient. But I still think batchtools futures will make a nice task queue because async can speed up the startup/polling/cleanup for large collections of futures (i.e. to work around https://github.com/HenrikBengtsson/future/discussions/602).

[wlandau]

Got it figured out in branch 11.