Since ProcessQueue spawns a bunch of processes on the local machine, it should be possible to set it up to dispatch the command lines to an AWS batch queue.
Proposed enhancement
Spawn the actual processing codes via a batch queue instead of directly on the machine running ProcessQueue.
IPC required:
Filesystem needs to be available on the spawning instance and the batch-executing instance
Full command line goes into the batch queue
stdout/stderr is ordinarily captured by the spawning ProcessQueue and so this needs to be captured via some other means
Final created file is normally stored in a temp directory, either ProcessQueue can handle putting this on a shared filesystem or there needs to be some way of moving it across
Alternatives
Could go for fuller clustering support, but this seems like the smallest change that supports good horizontal scaling.
Closure condition
Implemented and tested. Unit testing might be "interesting", either "mock" AWS or pull out the actual batch injection as a very thin shim that is not tested in unit tests (only functional testing).
Since ProcessQueue spawns a bunch of processes on the local machine, it should be possible to set it up to dispatch the command lines to an AWS batch queue.
Proposed enhancement
Spawn the actual processing codes via a batch queue instead of directly on the machine running ProcessQueue.
IPC required:
Alternatives
Could go for fuller clustering support, but this seems like the smallest change that supports good horizontal scaling.
Closure condition
Implemented and tested. Unit testing might be "interesting", either "mock" AWS or pull out the actual batch injection as a very thin shim that is not tested in unit tests (only functional testing).