nested crews

[wlandau]

Solution to #3. Let's create outer callr workers that run inner crews of cloud workers. We can accomplish this with a special subclass of class_worker_callr() where launch() has a new event loop that creates an inner crew and forwards outer jobs to it. #2 is no longer needed.

[wlandau]

Supplying an inner crew spec could get complicated. Probably best to have the user supply an expression that creates one, including defs of the workers and store..

[wlandau]

Implemented classed messages. Maybe the inner loop can handle this stuff? Next steps:

• Create the crew at the beginning of the inner loop.
• Various methods to do stuff with inner crews:
  • recruit workers
  • sendable
  • forward jobs
  • receivable
  • receive jobs
  • shut down
  • stuck
  • restart

[wlandau]

Probably should start the Amazon AWS Batch backend before going further to implement nested crews. Nested crews have complicated enough business logic to be error prone, and methods like sendable() and receivable() would themselves need to be (outer) jobs. A specialized AWS Batch worker could have specialized methods that work asynchronously, with some caching to manage outputs that get returned at odd times. Let's see how far that goes first.

[wlandau]

Or maybe nested crews need a queue so sendable() and receivable() don't matter anymore.

[wlandau]

Closing. I no longer think nested crews will be a successful approach.