Task queue workers as a service

[andywer]

Upload a package.json and some JS code to the cloud to spawn auto-scaling workers for a task queue. Somewhat like now by zeit.co, but optimized for the auto-scaling task queue use case.

Would be helpful to have it for running shutter.sh and bundle.sh.

Nice to have features

• Prioritization / rate limiting based on custom conditions / custom logic
• Poll streams - Poll for input data instead of pushing it actively
• Cron jobs
• Provide data storage

Competitors

• iron.io

Potential first step

• simple service for scheduling docker tasks & storing logs
• simple service for uploading and managing tasks
• make dockerized job run on swarm stack

[andywer]

Use case: Might be fun to run ML software in the workers: https://www.npmjs.com/package/opencv4nodejs

[andywer]

Use case

For dev purposes users should definitely be able to run it very conveniently from their CLI. But it actually becomes useful if not triggered manually, but by some service.

Additionally I want to be able to manage my task logic from the CLI:

• Upload a new task runner
  • Name
  • Version
  • Docker image or just tarball of npm package
• List task runners and versions
• Tag task runner versions
• Delete task runners / versions