Executor Queue MVP

[stjohnjohnson]

As a user, I want my build to enqueue for later execution if all the execution engines are offline or full, so that I do not have to worry about forgotten builds

Right now, Screwdriver directly calls the executor when a build is to be started. If the executor target is offline or full, the build is generally dropped with an error message. This isn't a good user experience, especially if the error is hidden due to the build being started from a PR or commit.

Instead, Screwdriver should be pushing planned builds into a queue. And all available executors should be listening on that queue and taking items off when they have capacity. This will allow us to distribute between N number of Docker Swarm clusters for example.

Rules:

• Executors can determine if they are capable of running a new build.
• Multiple instances of the same executor can connect to the queue.
• Multiple executor types can connect to the same queue.
• JWT is not created until build is sent to the executor.
• Users know how far in the queue a build is.

---

Notes:

• We should investigate what other technologies like Travis or Wercker are using for this internally.

---

Edit (petey): Removed work items - See list below

[d2lam]

@tk3fftk @catto It would be great if you guys could update the issue with your findings, decisions, progress, etc. Also feel free to let us know if you have any questions/concerns! :)

[catto]

@d2lam 🙆 We have started designing an architecture of the executor queue and are going to share it with you when completed.

[d2lam]

@catto awesome! Thanks for the update!

[catto]

I talked to @d2lam today, and we have decided that we are going to work on #300 instead of this because we don't have enough time to do.

[jerryzhang222]

Travis queuing

From what I've seen, Travis uses Sidekiq to queue up builds and perform tasks. They have some code that uses RabbitMQ (AMQP), but I don't see this code used anywhere and it appears that they are trying to phase RabbitMQ out.

[petey]

We're currently settled on redis/node-resque as the queue solution for this feature.

• [x] Publish doc on decisions + UML

MVP: The executor queue MVP should perform exactly the same tasks as the current execution mechanism, but have a queue in between the api and the executor

• [x] Set up redis/resque service (#661)
• [x] Spike: API wrapper for redis/resque service needed?
• [x] Executor interface to queue jobs
  • [x] Use annotations as a feature flag to use queue
• [x] Queue workers that start an executor

[Edit] Removed stretch goal.