This adds an AWS backend and is the first step towards a better abstraction that I am hoping to work on. The AWS Connection makes it possible to use SNS or SQS to enqueue jobs and SQS to dequeue. All state of known queues and workers is stored in dynamodb.
The abstraction I want to change is from "backend" to "state" and "queue". Right now, the requirements of a "backend" require storing arbitrary things (workers, queues) that all of the real and great queues don't support. Also, most of this arbitrary information is used only for UI purposes (of which there is not one currently).
Separating backend into state and queue means that all queues can work with qu and one can store the UI state where it makes more sense (mysql, redis, mongo, dynamo, simpledb, whatever) or not at all. Hoping to work on that next, but need to power through the AWS stuff in order to see what was needed.
Also, this adds the idea of required services for a backend spec. It means if you provide services, we'll make sure they are up when the specs run and if they aren't we'll skip those specs so you dont have to have every queue and database in the world running with exactly the right options all the time to work on qu.
See what I did there. Yeaaaaaahhhhh.
This adds an AWS backend and is the first step towards a better abstraction that I am hoping to work on. The AWS Connection makes it possible to use SNS or SQS to enqueue jobs and SQS to dequeue. All state of known queues and workers is stored in dynamodb.
The abstraction I want to change is from "backend" to "state" and "queue". Right now, the requirements of a "backend" require storing arbitrary things (workers, queues) that all of the real and great queues don't support. Also, most of this arbitrary information is used only for UI purposes (of which there is not one currently).
Separating backend into state and queue means that all queues can work with qu and one can store the UI state where it makes more sense (mysql, redis, mongo, dynamo, simpledb, whatever) or not at all. Hoping to work on that next, but need to power through the AWS stuff in order to see what was needed.
Also, this adds the idea of required services for a backend spec. It means if you provide services, we'll make sure they are up when the specs run and if they aren't we'll skip those specs so you dont have to have every queue and database in the world running with exactly the right options all the time to work on qu.