dramatiq_sqs

A Dramatiq broker that can be used with Amazon SQS.

This backend has a number of limitations compared to the built-in Redis and RMQ backends:

• the max amount of time messages can be delayed by is 15 minutes,
• messages can be at most 256KiB large and
• messages must be processed within 2 hours of being pulled, otherwise they will be redelivered.

The backend uses boto3 under the hood. For details on how authorization works, check out its docs.

Installation

pip install dramatiq_sqs

Usage

import dramatiq

from dramatiq.middleware import AgeLimit, TimeLimit, Callbacks, Pipelines, Prometheus, Retries
from dramatiq_sqs import SQSBroker

broker = SQSBroker(
    namespace="dramatiq_sqs_tests",
    middleware=[
        Prometheus(),
        AgeLimit(),
        TimeLimit(),
        Callbacks(),
        Pipelines(),
        Retries(min_backoff=1000, max_backoff=900000, max_retries=96),
    ],
)
dramatiq.set_broker(broker)

Usage with ElasticMQ

broker = SQSBroker(
    # ...
    endpoint_url="http://127.0.0.1:9324",
)

Example IAM Policy

Here are the IAM permissions needed by Dramatiq:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "sqs:CreateQueue",
                "sqs:ReceiveMessage",
                "sqs:DeleteMessage",
                "sqs:DeleteMessageBatch",
                "sqs:SendMessage",
                "sqs:SendMessageBatch"
            ],
            "Resource": ["*"]
        }
    ]
}

License

dramatiq_sqs is licensed under Apache 2.0. Please see LICENSE for licensing details.