Using multiple queues on a single connection: is this possible?

[arondeparon]

What is the current issue?

With the setup below, my jobs seem to be dispatched to the correct SQS queue, but after this it seems like deleteMessage is called (which I don't think should happen if I dispatch a job that does not fail?). Further more, this call to deleteMessage seems cause another error because it is trying to delete the message on the default (SQS_QUEUE) queue instead of the queue that the message has been pushed to.

In short: am I doing this wrong? What is the best way to configure multiple queues with Bref and Laravel? Ideally, I would not have to configure separate connections, because with that, I would have specify these connections in my application code (FoobarJob::dispatch()->onConnection('...')) which I feel is a bit awkward, since technically, SQS is the connection and we should just use the native onQueue method.

What I want to achieve

• SQS_QUEUE should be my "default" queue
• SQS_QUEUE_HIGH should be my high priority queue for high priority jobs
• SQS_QUEUE_LOW for jobs that can take a bit longer (background jobs)

I then map these queue urls to config variables in Laravel.

From my code, I then reference this queues as follows: FoobarJob::dispatch()->onQueue(config('queue.queues.high'))

My configuration

serverless.yml:

provider:
.........................
    environment:
        QUEUE_CONNECTION: sqs
        SQS_QUEUE: ${construct:jobs.queueUrl}
        SQS_QUEUE_HIGH: ${construct:jobs-high.queueUrl}
        SQS_QUEUE_LOW: ${construct:jobs-low.queueUrl}
.....................
    jobs:
        type: queue
        maxConcurrency: 25
        worker:
            handler: Bref\LaravelBridge\Queue\QueueHandler
            runtime: php-82
            timeout: 300

    jobs-high:
        type: queue
        maxConcurrency: 50
        worker:
            handler: Bref\LaravelBridge\Queue\QueueHandler
            runtime: php-82
            timeout: 300

    jobs-low:
        type: queue
        maxConcurrency: 25
        worker:
            handler: Bref\LaravelBridge\Queue\QueueHandler
            runtime: php-82
            timeout: 300

config/queue.php:

    'queues' => [
        'default' => env('SQS_QUEUE', 'default'),
        'high' => env('SQS_QUEUE_HIGH', 'high'),
        'low' => env('SQS_QUEUE_LOW', 'low'),
    ],

Test file:

            dispatch(function () {
                return true;
            })->onQueue(config('queue.queues.high'));

[mnapoli]

Hey Aaron,

In your example, all 3 workers look identical:

    jobs:
        type: queue
        maxConcurrency: 25
        worker:
            handler: Bref\LaravelBridge\Queue\QueueHandler
            runtime: php-82
            timeout: 300

    jobs-high:
        type: queue
        maxConcurrency: 50
        worker:
            handler: Bref\LaravelBridge\Queue\QueueHandler
            runtime: php-82
            timeout: 300

    jobs-low:
        type: queue
        maxConcurrency: 25
        worker:
            handler: Bref\LaravelBridge\Queue\QueueHandler
            runtime: php-82
            timeout: 300

I would expect all 3 of them to process the max queue.

Here is what I would try instead:

provider:
    ...
    environment:
        # default queue
        QUEUE_CONNECTION: sqs
        SQS_QUEUE: ${construct:jobs.queueUrl}

constructs:

  jobs:
    type: queue
    worker:
      handler: Bref\LaravelBridge\Queue\QueueHandler
      runtime: php-81

  emails:
    type: queue
    worker:
      handler: Bref\LaravelBridge\Queue\QueueHandler
      runtime: php-81
      environment:
        # Override the queue to process
        QUEUE_CONNECTION: emails

does that make sense?

[arondeparon]

Thanks Mathieu, this never registered in my head.

Will implement this somewhere next week, but this should definitely do the trick. Would perhaps make a nice addition to the documentation? It's definitely not uncommon to use multiple queues.

[mnapoli]

💯 would happily merge a pull request, I'm trying to keep up with everything lately so I won't have time to jump on it myself.