Celery workers do not consume tasks with HTTPS on MacOS

When running SmoothMQ with HTTPS, we can add tasks with Celery but workers do not pick them up. This is only on MacOS - everything works as expected on Linux.

Here's a code example:

The Problem

from celery import Celery

app = Celery(
    "tasks",
    broker_url='sqs://7399377469860198052:27018ec9cb39d6d4d09b0f0378d3991c@7399377469860198052.demo.smoothmq.com:3001',
    broker_transport_options={'is_secure': True}
)

@app.task
def hello():
    return 'hello world'

if __name__ == '__main__':
        hello.delay()

This will successfully place the task on a queue, which you can see on this dashbaord. However, the worker will not pick up tasks:

celery -A tasks worker -l DEBUG --concurrency 1

It will successfully create queues, list queues, and prepare to call ReceiveMessage, but then loops infinitely:

[2024-08-04 16:07:29,165: DEBUG/MainProcess] Event choose-signer.sqs.ReceiveMessage: calling handler <function set_operation_specific_signer at 0x105778360>
[2024-08-04 16:07:29,166: DEBUG/MainProcess] Calculating signature using v4 auth.
[2024-08-04 16:07:29,166: DEBUG/MainProcess] CanonicalRequest:
POST
/

content-type:application/x-amz-json-1.0
host:7399377469860198052.demo.smoothmq.com:3001
x-amz-date:20240804T200729Z
x-amz-target:AmazonSQS.ReceiveMessage

content-type;host;x-amz-date;x-amz-target
1a762bf8b84958772e686dd62bcc0d4c11957890e258359aaf2d6c9f894d2c12
[2024-08-04 16:07:29,166: DEBUG/MainProcess] StringToSign:
AWS4-HMAC-SHA256
20240804T200729Z
20240804/us-east-1/sqs/aws4_request
3d33e059354446a823a67b47de5ad55fdb7ac7326d99013d3c147ceb4c84a6cf
[2024-08-04 16:07:29,166: DEBUG/MainProcess] Signature:
10cb9d37b40482dc470e6b17b556f7373a43399d6b3e39d65b0494d0192b5904

Interestingly, the following scenarios do work:

Running SmoothMQ locally

This will successfully connect and consume messages - non-HTTPS:

$ go run . server


app = Celery(
    "tasks",
    broker_url='sqs://DEV_ACCESS_KEY_ID:DEV_SECRET_ACCESS_KEY@localhost:3001',
    broker_transport_options={'is_secure': False}
)

Kombu without Celery

This will also successfully consume messages, even over HTTPS:


from kombu import Connection

with Connection(f'sqs://7399377469860198052:27018ec9cb39d6d4d09b0f0378d3991c@7399377469860198052.demo.smoothmq.com:3001', transport_options={'is_secure':True}) as conn:
    simple_queue = conn.SimpleQueue('q1')

    simple_queue.put('simple queue message')

    message = simple_queue.get(block=True, timeout=1)
    print(f'Received: {message.payload}')
    message.ack()
    simple_queue.close()

poundifdef / smoothmq