BUG: Celery container halts when running the application locally

[kalbfled]

Description

• [ ] Ticket is understood, and QA has been contacted (if the ticket has a QA label).

Bringing up the Notify application locally yields a traceback that halts the Celery container.

celery-1      | [2024-07-18 21:18:17,135: CRITICAL/MainProcess] Unrecoverable error: ClientError('An error occurred (500) when calling the ListQueues operation (reached max retries: 4): <?xml version=\'1.0\' encoding=\'utf-8\'?>\n<ErrorResponse xmlns="http://queue.amazonaws.com/doc/2012-11-05/"><Error><Code>InternalError</Code><Message>exception while calling sqs with unknown operation: Operation detection failed. Missing Action in request for query-protocol service ServiceModel(sqs).</Message></Error><RequestId>8002f267-5550-4b19-9f9c-73c6e94314fb</RequestId></ErrorResponse>')
celery-1      | Traceback (most recent call last):
celery-1      |   File "/.venv/lib/python3.10/site-packages/kombu/transport/virtual/base.py", line 951, in create_channel
celery-1      |     return self._avail_channels.pop()
celery-1      | IndexError: pop from empty list
celery-1      | 
celery-1      | During handling of the above exception, another exception occurred:
celery-1      | 
celery-1      | Traceback (most recent call last):
celery-1      |   File "/.venv/lib/python3.10/site-packages/celery/worker/worker.py", line 202, in start
celery-1      |     self.blueprint.start(self)
celery-1      |   File "/.venv/lib/python3.10/site-packages/celery/bootsteps.py", line 116, in start
celery-1      |     step.start(parent)
celery-1      |   File "/.venv/lib/python3.10/site-packages/celery/bootsteps.py", line 365, in start
celery-1      |     return self.obj.start()
celery-1      |   File "/.venv/lib/python3.10/site-packages/celery/worker/consumer/consumer.py", line 340, in start
celery-1      |     blueprint.start(self)
celery-1      |   File "/.venv/lib/python3.10/site-packages/celery/bootsteps.py", line 116, in start
celery-1      |     step.start(parent)
celery-1      |   File "/.venv/lib/python3.10/site-packages/celery/worker/consumer/connection.py", line 21, in start
celery-1      |     c.connection = c.connect()
celery-1      |   File "/.venv/lib/python3.10/site-packages/celery/worker/consumer/consumer.py", line 470, in connect
celery-1      |     conn = self.connection_for_read(heartbeat=self.amqheartbeat)
celery-1      |   File "/.venv/lib/python3.10/site-packages/celery/worker/consumer/consumer.py", line 476, in connection_for_read
celery-1      |     return self.ensure_connected(
celery-1      |   File "/.venv/lib/python3.10/site-packages/celery/worker/consumer/consumer.py", line 527, in ensure_connected
celery-1      |     conn = conn.ensure_connection(
celery-1      |   File "/.venv/lib/python3.10/site-packages/kombu/connection.py", line 406, in ensure_connection
celery-1      |     self._ensure_connection(*args, **kwargs)
celery-1      |   File "/.venv/lib/python3.10/site-packages/kombu/connection.py", line 459, in _ensure_connection
celery-1      |     return retry_over_time(
celery-1      |   File "/.venv/lib/python3.10/site-packages/kombu/utils/functional.py", line 318, in retry_over_time
celery-1      |     return fun(*args, **kwargs)
celery-1      |   File "/.venv/lib/python3.10/site-packages/kombu/connection.py", line 934, in _connection_factory
celery-1      |     self._connection = self._establish_connection()
celery-1      |   File "/.venv/lib/python3.10/site-packages/kombu/connection.py", line 860, in _establish_connection
celery-1      |     conn = self.transport.establish_connection()
celery-1      |   File "/.venv/lib/python3.10/site-packages/kombu/transport/virtual/base.py", line 975, in establish_connection
celery-1      |     self._avail_channels.append(self.create_channel(self))
celery-1      |   File "/.venv/lib/python3.10/site-packages/kombu/transport/virtual/base.py", line 953, in create_channel
celery-1      |     channel = self.Channel(connection)
celery-1      |   File "/.venv/lib/python3.10/site-packages/kombu/transport/SQS.py", line 263, in __init__
celery-1      |     self._update_queue_cache(self.queue_name_prefix)
celery-1      |   File "/.venv/lib/python3.10/site-packages/kombu/transport/SQS.py", line 293, in _update_queue_cache
celery-1      |     resp = self.sqs().list_queues(QueueNamePrefix=queue_name_prefix)
celery-1      |   File "/.venv/lib/python3.10/site-packages/botocore/client.py", line 565, in _api_call
celery-1      |     return self._make_api_call(operation_name, kwargs)
celery-1      |   File "/.venv/lib/python3.10/site-packages/botocore/client.py", line 1021, in _make_api_call
celery-1      |     raise error_class(parsed_response, operation_name)
celery-1      | botocore.exceptions.ClientError: An error occurred (500) when calling the ListQueues operation (reached max retries: 4): <?xml version='1.0' encoding='utf-8'?>
celery-1      | <ErrorResponse xmlns="http://queue.amazonaws.com/doc/2012-11-05/"><Error><Code>InternalError</Code><Message>exception while calling sqs with unknown operation: Operation detection failed. Missing Action in request for query-protocol service ServiceModel(sqs).</Message></Error><RequestId>8002f267-5550-4b19-9f9c-73c6e94314fb</RequestId></ErrorResponse>
celery-1      | 2024-07-18T21:18:17 delivery delivery INFO no-request-id "Pool worker shutdown: pid = 45, exitcode = 0" [in /app/app/celery/celery.py:23]
celery-1      | 2024-07-18T21:18:17 delivery delivery INFO no-request-id "Pool worker shutdown: pid = 47, exitcode = 0" [in /app/app/celery/celery.py:23]
celery-1      | 2024-07-18T21:18:17 delivery delivery INFO no-request-id "Pool worker shutdown: pid = 46, exitcode = 0" [in /app/app/celery/celery.py:23]
celery-1      | [2024-07-18 21:18:17,138: INFO/ForkPoolWorker-1] Pool worker shutdown: pid = 45, exitcode = 0
celery-1      | [2024-07-18 21:18:17,138: INFO/ForkPoolWorker-3] Pool worker shutdown: pid = 47, exitcode = 0
celery-1      | [2024-07-18 21:18:17,138: INFO/ForkPoolWorker-2] Pool worker shutdown: pid = 46, exitcode = 0
celery-1      | 2024-07-18T21:18:17 delivery delivery INFO no-request-id "Pool worker shutdown: pid = 48, exitcode = 0" [in /app/app/celery/celery.py:23]
celery-1      | [2024-07-18 21:18:17,138: INFO/ForkPoolWorker-4] Pool worker shutdown: pid = 48, exitcode = 0
celery-1 exited with code 1

Steps to Reproduce

1. Attempt to start the application locally as described in the README file.

Impact/Urgency

The changes for #1854 includes a new Celery task that doesn't seem to get run. Not having Celery running locally is forcing us to test all prospective fixes by deploying to Dev. This is very slow.

Expected Behavior

• [ ] All containers run locally without crashing

Additional Info & Resources

[k-macmillan]

Cannot replicate. Blew away all containers, images, and volumes and built per the documentation without issue.