SQS elasticmq connection failed

[brettlangdon]

I am trying to use https://github.com/adamw/elasticmq in development with celery but I am receiving an error from kombu.

I am running elasticmq and celery in separate containers via docker-compose and trying to override the endpoint_url via BROKER_URL=sqs://key:secret@sqs:9324.

$ docker-compose run --rm worker
/root/.local/share/virtualenvs/app-lp47FrbD/lib/python3.6/site-packages/celery/platforms.py:795: RuntimeWarning: You're running the worker with superuser privileges: this is
absolutely not recommended!

Please specify a different user using the -u option.

User information: uid=0 euid=0 gid=0 egid=0

  uid=uid, euid=euid, gid=gid, egid=egid,

 -------------- celery@1813418a8fc5 v4.1.0 (latentcall)
---- **** -----
--- * ***  * -- Linux-4.9.87-linuxkit-aufs-x86_64-with 2018-04-28 01:12:11
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app:         tasks:0x7fd3e2bcceb8
- ** ---------- .> transport:   sqs://key:**@sqs:9324//
- ** ---------- .> results:     disabled://
- *** --- * --- .> concurrency: 4 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
 -------------- [queues]
                .> celery           exchange=celery(direct) key=celery

[tasks]

[2018-04-28 01:12:11,318: INFO/MainProcess] Starting new HTTP connection (1): sqs
[2018-04-28 01:12:11,338: INFO/MainProcess] Connected to sqs://key:**@sqs:9324//
[2018-04-28 01:12:11,368: INFO/MainProcess] Starting new HTTP connection (1): sqs
[2018-04-28 01:12:11,402: INFO/MainProcess] celery@1813418a8fc5 ready.
[2018-04-28 01:12:13,415: ERROR/MainProcess] Error in timer: Exception('Request Empty body  HTTP 599  Failed to connect to localhost port 9324: Connection refused (None)',)
Traceback (most recent call last):
  File "/root/.local/share/virtualenvs/app-lp47FrbD/lib/python3.6/site-packages/kombu/async/hub.py", line 136, in fire_timers
    entry()
  File "/root/.local/share/virtualenvs/app-lp47FrbD/lib/python3.6/site-packages/kombu/async/timer.py", line 68, in __call__
    return self.fun(*self.args, **self.kwargs)
  File "/root/.local/share/virtualenvs/app-lp47FrbD/lib/python3.6/site-packages/kombu/async/timer.py", line 127, in _reschedules
    return fun(*args, **kwargs)
  File "/root/.local/share/virtualenvs/app-lp47FrbD/lib/python3.6/site-packages/kombu/async/http/curl.py", line 108, in _timeout_check
    self._process_pending_requests()
  File "/root/.local/share/virtualenvs/app-lp47FrbD/lib/python3.6/site-packages/kombu/async/http/curl.py", line 132, in _process_pending_requests
    self._process(curl, errno, reason)
  File "/root/.local/share/virtualenvs/app-lp47FrbD/lib/python3.6/site-packages/kombu/async/http/curl.py", line 178, in _process
    buffer=buffer, effective_url=effective_url, error=error,
  File "/root/.local/share/virtualenvs/app-lp47FrbD/lib/python3.6/site-packages/vine/promises.py", line 150, in __call__
    svpending(*ca, **ck)
  File "/root/.local/share/virtualenvs/app-lp47FrbD/lib/python3.6/site-packages/vine/promises.py", line 143, in __call__
    return self.throw()
  File "/root/.local/share/virtualenvs/app-lp47FrbD/lib/python3.6/site-packages/vine/promises.py", line 140, in __call__
    retval = fun(*final_args, **final_kwargs)
  File "/root/.local/share/virtualenvs/app-lp47FrbD/lib/python3.6/site-packages/vine/funtools.py", line 100, in _transback
    return callback(ret)
  File "/root/.local/share/virtualenvs/app-lp47FrbD/lib/python3.6/site-packages/vine/promises.py", line 143, in __call__
    return self.throw()
  File "/root/.local/share/virtualenvs/app-lp47FrbD/lib/python3.6/site-packages/vine/promises.py", line 140, in __call__
    retval = fun(*final_args, **final_kwargs)
  File "/root/.local/share/virtualenvs/app-lp47FrbD/lib/python3.6/site-packages/vine/funtools.py", line 98, in _transback
    callback.throw()
  File "/root/.local/share/virtualenvs/app-lp47FrbD/lib/python3.6/site-packages/vine/funtools.py", line 96, in _transback
    ret = filter_(*args + (ret,), **kwargs)
  File "/root/.local/share/virtualenvs/app-lp47FrbD/lib/python3.6/site-packages/kombu/async/aws/connection.py", line 233, in _on_list_ready
    raise self._for_status(response, response.read())
Exception: Request Empty body  HTTP 599  Failed to connect to localhost port 9324: Connection refused (None)

$ docker-compose run --rm worker celery -A tasks report

software -> celery:4.1.0 (latentcall) kombu:4.1.0 py:3.6.5
            billiard:3.5.0.3 sqs:N/A
platform -> system:Linux arch:64bit imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:sqs results:disabled

BROKER_TRANSPORT_OPTIONS: {
 'region': 'us-east-1'}
BROKER_URL: 'sqs://key:********@sqs:9324//'
GPG_KEY: '********'
HOME: '/root'
HOSTNAME: '9b12129467c8'
LANG: 'C.UTF-8'
PATH: '/root/.local/share/virtualenvs/app-lp47FrbD/bin:/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'
PIP_PYTHON_PATH: '/usr/local/bin/python'
PYTHONDONTWRITEBYTECODE: 1
PYTHON_PIP_VERSION: '10.0.0'
PYTHON_VERSION: '3.6.5'
TERM: 'xterm'

I am not sure where the localhost is coming from as everything else appears to be correct.

I tried using pdb to ensure the transport/connection has the correct endpoint_url, which is does. Not sure where/how else to debug this issue.

I have also tested the link between the two containers via curl and everything is working as expected there.

Any insights would be greatly appreciated :)

[brettlangdon]

I figured it out, this was an issue with elasticmqs configuration.

By default it will return queue urls with the hostname localhost. It needed to be configured to return my expected docker container hostname.

I am running https://github.com/localstack/localstack so this meant setting the HOSTNAME and HOSTNAME_EXTERNAL environment variables to the name of my localstack docker-compose service.

[georgepsarakis]

@brettlangdon thanks for sharing the solution, it might help others.