Issue Starting with AMQP Backend with Python 3.7

[dmlb2000]

I'm running the code through celery and an not sure how to get the right error message out of this stack dump. Seems like an error occurred and then when celery/kombu tried to handle it that process also barfed.

ingestbackend_1     | [2018-07-11 04:25:25,661: CRITICAL/MainProcess] Unrecoverable error: SyntaxError('invalid syntax', ('/usr/local/lib/python3.7/site-packages/celery/backends/rpc.py', 20, 11, 'from .async import AsyncBackendMixin, BaseResultConsumer\n'))
ingestbackend_1     | Traceback (most recent call last):
ingestbackend_1     |   File "/usr/local/lib/python3.7/site-packages/kombu/utils/objects.py", line 42, in __get__
ingestbackend_1     |     return obj.__dict__[self.__name__]
ingestbackend_1     | KeyError: 'backend'
ingestbackend_1     |
ingestbackend_1     | During handling of the above exception, another exception occurred:
ingestbackend_1     |
ingestbackend_1     | Traceback (most recent call last):
ingestbackend_1     |   File "/usr/local/lib/python3.7/site-packages/celery/worker/worker.py", line 205, in start
ingestbackend_1     |     self.blueprint.start(self)
ingestbackend_1     |   File "/usr/local/lib/python3.7/site-packages/celery/bootsteps.py", line 115, in start
ingestbackend_1     |     self.on_start()
ingestbackend_1     |   File "/usr/local/lib/python3.7/site-packages/celery/apps/worker.py", line 139, in on_start
ingestbackend_1     |     self.emit_banner()
ingestbackend_1     |   File "/usr/local/lib/python3.7/site-packages/celery/apps/worker.py", line 154, in emit_banner
ingestbackend_1     |     ' \n', self.startup_info(artlines=not use_image))),
ingestbackend_1     |   File "/usr/local/lib/python3.7/site-packages/celery/apps/worker.py", line 217, in startup_info
ingestbackend_1     |     results=self.app.backend.as_uri(),
ingestbackend_1     |   File "/usr/local/lib/python3.7/site-packages/kombu/utils/objects.py", line 44, in __get__
ingestbackend_1     |     value = obj.__dict__[self.__name__] = self.__get(obj)
ingestbackend_1     |   File "/usr/local/lib/python3.7/site-packages/celery/app/base.py", line 1196, in backend
ingestbackend_1     |     return self._get_backend()
ingestbackend_1     |   File "/usr/local/lib/python3.7/site-packages/celery/app/base.py", line 914, in _get_backend
ingestbackend_1     |     self.loader)
ingestbackend_1     |   File "/usr/local/lib/python3.7/site-packages/celery/app/backends.py", line 70, in by_url
ingestbackend_1     |     return by_name(backend, loader), url
ingestbackend_1     |   File "/usr/local/lib/python3.7/site-packages/celery/app/backends.py", line 50, in by_name
ingestbackend_1     |     cls = symbol_by_name(backend, aliases)
ingestbackend_1     |   File "/usr/local/lib/python3.7/site-packages/kombu/utils/imports.py", line 56, in symbol_by_name
ingestbackend_1     |     module = imp(module_name, package=package, **kwargs)
ingestbackend_1     |   File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module
ingestbackend_1     |     return _bootstrap._gcd_import(name[level:], package, level)
ingestbackend_1     |   File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
ingestbackend_1     |   File "<frozen importlib._bootstrap>", line 983, in _find_and_load
ingestbackend_1     |   File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
ingestbackend_1     |   File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
ingestbackend_1     |   File "<frozen importlib._bootstrap_external>", line 724, in exec_module
ingestbackend_1     |   File "<frozen importlib._bootstrap_external>", line 860, in get_code
ingestbackend_1     |   File "<frozen importlib._bootstrap_external>", line 791, in source_to_code
ingestbackend_1     |   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ingestbackend_1     |   File "/usr/local/lib/python3.7/site-packages/celery/backends/rpc.py", line 20
ingestbackend_1     |     from .async import AsyncBackendMixin, BaseResultConsumer
ingestbackend_1     |               ^
ingestbackend_1     | SyntaxError: invalid syntax

Any thoughts on how to debug this would be good.

[dmlb2000]

Pip freeze output...

amqp==2.3.2
asn1crypto==0.24.0
backports.functools-lru-cache==1.5
billiard==3.5.0.4
celery==4.2.0
certifi==2018.4.16
cffi==1.11.5
chardet==3.0.4
cheroot==6.3.3
CherryPy==16.0.3
cryptography==2.2.2
idna==2.7
kombu==4.2.1
librabbitmq==2.0.0
more-itertools==4.2.0
peewee==3.5.2
portend==2.3
pycparser==2.18
PyMySQL==0.9.2
pytz==2018.5
requests==2.19.1
six==1.11.0
tempora==1.12
urllib3==1.23
vine==1.1.4

[dmlb2000]

Also using Python 3.7

[auvipy]

seems legit. Will take care of it

[auvipy]

well seems it's celery issue and its fixed in celery master recently. celery isn;t officially supporting 3.7 yet, so either down grade to 3.6 or try celery master and report further issues

[grzegorzk]

I see similar problem on 3.6 (however as it happens it is not exactly the same as it manifested to original poster):

File "/lib/python3.6/site-packages/celery/app/task.py", line 408, in delay
  return self.apply_async(args, kwargs)
File "/lib/python3.6/site-packages/celery/app/task.py", line 535, in apply_async
  **options
File "/lib/python3.6/site-packages/celery/app/base.py", line 741, in send_task
  with self.producer_or_acquire(producer) as P:
File "/lib/python3.6/site-packages/celery/app/base.py", line 876, in producer_or_acquire
  producer, self.producer_pool.acquire, block=True,
File "/lib/python3.6/site-packages/celery/app/base.py", line 1246, in producer_pool
  return self.amqp.producer_pool
File "/lib/python3.6/site-packages/celery/app/amqp.py", line 612, in producer_pool
  self.app.connection_for_write()]
File "/lib/python3.6/site-packages/celery/app/base.py", line 773, in connection_for_write
  return self._connection(url or self.conf.broker_write_url, **kwargs)
File "/lib/python3.6/site-packages/celery/app/base.py", line 841, in _connection
  'broker_connection_timeout', connect_timeout
File "/lib/python3.6/site-packages/kombu/connection.py", line 189, in __init__
  self._init_params(**params)
File "/lib/python3.6/site-packages/kombu/connection.py", line 235, in _init_params
  if transport == 'amqp' and supports_librabbitmq():
File "/lib/python3.6/site-packages/kombu/transport/__init__.py", line 13, in supports_librabbitmq
  import librabbitmq  # noqa
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 951, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 894, in _find_spec
File "<frozen importlib._bootstrap_external>", line 1157, in find_spec
File "<frozen importlib._bootstrap_external>", line 1126, in _get_spec
File "<frozen importlib._bootstrap_external>", line 1090, in _path_importer_cache
  PermissionError: [Errno 13] Permission denied

amqp 2.3.2
celery 4.2.0
kombu 4.2.1

(I do not have librabbitmq installed, but it was never causing problems for me before)

[yznts]

Same issue, when using redis.

app.py

...
celery_app = Celery('tasks', backend='redis://localhost:6379', broker='redis://localhost:6379')
...

$ celery -A app.celery_app worker --loglevel=info

[2018-09-13 09:30:26,584: CRITICAL/MainProcess] Unrecoverable error: SyntaxError('invalid syntax', ('/Users/yzinets/Desktop/RealEstate/realestate-bac
kend/venv/lib/python3.7/site-packages/celery/backends/redis.py', 22, 19, 'from . import async, base\n'))
Traceback (most recent call last):
  File "/Users/yzinets/Desktop/RealEstate/realestate-backend/venv/lib/python3.7/site-packages/kombu/utils/objects.py", line 42, in __get__
    return obj.__dict__[self.__name__]
KeyError: 'backend'

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/Users/yzinets/Desktop/RealEstate/realestate-backend/venv/lib/python3.7/site-packages/celery/worker/worker.py", line 205, in start
    self.blueprint.start(self)
  File "/Users/yzinets/Desktop/RealEstate/realestate-backend/venv/lib/python3.7/site-packages/celery/bootsteps.py", line 115, in start    self.on_start()
  File "/Users/yzinets/Desktop/RealEstate/realestate-backend/venv/lib/python3.7/site-packages/celery/apps/worker.py", line 139, in on_start
    self.emit_banner()
  File "/Users/yzinets/Desktop/RealEstate/realestate-backend/venv/lib/python3.7/site-packages/celery/apps/worker.py", line 154, in emit_banner
    ' \n', self.startup_info(artlines=not use_image))),
  File "/Users/yzinets/Desktop/RealEstate/realestate-backend/venv/lib/python3.7/site-packages/celery/apps/worker.py", line 217, in startup_info
    results=self.app.backend.as_uri(),
  File "/Users/yzinets/Desktop/RealEstate/realestate-backend/venv/lib/python3.7/site-packages/kombu/utils/objects.py", line 44, in __get__
    value = obj.__dict__[self.__name__] = self.__get(obj)
  File "/Users/yzinets/Desktop/RealEstate/realestate-backend/venv/lib/python3.7/site-packages/celery/app/base.py", line 1196, in backend
    return self._get_backend()
  File "/Users/yzinets/Desktop/RealEstate/realestate-backend/venv/lib/python3.7/site-packages/celery/app/base.py", line 914, in _get_backend
    self.loader)
  File "/Users/yzinets/Desktop/RealEstate/realestate-backend/venv/lib/python3.7/site-packages/celery/app/backends.py", line 70, in by_url
    return by_name(backend, loader), url
  File "/Users/yzinets/Desktop/RealEstate/realestate-backend/venv/lib/python3.7/site-packages/celery/app/backends.py", line 50, in by_name
    cls = symbol_by_name(backend, aliases)
  File "/Users/yzinets/Desktop/RealEstate/realestate-backend/venv/lib/python3.7/site-packages/kombu/utils/imports.py", line 56, in symbol_by_name
    module = imp(module_name, package=package, **kwargs)
  File "/Users/yzinets/Desktop/RealEstate/realestate-backend/venv/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 724, in exec_module
  File "<frozen importlib._bootstrap_external>", line 860, in get_code
  File "<frozen importlib._bootstrap_external>", line 791, in source_to_code
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Users/yzinets/Desktop/RealEstate/realestate-backend/venv/lib/python3.7/site-packages/celery/backends/redis.py", line 22
    from . import async, base
                      ^
SyntaxError: invalid syntax

Versions: MacOS High Sierra 10.13 pip 18.0 python 3.7.0 celery 4.2.1

[yznts]

Also, here is possible solution, but with modification of library code. https://stackoverflow.com/questions/51273659/trouble-in-setting-celery-tasks-backend-in-python/51308631