nickjj / docker-flask-example

A production ready example Flask app that's using Docker and Docker Compose.
MIT License
617 stars 102 forks source link

i'm trying to learn celery and i don't understand an error #24

Closed garethjax closed 1 month ago

garethjax commented 1 month ago

the premise is that i'm trying to understand how this configuration works (and probably i'm being WAY ahead of myself).

I have a bit of experience with docker and python but it's not my main activity.

Since i want to do manage task with celery, but i want to keep a proper log of the activities, i was thinking about using redis as the broker url and postgress as the result_backend.

So i've tried this config (by asking ChatGPT.... i know, i know)

Celery.

CELERY_CONFIG = { "broker_url": os.getenv("REDIS_URL", "redis://redis:6379/0"), "result_backend": os.getenv("DATABASE_URL", db), "include": [], }

But when i try to rebuild the containers, the worker images display this error:

worker-1 | [2024-08-16 12:30:19,934: CRITICAL/MainProcess] Unrecoverable error: ModuleNotFoundError("No module named 'postgresql'") worker-1 | Traceback (most recent call last): worker-1 | File "/home/python/.local/lib/python3.12/site-packages/celery/worker/worker.py", line 202, in start worker-1 | self.blueprint.start(self) worker-1 | File "/home/python/.local/lib/python3.12/site-packages/celery/bootsteps.py", line 112, in start worker-1 | self.on_start() worker-1 | File "/home/python/.local/lib/python3.12/site-packages/celery/apps/worker.py", line 135, in on_start worker-1 | self.emit_banner() worker-1 | File "/home/python/.local/lib/python3.12/site-packages/celery/apps/worker.py", line 169, in emit_banner worker-1 | ' \n', self.startup_info(artlines=not use_image))), worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ worker-1 | File "/home/python/.local/lib/python3.12/site-packages/celery/apps/worker.py", line 231, in startup_info worker-1 | results=self.app.backend.as_uri(), worker-1 | ^^^^^^^^^^^^^^^^ worker-1 | File "/home/python/.local/lib/python3.12/site-packages/celery/app/base.py", line 1303, in backend worker-1 | self._backend = self._get_backend() worker-1 | ^^^^^^^^^^^^^^^^^^^ worker-1 | File "/home/python/.local/lib/python3.12/site-packages/celery/app/base.py", line 968, in _get_backend worker-1 | backend, url = backends.by_url( worker-1 | ^^^^^^^^^^^^^^^^ worker-1 | File "/home/python/.local/lib/python3.12/site-packages/celery/app/backends.py", line 69, in by_url worker-1 | return by_name(backend, loader), url worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^^ worker-1 | File "/home/python/.local/lib/python3.12/site-packages/celery/app/backends.py", line 49, in by_name worker-1 | cls = symbol_by_name(backend, aliases) worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ worker-1 | File "/home/python/.local/lib/python3.12/site-packages/kombu/utils/imports.py", line 59, in symbol_by_name worker-1 | module = imp(module_name, package=package, **kwargs) worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ worker-1 | File "/usr/local/lib/python3.12/importlib/init.py", line 90, in import_module worker-1 | return _bootstrap._gcd_import(name[level:], package, level) worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ worker-1 | File "", line 1387, in _gcd_import worker-1 | File "", line 1360, in _find_and_load worker-1 | File "", line 1324, in _find_and_load_unlocked worker-1 | ModuleNotFoundError: No module named 'postgresql' web-1 | [2024-08-16 12:30:19 +0000] [23] [INFO] Booting worker with pid: 23 worker-1 exited with code 1

things i've tried

i've altered the dockerfile to include a pip install

RUN if [ "${FLASK_DEBUG}" != "true" ]; then \ ln -s /public /app/public && flask digest compile && rm -rf /app/public; fi RUN pip install psycopg2-binary

But i'm still having the error.

What i'm doing wrong ?

nickjj commented 1 month ago

Hi,

This project already installs the psycopg library btw. You can see it referenced here in the connection string: https://github.com/nickjj/docker-flask-example/blob/b31f76fe9eb282c1a850b074db18433bcf9304e6/config/settings.py#L16

As for the issue around using postgres as your back-end, you didn't include what you have set for that so I'm not sure. You probably need to use postgresql+psycopg not postgresql.

I'd suggest checking out their docs: http://docs.celeryproject.org/en/latest/userguide/configuration.html#database-backend-settings

Sorry, I'm going to close this one as it's not related to this project and I haven't use Postgres as a results back-end for Celery. Feel free to reply back if get it working though.