search

jwtly10 / reddit-tiktok-gen

An automation application for RedditStories on TikTok
https://www.tiktok.com/@reddreelstoryz
MIT License
5 stars 2 forks source link

issue with celery worker and matplotlib #2

Open sonicboom171 opened 3 weeks ago

sonicboom171 commented 3 weeks ago

downloaded the git unzipped installed all dependencies and matplot lib seperatly.

when running docker compose up build i get the following error

[+] Building 1.2s (33/33) FINISHED docker:desktop-linux => [web internal] load build definition from Dockerfile 0.0s => => transferring dockerfile: 845B 0.0s => [celery-worker internal] load metadata for docker.io/library/python:3.12-slim 0.6s => [web internal] load .dockerignore 0.0s => => transferring context: 2B 0.0s => [celery-worker 1/14] FROM docker.io/library/python:3.12-slim@sha256:59c7332a4a24373861c4a5f0eec2c92b87e3efeb8ddef011744ef9a751b 0.0s => [web internal] load build context 0.0s => => transferring context: 2.01kB 0.0s => CACHED [celery-worker 2/14] WORKDIR /app 0.0s => CACHED [celery-worker 3/14] RUN mkdir -p /app/data 0.0s => CACHED [celery-worker 4/14] RUN apt-get update && apt-get install -y --no-install-recommends ffmpeg fontconfig && rm - 0.0s => CACHED [web 5/14] COPY requirements.txt /app/ 0.0s => CACHED [web 6/14] RUN pip install --no-cache-dir -r requirements.txt 0.0s => CACHED [web 7/14] COPY ./app /app/app 0.0s => CACHED [web 8/14] COPY ./assets /app/assets 0.0s => CACHED [web 9/14] COPY ./docker-compose.yml /app/ 0.0s => CACHED [web 10/14] COPY ./Dockerfile /app/ 0.0s => CACHED [web 11/14] COPY ./.env-docker /app/ 0.0s => CACHED [web 12/14] COPY ./assets/Poppins-SemiBold.ttf /usr/share/fonts/ 0.0s => CACHED [web 13/14] COPY ./assets/Mont-HeavyDEMO.otf /usr/share/fonts/ 0.0s => CACHED [web 14/14] RUN fc-cache -fv 0.0s => [web] exporting to image 0.0s => => exporting layers 0.0s => => writing image sha256:f0de2fce183c6f11cd0be5efd15d19f857aab0470b43fb214226cef487556f1e 0.0s => => naming to docker.io/library/reddit-tiktok-gen-main-web 0.0s => [celery-worker internal] load build definition from Dockerfile 0.0s => => transferring dockerfile: 845B 0.0s => [celery-worker internal] load .dockerignore 0.0s => => transferring context: 2B 0.0s => [celery-worker internal] load build context 0.0s => => transferring context: 2.01kB 0.0s => CACHED [celery-worker 5/14] COPY requirements.txt /app/ 0.0s => CACHED [celery-worker 6/14] RUN pip install --no-cache-dir -r requirements.txt 0.0s => CACHED [celery-worker 7/14] COPY ./app /app/app 0.0s => CACHED [celery-worker 8/14] COPY ./assets /app/assets 0.0s => CACHED [celery-worker 9/14] COPY ./docker-compose.yml /app/ 0.0s => CACHED [celery-worker 10/14] COPY ./Dockerfile /app/ 0.0s => CACHED [celery-worker 11/14] COPY ./.env-docker /app/ 0.0s => CACHED [celery-worker 12/14] COPY ./assets/Poppins-SemiBold.ttf /usr/share/fonts/ 0.0s => CACHED [celery-worker 13/14] COPY ./assets/Mont-HeavyDEMO.otf /usr/share/fonts/ 0.0s => CACHED [celery-worker 14/14] RUN fc-cache -fv 0.0s => [celery-worker] exporting to image 0.0s => => exporting layers 0.0s => => writing image sha256:8cc3029879a53237e7a260c7211418f333a5de2b61f2a8231b2f69d7aab41fd6 0.0s => => naming to docker.io/library/reddit-tiktok-gen-main-celery-worker 0.0s [+] Running 4/0 ✔ Container reddit-tiktok-gen-main-redis-1 Created 0.0s ✔ Container reddit-tiktok-gen-main-gentle-aligner-1 Created 0.0s ✔ Container reddit-tiktok-gen-main-web-1 Created 0.0s ✔ Container reddit-tiktok-gen-main-celery-worker-1 Created 0.0s Attaching to celery-worker-1, gentle-aligner-1, redis-1, web-1 redis-1 | 1:C 17 Aug 2024 19:07:51.683 oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo redis-1 | 1:C 17 Aug 2024 19:07:51.683 Redis version=7.4.0, bits=64, commit=00000000, modified=0, pid=1, just started redis-1 | 1:C 17 Aug 2024 19:07:51.683 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
redis-1 | 1:M 17 Aug 2024 19:07:51.684 monotonic clock: POSIX clock_gettime redis-1 | 1:M 17 Aug 2024 19:07:51.685 Running mode=standalone, port=6379.
redis-1 | 1:M 17 Aug 2024 19:07:51.685 Server initialized
redis-1 | 1:M 17 Aug 2024 19:07:51.686 Loading RDB produced by version 7.4.0
redis-1 | 1:M 17 Aug 2024 19:07:51.686 RDB age 60823 seconds
redis-1 | 1:M 17 Aug 2024 19:07:51.686 RDB memory usage when created 0.90 Mb redis-1 | 1:M 17 Aug 2024 19:07:51.686 Done loading RDB, keys loaded: 0, keys expired: 0.
redis-1 | 1:M 17 Aug 2024 19:07:51.686 DB loaded from disk: 0.001 seconds
redis-1 | 1:M 17 Aug 2024 19:07:51.686 * Ready to accept connections tcp
gentle-aligner-1 | INFO:root:gentle 0.10.1
gentle-aligner-1 | INFO:root:listening at 0.0.0.0:8765 gentle-aligner-1 |
gentle-aligner-1 | INFO:root:SERVE 8765, 0.0.0.0, 1
gentle-aligner-1 | INFO:root:about to listen
gentle-aligner-1 | INFO:root:listening celery-worker-1 | Usage: celery [OPTIONS] COMMAND [ARGS]...
celery-worker-1 | Try 'celery --help' for help. celery-worker-1 |
celery-worker-1 | Error: Invalid value for '-A' / '--app':
celery-worker-1 | Unable to load celery application.
celery-worker-1 | While trying to load the module app.service.celery_app the following error occurred:
celery-worker-1 | Traceback (most recent call last):
celery-worker-1 | File "/usr/local/lib/python3.12/site-packages/celery/app/utils.py", line 383, in find_app celery-worker-1 | sym = symbol_by_name(app, imp=imp)
celery-worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
celery-worker-1 | File "/usr/local/lib/python3.12/site-packages/kombu/utils/imports.py", line 64, in symbol_by_name
celery-worker-1 | return getattr(module, cls_name) if cls_name else module
celery-worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^
celery-worker-1 | AttributeError: module 'app.service' has no attribute 'celery_app' celery-worker-1 |
celery-worker-1 | During handling of the above exception, another exception occurred:
celery-worker-1 |
celery-worker-1 | Traceback (most recent call last):
celery-worker-1 | File "/usr/local/lib/python3.12/site-packages/celery/bin/celery.py", line 58, in convert
celery-worker-1 | return find_app(value)
celery-worker-1 | ^^^^^^^^^^^^^^^ celery-worker-1 | File "/usr/local/lib/python3.12/site-packages/celery/app/utils.py", line 386, in find_app
celery-worker-1 | sym = imp(app)
celery-worker-1 | ^^^^^^^^
celery-worker-1 | File "/usr/local/lib/python3.12/site-packages/celery/utils/imports.py", line 109, in import_from_cwd
celery-worker-1 | return imp(module, package=package) celery-worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
celery-worker-1 | File "/usr/local/lib/python3.12/importlib/init.py", line 90, in import_module
celery-worker-1 | return _bootstrap._gcd_import(name[level:], package, level)
web-1 | File "", line 88, in _run_code web-1 | File "/app/app/webapp.py", line 9, in web-1 | from app.service.task import generate_video web-1 | File "/app/app/service/task.py", line 1, in web-1 | from app.service.celery_app import celery_app web-1 | File "/app/app/service/celery_app.py", line 2, in web-1 | import app.config web-1 | File "/app/app/config.py", line 3, in web-1 | import matplotlib.font_manager as font_manager web-1 | ModuleNotFoundError: No module named 'matplotlib' celery-worker-1 exited with code 2 web-1 exited with code 1

im not sure where the issue with celery worker is and would appreciate the help

jwtly10 commented 3 weeks ago

Hi @sonicboom171 ,

I have not worked on this project for a little while, and I must admit, I didnt do as much testing of the docker setup as I should have.

I don't have much time atm to go back through this project and debug all issues, but from a quick check it looks like I was missing matplotlib from the python dependencies.

If you have cloned/forked the repo you should be able to run the following from inside the directory (note this is for mac/linux)

python3 -m venv .venv # or just python -m venv .venv (depends on your python setup, it just needs to be python3)
source .venv/bin/activate # Use the virtual environment we just made
pip install -r requirements.txt # Install the dependencies locally 
pip install matplotlib # It seems this was missing, so lets add it
pip freeze > requirements.txt # Regenerate the dependencies file

I recommend using virtual environments as it just makes dependency management so much easier, and at least you can test what docker will try to install via the requirements.txt.

Now you should be able to run the docker command again - if there anything else throwing please let me know! I'll update the repo shortly when i get time.

sonicboom171 commented 3 weeks ago

thanks i was able to solve the matplotlib issue by adding it to the requirements, and was able to figure out there was a need for openai api inorder to work so i added that.

it seems to be running fine from docker but im unable to launch localhost, is there anything im doing wrong or need to alter in the

sonicboom171 commented 3 weeks ago

[+] Building 0.9s (33/33) FINISHED docker:desktop-linux => [web internal] load build definition from Dockerfile 0.0s => => transferring dockerfile: 868B 0.0s => [celery-worker internal] load metadata for docker.io/library/python:3.12-slim 0.4s => [web internal] load .dockerignore 0.0s => => transferring context: 2B 0.0s => [celery-worker 1/14] FROM docker.io/library/python:3.12-slim@sha256:59c7332a4a24373861c4a5f0eec2c92b87e3efeb8ddef011744ef9a751b1d11c 0.0s => [web internal] load build context 0.0s => => transferring context: 2.01kB 0.0s => CACHED [celery-worker 2/14] WORKDIR /app 0.0s => CACHED [celery-worker 3/14] RUN mkdir -p /app/data 0.0s => CACHED [celery-worker 4/14] RUN apt-get update && apt-get install -y --no-install-recommends ffmpeg fontconfig procps && 0.0s => CACHED [web 5/14] COPY requirements.txt /app/ 0.0s => CACHED [web 6/14] RUN pip install --no-cache-dir -r requirements.txt 0.0s => CACHED [web 7/14] COPY ./app /app/app 0.0s => CACHED [web 8/14] COPY ./assets /app/assets 0.0s => CACHED [web 9/14] COPY ./docker-compose.yml /app/ 0.0s => CACHED [web 10/14] COPY ./Dockerfile /app/ 0.0s => CACHED [web 11/14] COPY ./.env-docker /app/ 0.0s => CACHED [web 12/14] COPY ./assets/Poppins-SemiBold.ttf /usr/share/fonts/ 0.0s => CACHED [web 13/14] COPY ./assets/Mont-HeavyDEMO.otf /usr/share/fonts/ 0.0s => CACHED [web 14/14] RUN fc-cache -fv 0.0s => [web] exporting to image 0.0s => => exporting layers 0.0s => => writing image sha256:cff69fb2a53cf0cd33dd56293f43a4b6a7ad1ae1986cd1b0a4d09aa8e6f31f6c 0.0s => => naming to docker.io/library/reddit-tiktok-gen-main-web 0.0s => [celery-worker internal] load build definition from Dockerfile 0.0s => => transferring dockerfile: 868B 0.0s => [celery-worker internal] load .dockerignore 0.0s => => transferring context: 2B 0.0s => [celery-worker internal] load build context 0.0s => => transferring context: 2.01kB 0.0s => CACHED [celery-worker 5/14] COPY requirements.txt /app/ 0.0s => CACHED [celery-worker 6/14] RUN pip install --no-cache-dir -r requirements.txt 0.0s => CACHED [celery-worker 7/14] COPY ./app /app/app 0.0s => CACHED [celery-worker 8/14] COPY ./assets /app/assets 0.0s => CACHED [celery-worker 9/14] COPY ./docker-compose.yml /app/ 0.0s => CACHED [celery-worker 10/14] COPY ./Dockerfile /app/ 0.0s => CACHED [celery-worker 11/14] COPY ./.env-docker /app/ 0.0s => CACHED [celery-worker 12/14] COPY ./assets/Poppins-SemiBold.ttf /usr/share/fonts/ 0.0s => CACHED [celery-worker 13/14] COPY ./assets/Mont-HeavyDEMO.otf /usr/share/fonts/ 0.0s => CACHED [celery-worker 14/14] RUN fc-cache -fv 0.0s => [celery-worker] exporting to image 0.0s => => exporting layers 0.0s => => writing image sha256:746b69640130648fca39ca23910b6f9526f702f94fdceeb7a7ef69b71039e71d 0.0s => => naming to docker.io/library/reddit-tiktok-gen-main-celery-worker 0.0s [+] Running 2/0 ✔ Container reddit-tiktok-gen-main-gentle-aligner-1 Created 0.0s ✔ Container reddit-tiktok-gen-main-redis-1 Created 0.0s Attaching to celery-worker-1, gentle-aligner-1, redis-1, web-1 redis-1 | 1:C 17 Aug 2024 21:23:18.906 oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo redis-1 | 1:C 17 Aug 2024 21:23:18.906 Redis version=7.4.0, bits=64, commit=00000000, modified=0, pid=1, just started redis-1 | 1:C 17 Aug 2024 21:23:18.906 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
redis-1 | 1:M 17 Aug 2024 21:23:18.906 monotonic clock: POSIX clock_gettime redis-1 | 1:M 17 Aug 2024 21:23:18.907 Running mode=standalone, port=6379. redis-1 | 1:M 17 Aug 2024 21:23:18.907 Server initialized
redis-1 | 1:M 17 Aug 2024 21:23:18.908 Ready to accept connections tcp gentle-aligner-1 | INFO:root:gentle 0.10.1
gentle-aligner-1 | INFO:root:listening at 0.0.0.0:8765 gentle-aligner-1 |
gentle-aligner-1 | INFO:root:SERVE 8765, 0.0.0.0, 1
gentle-aligner-1 | INFO:root:about to listen
gentle-aligner-1 | INFO:root:listening Gracefully stopping... (press Ctrl+C again to force)
dependency failed to start: container reddit-tiktok-gen-main-gentle-aligner-1 is unhealthy

seem to be getting this error which is preventing web interface from working now

jwtly10 commented 3 weeks ago

@sonicboom171

The gentle-aligner container is a 3rd party dependency, so I cant trouble shoot without seeing the logs from the container instance itself. https://github.com/lowerquality/gentle This is the original project, if you want some more context.

You could try to run the container itself like docker run -P lowerquality/gentle to see if its just an issue with your docker setup, but this should work out of the box as I am making no modifications to that service.

However, I can see the aligner did manage to start up and begin listening so I cant specifically see where the error has happened. Perhaps clean all the images docker has downloaded previously and start again?

sonicboom171 commented 3 weeks ago

thanks for the replies, gentle aligner is working and i can access it from 127.0.01:8765 for some reason web1 never sees it as running.

i also found another issues which ived fixed but just an FYI celery-worker-1 | Traceback (most recent call last):
web-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ celery-worker-1 | File "/usr/local/lib/python3.12/site-packages/celery/bin/celery.py", line 58, in convert
web-1 | File "/usr/local/lib/python3.12/subprocess.py", line 1026, in init
celery-worker-1 | return find_app(value)
web-1 | self._execute_child(args, executable, preexec_fn, close_fds,
celery-worker-1 | ^^^^^^^^^^^^^^^
web-1 | File "/usr/local/lib/python3.12/subprocess.py", line 1955, in _execute_child
celery-worker-1 | File "/usr/local/lib/python3.12/site-packages/celery/app/utils.py", line 386, in find_app web-1 | raise child_exception_type(errno_num, err_msg, err_filename)
celery-worker-1 | sym = imp(app)
web-1 | FileNotFoundError: [Errno 2] No such file or directory: 'pgrep'
celery-worker-1 | ^^^^^^^^
celery-worker-1 | File "/usr/local/lib/python3.12/site-packages/celery/utils/imports.py", line 109, in import_from_cwd
celery-worker-1 | return imp(module, package=package)
celery-worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ celery-worker-1 | File "/usr/local/lib/python3.12/importlib/init.py", line 90, in import_module
celery-worker-1 | return _bootstrap._gcd_import(name[level:], package, level)
celery-worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
celery-worker-1 | File "", line 1387, in _gcd_import
celery-worker-1 | File "", line 1360, in _find_and_load
celery-worker-1 | File "", line 1331, in _find_and_load_unlocked
celery-worker-1 | File "", line 935, in _load_unlocked celery-worker-1 | File "", line 995, in exec_module
celery-worker-1 | File "", line 488, in _call_with_frames_removed
celery-worker-1 | File "/app/app/service/celery_app.py", line 2, in
celery-worker-1 | import app.config
celery-worker-1 | File "/app/app/config.py", line 96, in
celery-worker-1 | load_env() celery-worker-1 | File "/app/app/config.py", line 46, in load_env
celery-worker-1 | if not check_process_running("gentle"):
celery-worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
celery-worker-1 | File "/app/app/config.py", line 18, in check_process_running
celery-worker-1 | output = subprocess.run(["pgrep", "-af", process_name], capture_output=True, text=True)
celery-worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
celery-worker-1 | File "/usr/local/lib/python3.12/subprocess.py", line 548, in run celery-worker-1 | with Popen(*popenargs, **kwargs) as process:
celery-worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^
celery-worker-1 | File "/usr/local/lib/python3.12/subprocess.py", line 1026, in init
celery-worker-1 | self._execute_child(args, executable, preexec_fn, close_fds,
celery-worker-1 | File "/usr/local/lib/python3.12/subprocess.py", line 1955, in _execute_child
celery-worker-1 | raise child_exception_type(errno_num, err_msg, err_filename)
celery-worker-1 | FileNotFoundError: [Errno 2] No such file or directory: 'pgrep'
celery-worker-1 | celery-worker-1 exited with code 2