hiftikha
commented
5 months ago Taking a look at this
Closed luke-thorburn closed 5 months ago
hiftikha
commented
5 months ago Taking a look at this
hiftikha
commented
5 months ago If yes you need to now use the format docker build -f docker/Dockerfile.job_scheduler -t job_scheduler . for example for the job scheduler, as the build context is now root. I will update the read me accordingly. Refer here for more info: https://github.com/HumanCompatibleAI/ranking-challenge/pull/46
luke-thorburn
commented
5 months ago Full output of make run:
[luke@luke-pc feed-span]$ sudo make run
[sudo] password for luke:
docker compose up --build
WARN[0000] The "TWITTER_SESSION_COOKIE" variable is not set. Defaulting to a blank string.
WARN[0000] The "TWITTER_USERNAME" variable is not set. Defaulting to a blank string.
WARN[0000] The "TWITTER_EMAIL" variable is not set. Defaulting to a blank string.
[+] Building 4.2s (51/62) docker:default
=> [celery-scorer-worker0 internal] load build definition from Dockerfile.scorer_worker 0.0s
=> => transferring dockerfile: 594B 0.0s
=> [scraper-ingester internal] load metadata for docker.io/library/python:3.11 0.5s
=> [ranker internal] load build definition from Dockerfile.ranking_server 0.0s
=> => transferring dockerfile: 449B 0.0s
=> [scraper-ingester internal] load build definition from Dockerfile.scraper_worker.ingester 0.0s
=> => transferring dockerfile: 468B 0.0s
=> [job-scheduler internal] load build definition from Dockerfile.job_scheduler 0.0s
=> => transferring dockerfile: 542B 0.0s
=> [celery-scorer-worker1 internal] load build definition from Dockerfile.scorer_worker 0.0s
=> => transferring dockerfile: 594B 0.0s
=> [celery-scraper-worker0 internal] load build definition from Dockerfile.scraper_worker.scraper 0.0s
=> => transferring dockerfile: 467B 0.0s
=> [celery-worker internal] load build definition from Dockerfile.sandbox_worker 0.0s
=> => transferring dockerfile: 501B 0.0s
=> [celery-scraper-worker0 internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [ranker internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [scraper-ingester internal] load .dockerignore 0.1s
=> => transferring context: 2B 0.0s
=> [job-scheduler internal] load .dockerignore 0.1s
=> => transferring context: 2B 0.0s
=> [celery-scorer-worker1 internal] load .dockerignore 0.1s
=> => transferring context: 2B 0.0s
=> [celery-scorer-worker0 internal] load .dockerignore 0.1s
=> => transferring context: 2B 0.0s
=> [celery-worker internal] load .dockerignore 0.1s
=> => transferring context: 2B 0.0s
=> [celery-scraper-worker0 internal] load build context 0.3s
=> => transferring context: 20.39kB 0.2s
=> [celery-worker 1/7] FROM docker.io/library/python:3.11@sha256:091e0f5da680e5c972c59cb7eca172141bb6350045b592c284e2fd3bf2916dd9 0.2s
=> => resolve docker.io/library/python:3.11@sha256:091e0f5da680e5c972c59cb7eca172141bb6350045b592c284e2fd3bf2916dd9 0.0s
=> [ranker internal] load build context 0.3s
=> => transferring context: 68B 0.3s
=> [celery-scorer-worker0 internal] load build context 0.3s
=> => transferring context: 20.39kB 0.2s
=> [job-scheduler internal] load build context 0.3s
=> => transferring context: 20.39kB 0.2s
=> [scraper-ingester internal] load build context 0.2s
=> => transferring context: 20.39kB 0.2s
=> [celery-worker internal] load build context 0.2s
=> => transferring context: 20.39kB 0.2s
=> [celery-scorer-worker1 internal] load build context 3.4s
=> => transferring context: 312.31MB 3.4s
=> CACHED [celery-scorer-worker1 2/7] WORKDIR /app 0.0s
=> CACHED [celery-scorer-worker1 3/7] RUN pip install poetry 0.0s
=> CACHED [celery-scraper-worker0 4/7] COPY pyproject.toml poetry.lock ./ 0.0s
=> CACHED [celery-scraper-worker0 5/7] RUN poetry export --without-hashes --format=requirements.txt > requirements.txt 0.0s
=> CACHED [scraper-ingester 6/7] RUN pip install --no-cache-dir -r requirements.txt 0.0s
=> CACHED [job-scheduler 7/7] COPY . . 0.0s
=> [celery-scraper-worker0] exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:00ebf69c4137f08dc21f59fbf46002b3b9efbbe4bf37aa054952f65225e979af 0.0s
=> => naming to docker.io/library/feed-span-celery-scraper-worker0 0.0s
=> [scraper-ingester] exporting to image 0.1s
=> => exporting layers 0.0s
=> => writing image sha256:827f9e6c59d9f2265bb961ceb22bfe0382f8ef6a53221ea9ba3e3cee65a59663 0.0s
=> => naming to docker.io/library/feed-span-scraper-ingester 0.0s
=> [job-scheduler] exporting to image 0.1s
=> => exporting layers 0.0s
=> => writing image sha256:a5db27ff137dec6dc4fcce64ad4a6044cd34db126b63563637510ae28638db50 0.0s
=> => naming to docker.io/library/feed-span-job-scheduler 0.0s
=> CACHED [celery-worker 4/8] COPY pyproject.toml poetry.lock ./ 0.0s
=> CACHED [celery-worker 5/8] RUN poetry export --without-hashes --format=requirements.txt > requirements.txt 0.0s
=> CACHED [celery-worker 6/8] RUN pip install --no-cache-dir -r requirements.txt 0.0s
=> CACHED [celery-worker 7/8] RUN python -m nltk.downloader maxent_ne_chunker words punkt averaged_perceptron_tagger 0.0s
=> CACHED [celery-worker 8/8] COPY . . 0.0s
=> CACHED [celery-scorer-worker0 4/8] COPY pyproject.toml poetry.lock ./ 0.0s
=> CACHED [celery-scorer-worker0 5/8] RUN poetry export --without-hashes --format=requirements.txt > requirements.txt 0.0s
=> CACHED [celery-scorer-worker0 6/8] RUN pip install --no-cache-dir -r requirements.txt 0.0s
=> CACHED [celery-scorer-worker0 7/8] RUN python -m nltk.downloader vader_lexicon movie_reviews punkt 0.0s
=> CACHED [celery-scorer-worker0 8/8] COPY . . 0.0s
=> [celery-worker] exporting to image 0.1s
=> => exporting layers 0.0s
=> => writing image sha256:75ed33b62039f86df4b60172899e7e656f60be85b62f81b6207ac09fd3c5dd1c 0.0s
=> => naming to docker.io/library/feed-span-celery-worker 0.0s
=> [ranker] exporting to image 0.1s
=> => exporting layers 0.0s
=> => writing image sha256:5f3acdb26c14d3af0a7912a6dd2c29ffb4428d472a29a4acaee1607e9d3a9d88 0.0s
=> => naming to docker.io/library/feed-span-ranker 0.0s
=> [celery-scorer-worker0] exporting to image 0.2s
=> => exporting layers 0.0s
=> => writing image sha256:42b917ddaaaeb1c5cf5658493c1e6dc9fc2306f8c065ea6eef9fd85aa45ecf5d 0.0s
=> => naming to docker.io/library/feed-span-celery-scorer-worker0 0.0s
=> CACHED [celery-scorer-worker1 4/8] COPY pyproject.toml poetry.lock ./ 0.0s
=> CACHED [celery-scorer-worker1 5/8] RUN poetry export --without-hashes --format=requirements.txt > requirements.txt 0.0s
=> CACHED [celery-scorer-worker1 6/8] RUN pip install --no-cache-dir -r requirements.txt 0.0s
=> CACHED [celery-scorer-worker1 7/8] RUN python -m nltk.downloader vader_lexicon movie_reviews punkt 0.0s
=> CACHED [celery-scorer-worker1 8/8] COPY . . 0.0s
=> [celery-scorer-worker1] exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:1f99beb291c80dd024f8816b05c3c1806b73e79f7e9a1cad5e00c10fd79e390b 0.0s
=> => naming to docker.io/library/feed-span-celery-scorer-worker1 0.0s
[+] Running 10/0
✔ Container feed-span-redis-1 Created 0.0s
✔ Container feed-span-redis-celery-broker-1 Created 0.0s
✔ Container feed-span-scraper-ingester-1 Created 0.0s
✔ Container feed-span-database-1 Created 0.0s
✔ Container feed-span-ranker-1 Created 0.0s
✔ Container feed-span-celery-scraper-worker0-1 Created 0.0s
✔ Container feed-span-job-scheduler-1 Created 0.0s
✔ Container feed-span-celery-scorer-worker0-1 Created 0.0s
✔ Container feed-span-celery-scorer-worker1-1 Created 0.0s
✔ Container feed-span-celery-worker-1 Created 0.0s
Attaching to celery-scorer-worker0-1, celery-scorer-worker1-1, celery-scraper-worker0-1, celery-worker-1, database-1, job-scheduler-1, ranker-1, redis-1, redis-celery-broker-1, scraper-ingester-1
redis-1 | 1:C 30 May 2024 23:15:13.418 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
redis-1 | 1:C 30 May 2024 23:15:13.418 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
redis-1 | 1:C 30 May 2024 23:15:13.418 * Redis version=7.2.5, bits=64, commit=00000000, modified=0, pid=1, just started
redis-1 | 1:C 30 May 2024 23:15:13.419 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
redis-1 | 1:M 30 May 2024 23:15:13.420 * monotonic clock: POSIX clock_gettime
redis-celery-broker-1 | 1:C 30 May 2024 23:15:13.420 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
redis-celery-broker-1 | 1:C 30 May 2024 23:15:13.422 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
redis-celery-broker-1 | 1:C 30 May 2024 23:15:13.422 * Redis version=7.2.5, bits=64, commit=00000000, modified=0, pid=1, just started
redis-celery-broker-1 | 1:C 30 May 2024 23:15:13.422 * Configuration loaded
redis-1 | 1:M 30 May 2024 23:15:13.423 * Running mode=standalone, port=6379.
redis-1 | 1:M 30 May 2024 23:15:13.424 * Server initialized
redis-celery-broker-1 | 1:M 30 May 2024 23:15:13.424 * monotonic clock: POSIX clock_gettime
redis-celery-broker-1 | 1:M 30 May 2024 23:15:13.429 * Running mode=standalone, port=6380.
redis-1 | 1:M 30 May 2024 23:15:13.429 * Loading RDB produced by version 7.2.5
redis-1 | 1:M 30 May 2024 23:15:13.429 * RDB age 45172 seconds
redis-1 | 1:M 30 May 2024 23:15:13.429 * RDB memory usage when created 0.83 Mb
redis-1 | 1:M 30 May 2024 23:15:13.429 * Done loading RDB, keys loaded: 0, keys expired: 0.
redis-1 | 1:M 30 May 2024 23:15:13.429 * DB loaded from disk: 0.005 seconds
redis-1 | 1:M 30 May 2024 23:15:13.429 * Ready to accept connections tcp
redis-celery-broker-1 | 1:M 30 May 2024 23:15:13.431 * Server initialized
redis-celery-broker-1 | 1:M 30 May 2024 23:15:13.431 * Loading RDB produced by version 7.2.5
redis-celery-broker-1 | 1:M 30 May 2024 23:15:13.431 * RDB age 45172 seconds
redis-celery-broker-1 | 1:M 30 May 2024 23:15:13.431 * RDB memory usage when created 0.83 Mb
redis-celery-broker-1 | 1:M 30 May 2024 23:15:13.431 * Done loading RDB, keys loaded: 0, keys expired: 0.
redis-celery-broker-1 | 1:M 30 May 2024 23:15:13.431 * DB loaded from disk: 0.001 seconds
redis-celery-broker-1 | 1:M 30 May 2024 23:15:13.431 * Ready to accept connections tcp
database-1 |
database-1 | PostgreSQL Database directory appears to contain a database; Skipping initialization
database-1 |
database-1 | 2024-05-30 23:15:13.694 UTC [1] LOG: starting PostgreSQL 15.7 on x86_64-pc-linux-musl, compiled by gcc (Alpine 13.2.1_git20240309) 13.2.1 20240309, 64-bit
database-1 | 2024-05-30 23:15:13.694 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
database-1 | 2024-05-30 23:15:13.694 UTC [1] LOG: listening on IPv6 address "::", port 5432
database-1 | 2024-05-30 23:15:13.700 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
database-1 | 2024-05-30 23:15:13.715 UTC [24] LOG: database system was shut down at 2024-05-30 10:42:21 UTC
database-1 | 2024-05-30 23:15:13.753 UTC [1] LOG: database system is ready to accept connections
scraper-ingester-1 | Traceback (most recent call last):
scraper-ingester-1 | File "/usr/local/bin/uvicorn", line 8, in <module>
scraper-ingester-1 | sys.exit(main())
scraper-ingester-1 | ^^^^^^
scraper-ingester-1 | File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
scraper-ingester-1 | return self.main(*args, **kwargs)
scraper-ingester-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^
scraper-ingester-1 | File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1078, in main
scraper-ingester-1 | rv = self.invoke(ctx)
scraper-ingester-1 | ^^^^^^^^^^^^^^^^
scraper-ingester-1 | File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
scraper-ingester-1 | return ctx.invoke(self.callback, **ctx.params)
scraper-ingester-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
scraper-ingester-1 | File "/usr/local/lib/python3.11/site-packages/click/core.py", line 783, in invoke
scraper-ingester-1 | return __callback(*args, **kwargs)
scraper-ingester-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^
scraper-ingester-1 | File "/usr/local/lib/python3.11/site-packages/uvicorn/main.py", line 409, in main
scraper-ingester-1 | run(
scraper-ingester-1 | File "/usr/local/lib/python3.11/site-packages/uvicorn/main.py", line 575, in run
scraper-ingester-1 | server.run()
scraper-ingester-1 | File "/usr/local/lib/python3.11/site-packages/uvicorn/server.py", line 65, in run
scraper-ingester-1 | return asyncio.run(self.serve(sockets=sockets))
scraper-ingester-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
scraper-ingester-1 | File "/usr/local/lib/python3.11/asyncio/runners.py", line 190, in run
scraper-ingester-1 | return runner.run(main)
scraper-ingester-1 | ^^^^^^^^^^^^^^^^
scraper-ingester-1 | File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run
scraper-ingester-1 | return self._loop.run_until_complete(task)
scraper-ingester-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
scraper-ingester-1 | File "uvloop/loop.pyx", line 1517, in uvloop.loop.Loop.run_until_complete
scraper-ingester-1 | File "/usr/local/lib/python3.11/site-packages/uvicorn/server.py", line 69, in serve
scraper-ingester-1 | await self._serve(sockets)
scraper-ingester-1 | File "/usr/local/lib/python3.11/site-packages/uvicorn/server.py", line 76, in _serve
scraper-ingester-1 | config.load()
scraper-ingester-1 | File "/usr/local/lib/python3.11/site-packages/uvicorn/config.py", line 433, in load
scraper-ingester-1 | self.loaded_app = import_from_string(self.app)
scraper-ingester-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
scraper-ingester-1 | File "/usr/local/lib/python3.11/site-packages/uvicorn/importer.py", line 22, in import_from_string
scraper-ingester-1 | raise exc from None
scraper-ingester-1 | File "/usr/local/lib/python3.11/site-packages/uvicorn/importer.py", line 19, in import_from_string
scraper-ingester-1 | module = importlib.import_module(module_str)
scraper-ingester-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
scraper-ingester-1 | File "/usr/local/lib/python3.11/importlib/__init__.py", line 126, in import_module
scraper-ingester-1 | return _bootstrap._gcd_import(name[level:], package, level)
scraper-ingester-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
scraper-ingester-1 | File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
scraper-ingester-1 | File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
scraper-ingester-1 | File "<frozen importlib._bootstrap>", line 1126, in _find_and_load_unlocked
scraper-ingester-1 | File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
scraper-ingester-1 | File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
scraper-ingester-1 | File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
scraper-ingester-1 | File "<frozen importlib._bootstrap>", line 1140, in _find_and_load_unlocked
scraper-ingester-1 | ModuleNotFoundError: No module named 'scraper_worker'
ranker-1 | Usage: uvicorn [OPTIONS] APP
ranker-1 | Try 'uvicorn --help' for help.
ranker-1 |
ranker-1 | Error: Invalid value for '--log-config': Path 'ranking_server/logging_config.yaml' does not exist.
ranker-1 exited with code 2
scraper-ingester-1 exited with code 1
job-scheduler-1 | Usage: celery [OPTIONS] COMMAND [ARGS]...
job-scheduler-1 | Try 'celery --help' for help.
job-scheduler-1 |
job-scheduler-1 | Error: Invalid value for '-A' / '--app':
job-scheduler-1 | Unable to load celery application.
job-scheduler-1 | While trying to load the module job_scheduler.scheduler the following error occurred:
job-scheduler-1 | Traceback (most recent call last):
job-scheduler-1 | File "/usr/local/lib/python3.11/site-packages/celery/bin/celery.py", line 58, in convert
job-scheduler-1 | return find_app(value)
job-scheduler-1 | ^^^^^^^^^^^^^^^
job-scheduler-1 | File "/usr/local/lib/python3.11/site-packages/celery/app/utils.py", line 383, in find_app
job-scheduler-1 | sym = symbol_by_name(app, imp=imp)
job-scheduler-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
job-scheduler-1 | File "/usr/local/lib/python3.11/site-packages/kombu/utils/imports.py", line 59, in symbol_by_name
job-scheduler-1 | module = imp(module_name, package=package, **kwargs)
job-scheduler-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
job-scheduler-1 | File "/usr/local/lib/python3.11/site-packages/celery/utils/imports.py", line 109, in import_from_cwd
job-scheduler-1 | return imp(module, package=package)
job-scheduler-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
job-scheduler-1 | File "/usr/local/lib/python3.11/importlib/__init__.py", line 126, in import_module
job-scheduler-1 | return _bootstrap._gcd_import(name[level:], package, level)
job-scheduler-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
job-scheduler-1 | File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
job-scheduler-1 | File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
job-scheduler-1 | File "<frozen importlib._bootstrap>", line 1140, in _find_and_load_unlocked
job-scheduler-1 | ModuleNotFoundError: No module named 'job_scheduler'
job-scheduler-1 |
celery-scraper-worker0-1 | Usage: celery [OPTIONS] COMMAND [ARGS]...
celery-scraper-worker0-1 | Try 'celery --help' for help.
celery-scraper-worker0-1 |
celery-scorer-worker0-1 | Usage: celery [OPTIONS] COMMAND [ARGS]...
celery-scraper-worker0-1 | Error: Invalid value for '-A' / '--app':
celery-scorer-worker0-1 | Try 'celery --help' for help.
celery-scraper-worker0-1 | Unable to load celery application.
celery-scraper-worker0-1 | While trying to load the module scraper_worker.tasks the following error occurred:
celery-scraper-worker0-1 | Traceback (most recent call last):
celery-scraper-worker0-1 | File "/usr/local/lib/python3.11/site-packages/celery/bin/celery.py", line 58, in convert
celery-scraper-worker0-1 | return find_app(value)
celery-scorer-worker0-1 |
celery-scorer-worker0-1 | Error: Invalid value for '-A' / '--app':
celery-scorer-worker0-1 | Unable to load celery application.
celery-scorer-worker0-1 | While trying to load the module scorer_worker.tasks the following error occurred:
celery-scorer-worker0-1 | Traceback (most recent call last):
celery-scorer-worker0-1 | File "/usr/local/lib/python3.11/site-packages/celery/bin/celery.py", line 58, in convert
celery-scorer-worker0-1 | return find_app(value)
celery-scraper-worker0-1 | ^^^^^^^^^^^^^^^
celery-scorer-worker0-1 | ^^^^^^^^^^^^^^^
celery-scraper-worker0-1 | File "/usr/local/lib/python3.11/site-packages/celery/app/utils.py", line 383, in find_app
celery-scorer-worker0-1 | File "/usr/local/lib/python3.11/site-packages/celery/app/utils.py", line 383, in find_app
celery-scorer-worker0-1 | sym = symbol_by_name(app, imp=imp)
celery-scorer-worker0-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
celery-scorer-worker0-1 | File "/usr/local/lib/python3.11/site-packages/kombu/utils/imports.py", line 59, in symbol_by_name
celery-scorer-worker0-1 | module = imp(module_name, package=package, **kwargs)
celery-scorer-worker0-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
celery-scorer-worker0-1 | File "/usr/local/lib/python3.11/site-packages/celery/utils/imports.py", line 109, in import_from_cwd
celery-scraper-worker0-1 | sym = symbol_by_name(app, imp=imp)
celery-scorer-worker0-1 | return imp(module, package=package)
celery-scraper-worker0-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
celery-scraper-worker0-1 | File "/usr/local/lib/python3.11/site-packages/kombu/utils/imports.py", line 59, in symbol_by_name
celery-scraper-worker0-1 | module = imp(module_name, package=package, **kwargs)
celery-scraper-worker0-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
celery-scraper-worker0-1 | File "/usr/local/lib/python3.11/site-packages/celery/utils/imports.py", line 109, in import_from_cwd
celery-scraper-worker0-1 | return imp(module, package=package)
celery-scraper-worker0-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
celery-scraper-worker0-1 | File "/usr/local/lib/python3.11/importlib/__init__.py", line 126, in import_module
celery-scraper-worker0-1 | return _bootstrap._gcd_import(name[level:], package, level)
celery-scraper-worker0-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
celery-scraper-worker0-1 | File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
celery-scraper-worker0-1 | File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
celery-scraper-worker0-1 | File "<frozen importlib._bootstrap>", line 1140, in _find_and_load_unlocked
celery-scorer-worker0-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
celery-scraper-worker0-1 | ModuleNotFoundError: No module named 'scraper_worker'
celery-scraper-worker0-1 |
celery-scorer-worker0-1 | File "/usr/local/lib/python3.11/importlib/__init__.py", line 126, in import_module
celery-scorer-worker0-1 | return _bootstrap._gcd_import(name[level:], package, level)
celery-scorer-worker0-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
celery-scorer-worker0-1 | File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
celery-scorer-worker0-1 | File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
celery-scorer-worker0-1 | File "<frozen importlib._bootstrap>", line 1140, in _find_and_load_unlocked
celery-scorer-worker0-1 | ModuleNotFoundError: No module named 'scorer_worker'
celery-scorer-worker0-1 |
celery-worker-1 | Usage: celery [OPTIONS] COMMAND [ARGS]...
celery-worker-1 | Try 'celery --help' for help.
celery-worker-1 |
celery-worker-1 | Error: Invalid value for '-A' / '--app':
celery-worker-1 | Unable to load celery application.
celery-worker-1 | While trying to load the module sandbox_worker.tasks the following error occurred:
celery-worker-1 | Traceback (most recent call last):
celery-worker-1 | File "/usr/local/lib/python3.11/site-packages/celery/bin/celery.py", line 58, in convert
celery-worker-1 | return find_app(value)
celery-worker-1 | ^^^^^^^^^^^^^^^
celery-worker-1 | File "/usr/local/lib/python3.11/site-packages/celery/app/utils.py", line 383, in find_app
celery-worker-1 | sym = symbol_by_name(app, imp=imp)
celery-worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
celery-worker-1 | File "/usr/local/lib/python3.11/site-packages/kombu/utils/imports.py", line 59, in symbol_by_name
celery-worker-1 | module = imp(module_name, package=package, **kwargs)
celery-worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
celery-worker-1 | File "/usr/local/lib/python3.11/site-packages/celery/utils/imports.py", line 109, in import_from_cwd
celery-worker-1 | return imp(module, package=package)
celery-worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
celery-worker-1 | File "/usr/local/lib/python3.11/importlib/__init__.py", line 126, in import_module
celery-worker-1 | return _bootstrap._gcd_import(name[level:], package, level)
celery-worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
celery-worker-1 | File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
celery-worker-1 | File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
celery-worker-1 | File "<frozen importlib._bootstrap>", line 1140, in _find_and_load_unlocked
celery-worker-1 | ModuleNotFoundError: No module named 'sandbox_worker'
celery-worker-1 |
celery-scorer-worker1-1 | Usage: celery [OPTIONS] COMMAND [ARGS]...
celery-scorer-worker1-1 | Try 'celery --help' for help.
celery-scorer-worker1-1 |
celery-scorer-worker1-1 | Error: Invalid value for '-A' / '--app':
celery-scorer-worker1-1 | Unable to load celery application.
celery-scorer-worker1-1 | While trying to load the module scorer_worker.tasks the following error occurred:
celery-scorer-worker1-1 | Traceback (most recent call last):
celery-scorer-worker1-1 | File "/usr/local/lib/python3.11/site-packages/celery/bin/celery.py", line 58, in convert
celery-scorer-worker1-1 | return find_app(value)
celery-scorer-worker1-1 | ^^^^^^^^^^^^^^^
celery-scorer-worker1-1 | File "/usr/local/lib/python3.11/site-packages/celery/app/utils.py", line 383, in find_app
celery-scorer-worker1-1 | sym = symbol_by_name(app, imp=imp)
celery-scorer-worker1-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
celery-scorer-worker1-1 | File "/usr/local/lib/python3.11/site-packages/kombu/utils/imports.py", line 59, in symbol_by_name
celery-scorer-worker1-1 | module = imp(module_name, package=package, **kwargs)
celery-scorer-worker1-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
celery-scorer-worker1-1 | File "/usr/local/lib/python3.11/site-packages/celery/utils/imports.py", line 109, in import_from_cwd
celery-scorer-worker1-1 | return imp(module, package=package)
celery-scorer-worker1-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
celery-scorer-worker1-1 | File "/usr/local/lib/python3.11/importlib/__init__.py", line 126, in import_module
celery-scorer-worker1-1 | return _bootstrap._gcd_import(name[level:], package, level)
celery-scorer-worker1-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
celery-scorer-worker1-1 | File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
celery-scorer-worker1-1 | File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
celery-scorer-worker1-1 | File "<frozen importlib._bootstrap>", line 1140, in _find_and_load_unlocked
celery-scorer-worker1-1 | ModuleNotFoundError: No module named 'scorer_worker'
celery-scorer-worker1-1 |
job-scheduler-1 exited with code 2
celery-worker-1 exited with code 2
celery-scraper-worker0-1 exited with code 2
celery-scorer-worker1-1 exited with code 2
celery-scorer-worker0-1 exited with code 2
^CGracefully stopping... (press Ctrl+C again to force)
[+] Stopping 10/10
✔ Container feed-span-ranker-1 Stopped 0.0s
✔ Container feed-span-job-scheduler-1 Stopped 0.0s
✔ Container feed-span-celery-worker-1 Stopped 0.0s
✔ Container feed-span-celery-scraper-worker0-1 Stopped 0.0s
✔ Container feed-span-scraper-ingester-1 Stopped 0.0s
✔ Container feed-span-celery-scorer-worker0-1 Stopped 0.0s
✔ Container feed-span-celery-scorer-worker1-1 Stopped 0.0s
✔ Container feed-span-database-1 Stopped 0.4s
✔ Container feed-span-redis-1 Stopped 0.5s
✔ Container feed-span-redis-celery-broker-1 Stopped 0.6s
canceled
make: *** [Makefile:17: run] Error 130
When I run
docker compose up --build, either from the root directory or fromexamples/combined, I get a lot ofModuleNotFoundErrors which seem to be triggered by commands listed indocker-compose.yml. E.g., the commands... etc can't find
ranking_server,sandbox_workerorscorer_worker.Any pointers? I've tried:
examples/combinedto the root directory which seems to just create a similar-but-different set of directory-structure-related errors.docker-compose.ymlto point to subdirectories, e.g., changingsandbox_worker.taskstoexamples.combined.sandbox_worker.tasks, but then get an error saying thatsandbox_workerdoes not have an attributetasks.Thanks for your help!