Running the docker compose locally does not start the API correctly

blenzi commented 2 years ago

As noted in #231, the README suggests that cloning the package and calling make run should start the API but:

$ curl http://localhost:8080/docs 
Empty reply from server.

The test performed in the CI also succeeds locally:

$ nc -vz localhost 8080
Connection to localhost port 8080 [tcp/http-alt] succeeded!

but unless I missed it, that is no check of the content with curl or similar. The output of make run is below and looks similar to that of the CI (ex.: step 3 of https://github.com/pyronear/pyro-api/actions/runs/3348292566/jobs/5547200913). @frgfm, could it be that you have those variables set locally so you have access to QARNOT ?

docker-compose up -d --build
WARNING: The QARNOT_TOKEN variable is not set. Defaulting to a blank string.
WARNING: The BUCKET_NAME variable is not set. Defaulting to a blank string.
WARNING: The BUCKET_MEDIA_FOLDER variable is not set. Defaulting to a blank string.
WARNING: The S3_ENDPOINT_URL variable is not set. Defaulting to a blank string.
WARNING: The S3_ACCESS_KEY variable is not set. Defaulting to a blank string.
WARNING: The S3_SECRET_KEY variable is not set. Defaulting to a blank string.
WARNING: The S3_REGION variable is not set. Defaulting to a blank string.
WARNING: The SENTRY_DSN variable is not set. Defaulting to a blank string.
WARNING: The SENTRY_SERVER_NAME variable is not set. Defaulting to a blank string.
Creating network "pyro-api_default" with the default driver
Building backend
[+] Building 4.5s (13/13) FINISHED                                                                                                                                          
 => [internal] load build definition from Dockerfile                                                                                                                   0.1s
 => => transferring dockerfile: 633B                                                                                                                                   0.0s
 => [internal] load .dockerignore                                                                                                                                      0.1s
 => => transferring context: 2B                                                                                                                                        0.1s
 => [internal] load metadata for docker.io/tiangolo/uvicorn-gunicorn-fastapi:python3.8-alpine3.10                                                                      4.1s
 => [auth] tiangolo/uvicorn-gunicorn-fastapi:pull token for registry-1.docker.io                                                                                       0.0s
 => [1/7] FROM docker.io/tiangolo/uvicorn-gunicorn-fastapi:python3.8-alpine3.10@sha256:c9102d06868412c85f81646ad541f4a0c889fdceb4296f27f4f197039f1a49eb                0.0s
 => [internal] load build context                                                                                                                                      0.1s
 => => transferring context: 115.72kB                                                                                                                                  0.1s
 => CACHED [2/7] WORKDIR /app                                                                                                                                          0.0s
 => CACHED [3/7] COPY app/requirements.txt /app/requirements.txt                                                                                                       0.0s
 => CACHED [4/7] RUN set -eux     && apk add --no-cache --virtual .build-deps build-base postgresql-dev gcc libffi-dev libressl-dev musl-dev     && pip install -r /a  0.0s
 => CACHED [5/7] COPY app /app/app                                                                                                                                     0.0s
 => CACHED [6/7] COPY alembic.ini /app/alembic.ini                                                                                                                     0.0s
 => CACHED [7/7] COPY alembic /app/alembic                                                                                                                             0.0s
 => exporting to image                                                                                                                                                 0.0s
 => => exporting layers                                                                                                                                                0.0s
 => => writing image sha256:b1c69c24135494447e6dba3f957cae2aaa16e0dee6854149de6798180dbc9fd6                                                                           0.0s
 => => naming to docker.io/library/pyro-api_backend                                                                                                                    0.0s

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
Building proxy
[+] Building 3.5s (8/8) FINISHED                                                                                                                                            
 => [internal] load build definition from Dockerfile                                                                                                                   0.1s
 => => transferring dockerfile: 36B                                                                                                                                    0.1s
 => [internal] load .dockerignore                                                                                                                                      0.1s
 => => transferring context: 2B                                                                                                                                        0.1s
 => [internal] load metadata for docker.io/library/nginx:latest                                                                                                        3.3s
 => [auth] library/nginx:pull token for registry-1.docker.io                                                                                                           0.0s
 => [internal] load build context                                                                                                                                      0.0s
 => => transferring context: 32B                                                                                                                                       0.0s
 => [1/2] FROM docker.io/library/nginx:latest@sha256:943c25b4b66b332184d5ba6bb18234273551593016c0e0ae906bab111548239f                                                  0.0s
 => CACHED [2/2] COPY nginx.conf /etc/nginx/nginx.conf                                                                                                                 0.0s
 => exporting to image                                                                                                                                                 0.0s
 => => exporting layers                                                                                                                                                0.0s
 => => writing image sha256:450d2e9a5fd5ce5dc21221620f1ae60df000ce00cf7a1a0de8c779212a8d5e3d                                                                           0.0s
 => => naming to docker.io/library/pyro-api_proxy                                                                                                                      0.0s

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
Creating pyro-api_db_1 ... done
Creating pyro-api_backend_1 ... done
Creating pyro-api_proxy_1   ... done

frgfm commented 2 years ago

Hey there :)

Thanks for reporting this! So I can see two topics here:

about curl, there is no content if the server didn't start / crashed. So that check should be equivalent to the server start (and we cannot draft a unittest for the exact HTML that will be spit out)
regarding "Running the docker compose locally does not start the API correctly", in your PR description you raise a question about Qarnot. There are two underlying questions:
- first, the PR description doesn't show where the process fails? :thinking: The console snippet you shared doesn't indicate whether the containers stay up afterwards (perhaps add the docker logs so we can investigate)
- second, about Qarnot & configs, as specified here yes there are some variables that need to be set for the service to work :sweat_smile:

Let me know if there are still some dark areas :)

blenzi commented 1 year ago

Thanks for the reply @frgfm .

I agree we cannot check the exact content but checking if the server spits some HTML or nothing should be enough
The part of the README that you mention comes after the Quick Tour, so to me the comment on #231 is valid. And the issue seems to come from the connection to Qarnot indeed (related to #217):

(pyro-api) blenzi@dphpmcc213:pyro-api$ docker compose ps
WARN[0000] The "QARNOT_TOKEN" variable is not set. Defaulting to a blank string. 
WARN[0000] The "BUCKET_NAME" variable is not set. Defaulting to a blank string. 
WARN[0000] The "BUCKET_MEDIA_FOLDER" variable is not set. Defaulting to a blank string. 
WARN[0000] The "S3_ENDPOINT_URL" variable is not set. Defaulting to a blank string. 
WARN[0000] The "S3_ACCESS_KEY" variable is not set. Defaulting to a blank string. 
WARN[0000] The "S3_SECRET_KEY" variable is not set. Defaulting to a blank string. 
WARN[0000] The "S3_REGION" variable is not set. Defaulting to a blank string. 
WARN[0000] The "SENTRY_DSN" variable is not set. Defaulting to a blank string. 
WARN[0000] The "SENTRY_SERVER_NAME" variable is not set. Defaulting to a blank string. 
NAME                 COMMAND                  SERVICE             STATUS              PORTS
pyro-api_backend_1   "uvicorn app.main:ap…"   backend             running             0.0.0.0:8080->8080/tcp
pyro-api_db_1        "docker-entrypoint.s…"   db                  running             0.0.0.0:5432->5432/tcp
pyro-api_proxy_1     "/docker-entrypoint.…"   proxy               running             0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp

(pyro-api) blenzi@dphpmcc213:pyro-api$ docker compose logs --follow
WARN[0000] The "QARNOT_TOKEN" variable is not set. Defaulting to a blank string. 
WARN[0000] The "BUCKET_NAME" variable is not set. Defaulting to a blank string. 
WARN[0000] The "BUCKET_MEDIA_FOLDER" variable is not set. Defaulting to a blank string. 
WARN[0000] The "S3_ENDPOINT_URL" variable is not set. Defaulting to a blank string. 
WARN[0000] The "S3_ACCESS_KEY" variable is not set. Defaulting to a blank string. 
WARN[0000] The "S3_SECRET_KEY" variable is not set. Defaulting to a blank string. 
WARN[0000] The "S3_REGION" variable is not set. Defaulting to a blank string. 
WARN[0000] The "SENTRY_DSN" variable is not set. Defaulting to a blank string. 
WARN[0000] The "SENTRY_SERVER_NAME" variable is not set. Defaulting to a blank string. 
backend_1  | INFO:     Will watch for changes in these directories: ['/app']
backend_1  | INFO:     Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)
backend_1  | INFO:     Started reloader process [1] using WatchGod
backend_1  | Process SpawnProcess-1:
backend_1  | Traceback (most recent call last):
backend_1  |   File "/usr/local/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
backend_1  |     self.run()
backend_1  |   File "/usr/local/lib/python3.8/multiprocessing/process.py", line 108, in run
backend_1  |     self._target(*self._args, **self._kwargs)
backend_1  |   File "/usr/local/lib/python3.8/site-packages/uvicorn/_subprocess.py", line 76, in subprocess_started
backend_1  |     target(sockets=sockets)
backend_1  |   File "/usr/local/lib/python3.8/site-packages/uvicorn/server.py", line 60, in run
backend_1  |     return asyncio.run(self.serve(sockets=sockets))
backend_1  |   File "/usr/local/lib/python3.8/asyncio/runners.py", line 43, in run
backend_1  |     return loop.run_until_complete(main)
backend_1  |   File "uvloop/loop.pyx", line 1501, in uvloop.loop.Loop.run_until_complete
backend_1  |   File "/usr/local/lib/python3.8/site-packages/uvicorn/server.py", line 67, in serve
backend_1  |     config.load()
backend_1  |   File "/usr/local/lib/python3.8/site-packages/uvicorn/config.py", line 479, in load
backend_1  |     self.loaded_app = import_from_string(self.app)
backend_1  |   File "/usr/local/lib/python3.8/site-packages/uvicorn/importer.py", line 21, in import_from_string
backend_1  |     module = importlib.import_module(module_str)
backend_1  |   File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
backend_1  |     return _bootstrap._gcd_import(name[level:], package, level)
backend_1  |   File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
backend_1  |   File "<frozen importlib._bootstrap>", line 991, in _find_and_load
backend_1  |   File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
backend_1  |   File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
backend_1  |   File "<frozen importlib._bootstrap_external>", line 783, in exec_module
backend_1  |   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
backend_1  |   File "./app/main.py", line 15, in <module>
backend_1  |     from app.api.routes import (
backend_1  |   File "./app/api/routes/media.py", line 21, in <module>
backend_1  |     from app.services import bucket_service, resolve_bucket_key
backend_1  |   File "./app/services/__init__.py", line 1, in <module>
backend_1  |     from .services import *
backend_1  |   File "./app/services/services.py", line 11, in <module>
backend_1  |     bucket_service = QarnotBucket()
backend_1  |   File "./app/services/bucket/s3.py", line 28, in __init__
backend_1  |     self._connect_to_storage()
backend_1  |   File "./app/services/bucket/qarnot.py", line 25, in _connect_to_storage
backend_1  |     self._s3 = Connection(client_token=cfg.QARNOT_TOKEN)._s3client
backend_1  |   File "/usr/local/lib/python3.8/site-packages/qarnot/connection.py", line 188, in __init__
backend_1  |     user = self.user_info
backend_1  |   File "/usr/local/lib/python3.8/site-packages/qarnot/connection.py", line 329, in user_info
backend_1  |     resp = self._get(get_url('user'))
backend_1  |   File "/usr/local/lib/python3.8/site-packages/qarnot/_retry.py", line 29, in _with_retry
backend_1  |     raise UnauthorizedException()
backend_1  | qarnot.exceptions.UnauthorizedException
db_1       | 
proxy_1    | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
proxy_1    | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
proxy_1    | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
proxy_1    | 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
proxy_1    | 10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
proxy_1    | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
proxy_1    | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
proxy_1    | /docker-entrypoint.sh: Configuration complete; ready for start up
db_1       | PostgreSQL Database directory appears to contain a database; Skipping initialization
db_1       | 
db_1       | 2022-11-09 22:47:46.049 UTC [1] LOG:  starting PostgreSQL 12.1 on x86_64-pc-linux-musl, compiled by gcc (Alpine 9.2.0) 9.2.0, 64-bit
db_1       | 2022-11-09 22:47:46.049 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
db_1       | 2022-11-09 22:47:46.049 UTC [1] LOG:  listening on IPv6 address "::", port 5432
db_1       | 2022-11-09 22:47:46.051 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
db_1       | 2022-11-09 22:47:46.068 UTC [21] LOG:  database system was shut down at 2022-11-08 22:51:55 UTC
db_1       | 2022-11-09 22:47:46.072 UTC [1] LOG:  database system is ready to accept connections

frgfm commented 1 year ago

Oh if that's about the order of sections in the README, that could be an improvement. The thing is that quick tour should provide a quick overlook, and subtle setup/tricks come later (such as the setup of storage connection).

So are we talking about a documentation problem? or an actual codebase bug?

blenzi commented 1 year ago

If that works for you, it is probably only a documentation problem, although I cannot test it :)

frgfm commented 1 year ago

What do you mean?

My point is that:

if going through the steps in the "Setup" section before running the service solves the issue
then it's only a section ordering problem
if the problem remains, then the issue is of another nature and we need to address it :sweat_smile:

So, what do you mean by "I cannot test it"? With the .env set, you can't run the service? :thinking: (it's also a major issue considering the CI is doing it at every PR, so if a user cannot, there is a misalignment between the CI & the actual behaviour)

blenzi commented 1 year ago

I created an account in Qarnot and now I can run it and see the API doc. The next step would be to know how to authenticate and use the API.

So yes, it is a documentation issue, and it would be nice to guide the developers, either inverting the sections or adding a link to anticipate how the configuration works. Plus explicitly saying what is needed (accounts, ...) and how to run.

frgfm commented 2 months ago

Closing this since this has been solved :)

pyronear / pyro-api

Running the docker compose locally does not start the API correctly #236