Closed robinwo closed 3 months ago
I'm getting the same...
After the above posted log messages, it eventually times out with this exception:
ERROR: Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 734, in lifespan
async with self.lifespan_context(app) as maybe_state:
File "/usr/local/lib/python3.11/contextlib.py", line 210, in __aenter__
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/app/danswer/main.py", line 245, in lifespan
warm_up_encoders(
File "/app/danswer/search/search_nlp_models.py", line 203, in warm_up_encoders
raise Exception("Failed to run test embedding.")
Exception: Failed to run test embedding.
ERROR: Application startup failed. Exiting.
Hey @dstrzelec / @robinwo! Could you post the output of:
docker logs danswer-stack-inference_model_server-1
Interesting, does not exist:
Error response from daemon: No such container: danswer-stack-inference_model_server-1
Containers running:
[ec2-user@ip-172-31-35-5 ~]$ sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
682f52d784d2 nginx:1.23.4-alpine "/docker-entrypoint.…" About a minute ago Up About a minute 0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp danswer-stack-nginx-1
da3313ef372a danswer/danswer-web-server:latest "docker-entrypoint.s…" About a minute ago Up About a minute danswer-stack-web_server-1
a71dc28f4843 danswer/danswer-backend:latest "/bin/sh -c 'alembic…" About a minute ago Up About a minute danswer-stack-api_server-1
79d4a1811198 vespaengine/vespa:8.277.17 "/usr/local/bin/star…" About a minute ago Up About a minute 0.0.0.0:8081->8081/tcp, :::8081->8081/tcp, 0.0.0.0:19071->19071/tcp, :::19071->19071/tcp danswer-stack-index-1
0b3a2559b023 postgres:15.2-alpine "docker-entrypoint.s…" About a minute ago Up About a minute 5432/tcp danswer-stack-relational_db-1
Could you try pulling in the latest version of Danswer (git pull
), and trying to bring things up again?
Yup I'm on the latest main branch.
After letting Danswer run for a while, I get the following final closing statement:
ERROR: Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 734, in lifespan
async with self.lifespan_context(app) as maybe_state:
File "/usr/local/lib/python3.11/contextlib.py", line 210, in __aenter__
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/app/danswer/main.py", line 245, in lifespan
warm_up_encoders(
File "/app/danswer/search/search_nlp_models.py", line 203, in warm_up_encoders
raise Exception("Failed to run test embedding.")
Exception: Failed to run test embedding.
ERROR: Application startup failed. Exiting.
Brand new EC2 instance, no other configurations done besides following the Danswer getting started guide. SSH into the machine, start copy/past'ing commands.
Solved with this workaround:
docker-compose -f docker-compose.dev.yml -p danswer-stack up -d --build --force-recreate
This will install everything including the inference model server container.Only then, run the letsencrypt to enable HTTPS for your instance. This will rebuild some of the containers, and will keep the inference model server running.
Looks like the QuickStart needs an update, and the letsencrypt script does skip some installation steps.
Thanks for pointing this out @robinwo ! Yea, I think it's an issue with the letsencrypt script—will have a fix out soon.
Fixed in https://github.com/danswer-ai/danswer/pull/1395! Thanks for the heads up all!
I followed the quickstart tutorial step-by-step, tried this on both EC2 and DigitalOcean, but the same issue persists.
While seeing: Nginx is not ready yet, retrying in 5 seconds..., in the docker logs of danswer-stack-api_server-1 I read the following: None of PyTorch, TensorFlow >= 2.0, or Flax have been found.
I manually installed pytorch on the droplet, but that does change nothing (and is not available in the container?). What makes me wonder, am I doing something wrong? On my Mac it runs like a charm locally, but spinning it up in the cloud does not work for me.