Weves
commented
6 months ago Hey @tigermask1978, could you try pulling up the logs from the api_server container? I suspect something is going wrong there
Open tigermask1978 opened 6 months ago
Weves
commented
6 months ago Hey @tigermask1978, could you try pulling up the logs from the api_server container? I suspect something is going wrong there
tigermask1978
commented
6 months ago Hi @Weves, Yes,the problem is the api_server cannot download the Embedding model "thenlper/gte-small" from the huggingface because of my network envrionment. So how to configure Danswer to use the local model? Thanks.
redcathode
commented
6 months ago I've been getting the same issue: the danswer-stack-api_server-1 container keeps restarting. Figure I can just paste the logs here:
danswer-stack-api_server-1 | INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
danswer-stack-api_server-1 | INFO [alembic.runtime.migration] Will assume transactional DDL.
danswer-stack-api_server-1 | 2024-01-05 23:01:21.030285: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:9342] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
danswer-stack-api_server-1 | 2024-01-05 23:01:21.030339: E tensorflow/compiler/xla/stream_executor/cuda/cuda_fft.cc:609] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
danswer-stack-api_server-1 | 2024-01-05 23:01:21.030366: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:1518] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
danswer-stack-api_server-1 | 2024-01-05 23:01:22.101697: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
danswer-stack-api_server-1 | /usr/local/lib/python3.11/site-packages/transformers/utils/hub.py:128: FutureWarning: Using `DISABLE_TELEMETRY` is deprecated and will be removed in v5 of Transformers. Use `HF_HUB_DISABLE_TELEMETRY` instead.
danswer-stack-api_server-1 | warnings.warn(
danswer-stack-api_server-1 | INFO: Started server process [10]
danswer-stack-api_server-1 | INFO: Waiting for application startup.
danswer-stack-api_server-1 | 01/05/2024 11:01:25 PM users.py 68 : Using Auth Type: disabled
danswer-stack-api_server-1 | 01/05/2024 11:01:25 PM main.py 217 : Danswer API Key: dn_fBgW2_BUa2cqOBm9JWcwkPaP41Sg5hmh-bMz-djM
danswer-stack-api_server-1 | 01/05/2024 11:01:25 PM main.py 225 : Using LLM Provider: openai
danswer-stack-api_server-1 | 01/05/2024 11:01:25 PM main.py 226 : Using LLM Model Version: gpt-3.5-turbo
danswer-stack-api_server-1 | 01/05/2024 11:01:25 PM chat_llm.py 48 : LLM Model Class: ChatLiteLLM, Model Config: {'model': 'openai/gpt-3.5-turbo', 'request_timeout': 60.0, 'model_kwargs': {'frequency_penalty': 0, 'presence_penalty': 0}, 'n': 1, 'max_tokens': 1024}
danswer-stack-api_server-1 | 01/05/2024 11:01:25 PM main.py 245 : Using Embedding model: "thenlper/gte-small"
danswer-stack-api_server-1 | 01/05/2024 11:01:25 PM main.py 255 : Warming up local NLP models.
danswer-stack-api_server-1 | 01/05/2024 11:01:25 PM search_nlp_models.py 58 : Loading thenlper/gte-small
danswer-stack-api_server-1 | 2024-01-05 23:01:26.218976: W tensorflow/tsl/framework/cpu_allocator_impl.cc:83] Allocation of 93763584 exceeds 10% of free system memory.
danswer-stack-api_server-1 | 2024-01-05 23:01:26.364838: W tensorflow/tsl/framework/cpu_allocator_impl.cc:83] Allocation of 93763584 exceeds 10% of free system memory.
danswer-stack-api_server-1 | 2024-01-05 23:01:26.402567: W tensorflow/tsl/framework/cpu_allocator_impl.cc:83] Allocation of 93763584 exceeds 10% of free system memory.
danswer-stack-api_server-1 | 2024-01-05 23:01:27.276982: W tensorflow/tsl/framework/cpu_allocator_impl.cc:83] Allocation of 93763584 exceeds 10% of free system memory.
danswer-stack-api_server-1 | 2024-01-05 23:01:27.315012: W tensorflow/tsl/framework/cpu_allocator_impl.cc:83] Allocation of 93763584 exceeds 10% of free system memory.
danswer-stack-api_server-1 | 01/05/2024 11:01:27 PM main.py 261 : GPU is not available
danswer-stack-api_server-1 | 01/05/2024 11:01:27 PM main.py 262 : Torch Threads: 4
danswer-stack-api_server-1 | 01/05/2024 11:01:27 PM main.py 264 : Verifying query preprocessing (NLTK) data is downloaded
danswer-stack-api_server-1 | 01/05/2024 11:01:27 PM main.py 269 : Verifying default connector/credential exist.
danswer-stack-api_server-1 | 01/05/2024 11:01:27 PM main.py 275 : Loading default Prompts and Personas
danswer-stack-api_server-1 | 01/05/2024 11:01:27 PM main.py 278 : Verifying Document Index(s) is/are available.
danswer-stack-api_server-1 | ERROR: Traceback (most recent call last):
danswer-stack-api_server-1 | File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 174, in _new_conn
danswer-stack-api_server-1 | conn = connection.create_connection(
danswer-stack-api_server-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
danswer-stack-api_server-1 | File "/usr/local/lib/python3.11/site-packages/urllib3/util/connection.py", line 72, in create_connection
danswer-stack-api_server-1 | for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
danswer-stack-api_server-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
danswer-stack-api_server-1 | File "/usr/local/lib/python3.11/socket.py", line 962, in getaddrinfo
danswer-stack-api_server-1 | for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
danswer-stack-api_server-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
danswer-stack-api_server-1 | socket.gaierror: [Errno -2] Name or service not known
danswer-stack-api_server-1 |
danswer-stack-api_server-1 | During handling of the above exception, another exception occurred:
danswer-stack-api_server-1 |
danswer-stack-api_server-1 | Traceback (most recent call last):
danswer-stack-api_server-1 | File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 715, in urlopen
danswer-stack-api_server-1 | httplib_response = self._make_request(
danswer-stack-api_server-1 | ^^^^^^^^^^^^^^^^^^^
danswer-stack-api_server-1 | File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 416, in _make_request
danswer-stack-api_server-1 | conn.request(method, url, **httplib_request_kw)
danswer-stack-api_server-1 | File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 244, in request
danswer-stack-api_server-1 | super(HTTPConnection, self).request(method, url, body=body, headers=headers)
danswer-stack-api_server-1 | File "/usr/local/lib/python3.11/http/client.py", line 1294, in request
danswer-stack-api_server-1 | self._send_request(method, url, body, headers, encode_chunked)
danswer-stack-api_server-1 | File "/usr/local/lib/python3.11/http/client.py", line 1340, in _send_request
danswer-stack-api_server-1 | self.endheaders(body, encode_chunked=encode_chunked)
danswer-stack-api_server-1 | File "/usr/local/lib/python3.11/http/client.py", line 1289, in endheaders
danswer-stack-api_server-1 | self._send_output(message_body, encode_chunked=encode_chunked)
danswer-stack-api_server-1 | File "/usr/local/lib/python3.11/http/client.py", line 1048, in _send_output
danswer-stack-api_server-1 | self.send(msg)
danswer-stack-api_server-1 | File "/usr/local/lib/python3.11/http/client.py", line 986, in send
danswer-stack-api_server-1 | self.connect()
danswer-stack-api_server-1 | File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 205, in connect
danswer-stack-api_server-1 | conn = self._new_conn()
danswer-stack-api_server-1 | ^^^^^^^^^^^^^^^^
danswer-stack-api_server-1 | File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 186, in _new_conn
danswer-stack-api_server-1 | raise NewConnectionError(
danswer-stack-api_server-1 | urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f77ebb89290>: Failed to establish a new connection: [Errno -2] Name or service not known
danswer-stack-api_server-1 |
danswer-stack-api_server-1 | During handling of the above exception, another exception occurred:
danswer-stack-api_server-1 |
danswer-stack-api_server-1 | Traceback (most recent call last):
danswer-stack-api_server-1 | File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 486, in send
danswer-stack-api_server-1 | resp = conn.urlopen(
danswer-stack-api_server-1 | ^^^^^^^^^^^^^
danswer-stack-api_server-1 | File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 799, in urlopen
danswer-stack-api_server-1 | retries = retries.increment(
danswer-stack-api_server-1 | ^^^^^^^^^^^^^^^^^^
danswer-stack-api_server-1 | File "/usr/local/lib/python3.11/site-packages/urllib3/util/retry.py", line 592, in increment
danswer-stack-api_server-1 | raise MaxRetryError(_pool, url, error or ResponseError(cause))
danswer-stack-api_server-1 | urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='index', port=19071): Max retries exceeded with url: /application/v2/tenant/default/prepareandactivate (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f77ebb89290>: Failed to establish a new connection: [Errno -2] Name or service
not known'))
danswer-stack-api_server-1 |
danswer-stack-api_server-1 | During handling of the above exception, another exception occurred:
danswer-stack-api_server-1 |
danswer-stack-api_server-1 | Traceback (most recent call last):
danswer-stack-api_server-1 | File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 677, in lifespan
danswer-stack-api_server-1 | async with self.lifespan_context(app) as maybe_state:
danswer-stack-api_server-1 | File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 566, in __aenter__
danswer-stack-api_server-1 | await self._router.startup()
danswer-stack-api_server-1 | File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 656, in startup
danswer-stack-api_server-1 | handler()
danswer-stack-api_server-1 | File "/app/danswer/main.py", line 279, in startup_event
danswer-stack-api_server-1 | get_default_document_index().ensure_indices_exist()
danswer-stack-api_server-1 | File "/app/danswer/document_index/vespa/index.py", line 606, in ensure_indices_exist
danswer-stack-api_server-1 | response = requests.post(deploy_url, headers=headers, data=f)
danswer-stack-api_server-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
danswer-stack-api_server-1 | File "/usr/local/lib/python3.11/site-packages/requests/api.py", line 115, in post
danswer-stack-api_server-1 | return request("post", url, data=data, json=json, **kwargs)
danswer-stack-api_server-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
danswer-stack-api_server-1 | File "/usr/local/lib/python3.11/site-packages/requests/api.py", line 59, in request
danswer-stack-api_server-1 | return session.request(method=method, url=url, **kwargs)
danswer-stack-api_server-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
danswer-stack-api_server-1 | File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
danswer-stack-api_server-1 | resp = self.send(prep, **send_kwargs)
danswer-stack-api_server-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
danswer-stack-api_server-1 | File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
danswer-stack-api_server-1 | r = adapter.send(request, **kwargs)
danswer-stack-api_server-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
danswer-stack-api_server-1 | File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 519, in send
danswer-stack-api_server-1 | raise ConnectionError(e, request=request)
danswer-stack-api_server-1 | requests.exceptions.ConnectionError: HTTPConnectionPool(host='index', port=19071): Max retries exceeded with url: /application/v2/tenant/default/prepareandactivate (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f77ebb89290>: Failed to establish a new connection: [Errno -2] Name or serv
ice not known'))
danswer-stack-api_server-1 |
danswer-stack-api_server-1 | ERROR: Application startup failed. Exiting. 
yuhongsun96
commented
6 months ago @tigermask1978, is there a reason you don't want to allow the container to download models? There are a couple models required to run Danswer. If you really want, you can manually copy over those models into the persistent volume:
This is however more complicated than necessary and I would avoid doing this unless absolutely necessary
yuhongsun96
commented
6 months ago @b0vik your issue is due to the document index container not being accessible. It's called index-1 if you're using docker compose. Can you check if that one is failing to start up (you can check the "Last Started" time to see if it has been restarting)
redcathode
commented
6 months ago @yuhongsun96 Yeah, didn't notice that. Here are the logs for the index container:
maximus@node05:~$ docker logs danswer-stack-index-1
/usr/local/bin/start-container.sh: line 67: 11 Illegal instruction /opt/vespa/bin/vespa-start-configserver
/usr/local/bin/start-container.sh: line 67: 10 Illegal instruction /opt/vespa/bin/vespa-start-configserver
/usr/local/bin/start-container.sh: line 67: 10 Illegal instruction /opt/vespa/bin/vespa-start-configserver
/usr/local/bin/start-container.sh: line 67: 10 Illegal instruction /opt/vespa/bin/vespa-start-configserver
/usr/local/bin/start-container.sh: line 67: 9 Illegal instruction /opt/vespa/bin/vespa-start-configserver
/usr/local/bin/start-container.sh: line 67: 10 Illegal instruction /opt/vespa/bin/vespa-start-configserver
/usr/local/bin/start-container.sh: line 67: 9 Illegal instruction /opt/vespa/bin/vespa-start-configserver
/usr/local/bin/start-container.sh: line 67: 10 Illegal instruction /opt/vespa/bin/vespa-start-configserver
/usr/local/bin/start-container.sh: line 67: 10 Illegal instruction /opt/vespa/bin/vespa-start-configserver@b0vik you're probably running on a CPU without AVX2 instruction support. Vespa requires AVX2 by default but they do provide a container that does not require it, see here: https://hub.docker.com/r/vespaengine/vespa-generic-intel-x86_64
If either of you want a faster channel of communications with us we invite you to join our Slack or Discord! Alternatively if you want to have a quick chat with the maintainer (Chris and myself), you can just select a time on my Cal
We'd love to hear more about your use cases and help you get set up for success
Details
All docker servers are up, when I load the web ui, something is wrong as the image shows: