search

zylon-ai / private-gpt

Interact with your documents using the power of GPT, 100% privately, no data leaks
https://docs.privategpt.dev
Apache License 2.0
53.07k stars 7.13k forks source link

Trying to run it dockerized and getting "HTTPConnectionPool(host='localhost', port=11434): Max retries exceeded with url" #1872

Closed BenBatsir closed 1 week ago

BenBatsir commented 3 months ago

It works perfectly when running it locally (without docker), but when I tried to import a doc via UI (inside a container), I'm getting the bellow:

Generating embeddings:   0%|          | 0/23 [00:00<?, ?it/s]Traceback (most recent call last):
privategpt-private-gpt-1  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/urllib3/connection.py", line 174, in _new_conn
privategpt-private-gpt-1  |     conn = connection.create_connection(
privategpt-private-gpt-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
privategpt-private-gpt-1  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/urllib3/util/connection.py", line 95, in create_connection
privategpt-private-gpt-1  |     raise err
privategpt-private-gpt-1  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/urllib3/util/connection.py", line 85, in create_connection
privategpt-private-gpt-1  |     sock.connect(sa)
privategpt-private-gpt-1  | ConnectionRefusedError: [Errno 111] Connection refused
privategpt-private-gpt-1  |
privategpt-private-gpt-1  | During handling of the above exception, another exception occurred:
privategpt-private-gpt-1  |
privategpt-private-gpt-1  | Traceback (most recent call last):
privategpt-private-gpt-1  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 715, in urlopen
privategpt-private-gpt-1  |     httplib_response = self._make_request(
privategpt-private-gpt-1  |                        ^^^^^^^^^^^^^^^^^^^
privategpt-private-gpt-1  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 416, in _make_request
privategpt-private-gpt-1  |     conn.request(method, url, **httplib_request_kw)
privategpt-private-gpt-1  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/urllib3/connection.py", line 244, in request
privategpt-private-gpt-1  |     super(HTTPConnection, self).request(method, url, body=body, headers=headers)
privategpt-private-gpt-1  |   File "/usr/local/lib/python3.11/http/client.py", line 1286, in request
privategpt-private-gpt-1  |     self._send_request(method, url, body, headers, encode_chunked)
privategpt-private-gpt-1  |   File "/usr/local/lib/python3.11/http/client.py", line 1332, in _send_request
privategpt-private-gpt-1  |     self.endheaders(body, encode_chunked=encode_chunked)
privategpt-private-gpt-1  |   File "/usr/local/lib/python3.11/http/client.py", line 1281, in endheaders
privategpt-private-gpt-1  |     self._send_output(message_body, encode_chunked=encode_chunked)
privategpt-private-gpt-1  |   File "/usr/local/lib/python3.11/http/client.py", line 1041, in _send_output
privategpt-private-gpt-1  |     self.send(msg)
privategpt-private-gpt-1  |   File "/usr/local/lib/python3.11/http/client.py", line 979, in send
privategpt-private-gpt-1  |     self.connect()
privategpt-private-gpt-1  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/urllib3/connection.py", line 205, in connect
privategpt-private-gpt-1  |     conn = self._new_conn()
privategpt-private-gpt-1  |            ^^^^^^^^^^^^^^^^
privategpt-private-gpt-1  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/urllib3/connection.py", line 186, in _new_conn
privategpt-private-gpt-1  |     raise NewConnectionError(
privategpt-private-gpt-1  | urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0xffff609ddf10>: Failed to establish a new connection: [Errno 111] Connection refused
privategpt-private-gpt-1  |
privategpt-private-gpt-1  | During handling of the above exception, another exception occurred:
privategpt-private-gpt-1  |
privategpt-private-gpt-1  | Traceback (most recent call last):
privategpt-private-gpt-1  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/requests/adapters.py", line 486, in send
privategpt-private-gpt-1  |     resp = conn.urlopen(
privategpt-private-gpt-1  |            ^^^^^^^^^^^^^
privategpt-private-gpt-1  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 799, in urlopen
privategpt-private-gpt-1  |     retries = retries.increment(
privategpt-private-gpt-1  |               ^^^^^^^^^^^^^^^^^^
privategpt-private-gpt-1  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/urllib3/util/retry.py", line 592, in increment
privategpt-private-gpt-1  |     raise MaxRetryError(_pool, url, error or ResponseError(cause))
privategpt-private-gpt-1  | urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=11434): Max retries exceeded with url: /api/embeddings (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xffff609ddf10>: Failed to establish a new connection: [Errno 111] Connection refused'))
privategpt-private-gpt-1  |
privategpt-private-gpt-1  | During handling of the above exception, another exception occurred:
privategpt-private-gpt-1  |
privategpt-private-gpt-1  | Traceback (most recent call last):
privategpt-private-gpt-1  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/gradio/queueing.py", line 495, in call_prediction
privategpt-private-gpt-1  |     output = await route_utils.call_process_api(
privategpt-private-gpt-1  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
privategpt-private-gpt-1  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/gradio/route_utils.py", line 235, in call_process_api
privategpt-private-gpt-1  |     output = await app.get_blocks().process_api(
privategpt-private-gpt-1  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
privategpt-private-gpt-1  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/gradio/blocks.py", line 1627, in process_api
privategpt-private-gpt-1  |     result = await self.call_function(
privategpt-private-gpt-1  |              ^^^^^^^^^^^^^^^^^^^^^^^^^
privategpt-private-gpt-1  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/gradio/blocks.py", line 1173, in call_function
privategpt-private-gpt-1  |     prediction = await anyio.to_thread.run_sync(
privategpt-private-gpt-1  |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
privategpt-private-gpt-1  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/anyio/to_thread.py", line 33, in run_sync
privategpt-private-gpt-1  |     return await get_asynclib().run_sync_in_worker_thread(
privategpt-private-gpt-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
privategpt-private-gpt-1  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
privategpt-private-gpt-1  |     return await future
privategpt-private-gpt-1  |            ^^^^^^^^^^^^
privategpt-private-gpt-1  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 807, in run
privategpt-private-gpt-1  |     result = context.run(func, *args)
privategpt-private-gpt-1  |              ^^^^^^^^^^^^^^^^^^^^^^^^
privategpt-private-gpt-1  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/gradio/utils.py", line 690, in wrapper
privategpt-private-gpt-1  |     response = f(*args, **kwargs)
privategpt-private-gpt-1  |                ^^^^^^^^^^^^^^^^^^
privategpt-private-gpt-1  |   File "/home/worker/app/private_gpt/ui/ui.py", line 252, in _upload_file
privategpt-private-gpt-1  |     self._ingest_service.bulk_ingest([(str(path.name), path) for path in paths])
privategpt-private-gpt-1  |   File "/home/worker/app/private_gpt/server/ingest/ingest_service.py", line 87, in bulk_ingest
privategpt-private-gpt-1  |     documents = self.ingest_component.bulk_ingest(files)
privategpt-private-gpt-1  |                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
privategpt-private-gpt-1  |   File "/home/worker/app/private_gpt/components/ingest/ingest_component.py", line 135, in bulk_ingest
privategpt-private-gpt-1  |     saved_documents.extend(self._save_docs(documents))
privategpt-private-gpt-1  |                            ^^^^^^^^^^^^^^^^^^^^^^^^^^
privategpt-private-gpt-1  |   File "/home/worker/app/private_gpt/components/ingest/ingest_component.py", line 142, in _save_docs
privategpt-private-gpt-1  |     self._index.insert(document, show_progress=True)
privategpt-private-gpt-1  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/llama_index/core/indices/base.py", line 225, in insert
privategpt-private-gpt-1  |     nodes = run_transformations(
privategpt-private-gpt-1  |             ^^^^^^^^^^^^^^^^^^^^
privategpt-private-gpt-1  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/llama_index/core/ingestion/pipeline.py", line 127, in run_transformations
privategpt-private-gpt-1  |     nodes = transform(nodes, **kwargs)
privategpt-private-gpt-1  |             ^^^^^^^^^^^^^^^^^^^^^^^^^^
privategpt-private-gpt-1  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/llama_index/core/base/embeddings/base.py", line 334, in __call__
privategpt-private-gpt-1  |     embeddings = self.get_text_embedding_batch(
privategpt-private-gpt-1  |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
privategpt-private-gpt-1  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/llama_index/core/base/embeddings/base.py", line 255, in get_text_embedding_batch
privategpt-private-gpt-1  |     embeddings = self._get_text_embeddings(cur_batch)
privategpt-private-gpt-1  |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
privategpt-private-gpt-1  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/llama_index/embeddings/ollama/base.py", line 64, in _get_text_embeddings
privategpt-private-gpt-1  |     embeddings = self.get_general_text_embedding(text)
privategpt-private-gpt-1  |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
privategpt-private-gpt-1  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/llama_index/embeddings/ollama/base.py", line 89, in get_general_text_embedding
privategpt-private-gpt-1  |     response = requests.post(
privategpt-private-gpt-1  |                ^^^^^^^^^^^^^^
privategpt-private-gpt-1  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/requests/api.py", line 115, in post
privategpt-private-gpt-1  |     return request("post", url, data=data, json=json, **kwargs)
privategpt-private-gpt-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
privategpt-private-gpt-1  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/requests/api.py", line 59, in request
privategpt-private-gpt-1  |     return session.request(method=method, url=url, **kwargs)
privategpt-private-gpt-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
privategpt-private-gpt-1  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
privategpt-private-gpt-1  |     resp = self.send(prep, **send_kwargs)
privategpt-private-gpt-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
privategpt-private-gpt-1  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
privategpt-private-gpt-1  |     r = adapter.send(request, **kwargs)
privategpt-private-gpt-1  |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
privategpt-private-gpt-1  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/requests/adapters.py", line 519, in send
privategpt-private-gpt-1  |     raise ConnectionError(e, request=request)
privategpt-private-gpt-1  | requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=11434): Max retries exceeded with url: /api/embeddings (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xffff609ddf10>: Failed to establish a new connection: [Errno 111] Connection refused'))
drupol commented 3 months ago

I'm pretty sure this has nothing to do with private-gpt but with networking shenanigans related to containers.

BenBatsir commented 3 months ago

I've pretty much tried to dockerize it according to the docs

drupol commented 3 months ago

Is Ollama running from inside the container too? (localhost)

link-kut commented 3 months ago

Please unsubscribe me.

2024년 4월 21일 (일) 오후 4:49, Pol Dellaiera @.***>님이 작성:

Is Ollama running from inside the container too? (localhost)

— Reply to this email directly, view it on GitHub https://github.com/zylon-ai/private-gpt/issues/1872#issuecomment-2067945925, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA4ZVIQMMQD7SRSFZI2E4G3Y6NVRVAVCNFSM6AAAAABGRE3LGOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANRXHE2DKOJSGU . You are receiving this because you are subscribed to this thread.Message ID: @.***>

BenBatsir commented 3 months ago

Is Ollama running from inside the container too? (localhost)

no

drupol commented 3 months ago

That's the issue. From the container localhost IS the container, therefore, it will never reach the Ollama instance.

BenBatsir commented 3 months ago

That's the issue. From the container localhost IS the container, therefore, it will never reach the Ollama instance.

it makes sense, but then again I'm surprised how come I need to change anything given I'm simply following the documentation.

Maybe I need to expose port 11434 to this container ?

drupol commented 3 months ago

Maybe I need to expose port 11434 to this container ?

No, that is pointless.

What you need to do instead is to replace occurrences of localhost:11434 with <host-ip>:11434. This way, the container will be able to connect to it.

BenBatsir commented 3 months ago

Maybe I need to expose port 11434 to this container ?

No, that is pointless.

What you need to do instead is to replace occurrences of localhost:11434 with <host-ip>:11434. This way, the container will be able to connect to it.

Not sure I understand where exactly. In the code itself?

drupol commented 3 months ago

private-gpt uses YAML files for its configuration. The ollama address to connect to is hardcoded in these configuration files:

https://github.com/zylon-ai/private-gpt/blob/c1802e7cf0e56a2603213ec3b6a4af8fadb8a17a/settings-ollama.yaml#L16

What you need to do is to create your custom YAML file and use it in Docker (via a directory binding?).

AFAIK and since there's no environment variable to change that Ollama URL, you don't have other choice.

BenBatsir commented 3 months ago

private-gpt uses YAML files for its configuration. The ollama address to connect to is hardcoded in these configuration files:

https://github.com/zylon-ai/private-gpt/blob/c1802e7cf0e56a2603213ec3b6a4af8fadb8a17a/settings-ollama.yaml#L16

What you need to do is to create your custom YAML file and use it in Docker (via a directory binding?).

AFAIK and since there's no environment variable to change that Ollama URL, you don't have other choice.

ok... so localhost should be change to what? 127.0.0.1 ? something else?

drupol commented 3 months ago

I guess your machine has a local IP address ? Then use it.

BenBatsir commented 3 months ago

I guess your machine has a local IP address ? Then use it.

I ended up changing it to the container name: 

ollama:
  llm_model: mistral
  embedding_model: nomic-embed-text
  api_base: http://privategpt-ollama-1:11434
  embedding_api_base: http://privategpt-ollama-1:11434

Now I'm having a different issue:

ValueError: Ollama call failed with status code 404. Details: model 'nomic-embed-text' not found, try pulling it first
drupol commented 3 months ago

That's very good news. That means you succeeded to connect to Ollama.

Now you need to read and hopefully tackle the next issue.

To fix it, just do: ollama pull nomic-embed-text

BenBatsir commented 3 months ago

That's very good news. That means you succeeded to connect to Ollama.

Now you need to read and hopefully tackle the next issue.

To fix it, just do: ollama pull nomic-embed-text

it worked! so should I add it in Dockerfile.external as a step ?

drupol commented 3 months ago

I have no knowledge in Dockerfile.external, but very curious to learn how you fixed the issue. It would be nice if you could document it here too for future reference.

BenBatsir commented 3 months ago

I have no knowledge in Dockerfile.external, but very curious to learn how you fixed the issue. It would be nice if you could document it here too for future reference.

actually this docker file belongs to the private-gpt image, so I'll need to figure this out somehow, but I will document it once I'll find a suitable solution.

Thanks a lot for your help

jaluma commented 1 week ago

@BenBatsir You can't add this line to Dockerfile.external, as it is something you need to run on the ollama container.

In this situation, I have three ideas on how to fix it:

  1. Modify the command in docker-compose and replace it with something like: ollama pull nomic-embed-text && ollama pull mistral && ollama serve.
  2. Open a bash terminal on the container and pull both models.
  3. Create a custom version of Ollama with the downloaded model.

Anyway, we will talk internally about this problem and try to fix a solution to prevent further related problems.