search

zylon-ai / private-gpt

Interact with your documents using the power of GPT, 100% privately, no data leaks
https://privategpt.dev
Apache License 2.0
53.57k stars 7.2k forks source link

[BUG] Failed to connect to Ollama #2071

Open mwsmws22 opened 2 weeks ago

mwsmws22 commented 2 weeks ago

Pre-check

Description

+] Running 3/0
 ⠿ Container private-gpt-ollama-cpu-1          Created                                                                                                                                                                                       0.0s
 ⠿ Container private-gpt-ollama-1              Created                                                                                                                                                                                       0.0s
 ⠿ Container private-gpt-private-gpt-ollama-1  Created                                                                                                                                                                                       0.0s
Attaching to private-gpt-ollama-1, private-gpt-ollama-cpu-1, private-gpt-private-gpt-ollama-1
private-gpt-ollama-cpu-1          | 2024/08/28 12:18:14 routes.go:1125: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_LLM_LIBRARY: OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/root/.ollama/models OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://*] OLLAMA_RUNNERS_DIR: OLLAMA_SCHED_SPREAD:false OLLAMA_TMPDIR: ROCR_VISIBLE_DEVICES:]"
private-gpt-ollama-cpu-1          | time=2024-08-28T12:18:14.485Z level=INFO source=images.go:753 msg="total blobs: 0"
private-gpt-ollama-cpu-1          | time=2024-08-28T12:18:14.485Z level=INFO source=images.go:760 msg="total unused blobs removed: 0"
private-gpt-ollama-cpu-1          | time=2024-08-28T12:18:14.486Z level=INFO source=routes.go:1172 msg="Listening on [::]:11434 (version 0.3.8)"
private-gpt-ollama-cpu-1          | time=2024-08-28T12:18:14.487Z level=INFO source=payload.go:30 msg="extracting embedded files" dir=/tmp/ollama1880285689/runners
private-gpt-ollama-1              | time="2024-08-28T12:18:14Z" level=info msg="Configuration loaded from flags."
private-gpt-private-gpt-ollama-1  | 12:18:15.035 [INFO    ] private_gpt.settings.settings_loader - Starting application with profiles=['default', 'docker']
private-gpt-private-gpt-ollama-1  | None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
private-gpt-private-gpt-ollama-1  | 12:18:20.162 [INFO    ] private_gpt.components.llm.llm_component - Initializing the LLM in mode=ollama
private-gpt-ollama-cpu-1          | time=2024-08-28T12:18:27.607Z level=INFO source=payload.go:44 msg="Dynamic LLM libraries [cuda_v12 rocm_v60102 cpu cpu_avx cpu_avx2 cuda_v11]"
private-gpt-ollama-cpu-1          | time=2024-08-28T12:18:27.607Z level=INFO source=gpu.go:200 msg="looking for compatible GPUs"
private-gpt-ollama-cpu-1          | time=2024-08-28T12:18:27.607Z level=WARN source=gpu.go:222 msg="CPU does not have minimum vector extensions, GPU inference disabled" required=avx detected="no vector extensions"
private-gpt-ollama-cpu-1          | time=2024-08-28T12:18:27.608Z level=INFO source=types.go:107 msg="inference compute" id=0 library=cpu variant="no vector extensions" compute="" driver=0.0 name="" total="15.4 GiB" available="10.7 GiB"
private-gpt-ollama-cpu-1          | [GIN] 2024/08/28 - 12:18:27 | 200 |      30.373µs |      172.22.0.2 | GET      "/"
private-gpt-ollama-cpu-1          | [GIN] 2024/08/28 - 12:18:35 | 200 |      67.709µs |      172.22.0.2 | GET      "/"
private-gpt-ollama-cpu-1          | [GIN] 2024/08/28 - 12:18:40 | 200 |        64.4µs |      172.22.0.2 | GET      "/"
private-gpt-ollama-cpu-1          | [GIN] 2024/08/28 - 12:18:45 | 200 |       30.68µs |      172.22.0.2 | GET      "/"
private-gpt-private-gpt-ollama-1  | 12:18:50.267 [INFO    ]                     httpx - HTTP Request: GET http://ollama:11434/api/tags "HTTP/1.1 504 Gateway Timeout"
private-gpt-private-gpt-ollama-1  | 12:18:50.268 [ERROR   ]  private_gpt.utils.ollama - Failed to connect to Ollama: Gateway Timeout
private-gpt-private-gpt-ollama-1  | Traceback (most recent call last):
private-gpt-private-gpt-ollama-1  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/injector/__init__.py", line 798, in get
private-gpt-private-gpt-ollama-1  |     return self._context[key]
private-gpt-private-gpt-ollama-1  |            ~~~~~~~~~~~~~^^^^^
private-gpt-private-gpt-ollama-1  | KeyError: <class 'private_gpt.ui.ui.PrivateGptUi'>
private-gpt-private-gpt-ollama-1  | 
private-gpt-private-gpt-ollama-1  | During handling of the above exception, another exception occurred:
private-gpt-private-gpt-ollama-1  | 
private-gpt-private-gpt-ollama-1  | Traceback (most recent call last):
private-gpt-private-gpt-ollama-1  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/injector/__init__.py", line 798, in get
private-gpt-private-gpt-ollama-1  |     return self._context[key]
private-gpt-private-gpt-ollama-1  |            ~~~~~~~~~~~~~^^^^^
private-gpt-private-gpt-ollama-1  | KeyError: <class 'private_gpt.server.ingest.ingest_service.IngestService'>
private-gpt-private-gpt-ollama-1  | 
private-gpt-private-gpt-ollama-1  | During handling of the above exception, another exception occurred:
private-gpt-private-gpt-ollama-1  | 
private-gpt-private-gpt-ollama-1  | Traceback (most recent call last):
private-gpt-private-gpt-ollama-1  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/injector/__init__.py", line 798, in get
private-gpt-private-gpt-ollama-1  |     return self._context[key]
private-gpt-private-gpt-ollama-1  |            ~~~~~~~~~~~~~^^^^^
private-gpt-private-gpt-ollama-1  | KeyError: <class 'private_gpt.components.llm.llm_component.LLMComponent'>
private-gpt-private-gpt-ollama-1  | 
private-gpt-private-gpt-ollama-1  | During handling of the above exception, another exception occurred:
private-gpt-private-gpt-ollama-1  | 
private-gpt-private-gpt-ollama-1  | Traceback (most recent call last):
private-gpt-private-gpt-ollama-1  |   File "<frozen runpy>", line 198, in _run_module_as_main
private-gpt-private-gpt-ollama-1  |   File "<frozen runpy>", line 88, in _run_code
private-gpt-private-gpt-ollama-1  |   File "/home/worker/app/private_gpt/__main__.py", line 5, in <module>
private-gpt-private-gpt-ollama-1  |     from private_gpt.main import app
private-gpt-private-gpt-ollama-1  |   File "/home/worker/app/private_gpt/main.py", line 6, in <module>
private-gpt-private-gpt-ollama-1  |     app = create_app(global_injector)
private-gpt-private-gpt-ollama-1  |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
private-gpt-private-gpt-ollama-1  |   File "/home/worker/app/private_gpt/launcher.py", line 66, in create_app
private-gpt-private-gpt-ollama-1  |     ui = root_injector.get(PrivateGptUi)
private-gpt-private-gpt-ollama-1  |          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
private-gpt-private-gpt-ollama-1  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/injector/__init__.py", line 91, in wrapper
private-gpt-private-gpt-ollama-1  |     return function(*args, **kwargs)
private-gpt-private-gpt-ollama-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^
private-gpt-private-gpt-ollama-1  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/injector/__init__.py", line 974, in get
private-gpt-private-gpt-ollama-1  |     provider_instance = scope_instance.get(interface, binding.provider)
private-gpt-private-gpt-ollama-1  |                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
private-gpt-private-gpt-ollama-1  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/injector/__init__.py", line 91, in wrapper
private-gpt-private-gpt-ollama-1  |     return function(*args, **kwargs)
private-gpt-private-gpt-ollama-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^
private-gpt-private-gpt-ollama-1  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/injector/__init__.py", line 800, in get
private-gpt-private-gpt-ollama-1  |     instance = self._get_instance(key, provider, self.injector)
private-gpt-private-gpt-ollama-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
private-gpt-private-gpt-ollama-1  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/injector/__init__.py", line 811, in _get_instance
private-gpt-private-gpt-ollama-1  |     return provider.get(injector)
private-gpt-private-gpt-ollama-1  |            ^^^^^^^^^^^^^^^^^^^^^^
private-gpt-private-gpt-ollama-1  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/injector/__init__.py", line 264, in get
private-gpt-private-gpt-ollama-1  |     return injector.create_object(self._cls)
private-gpt-private-gpt-ollama-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
private-gpt-private-gpt-ollama-1  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/injector/__init__.py", line 998, in create_object
private-gpt-private-gpt-ollama-1  |     self.call_with_injection(init, self_=instance, kwargs=additional_kwargs)
private-gpt-private-gpt-ollama-1  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/injector/__init__.py", line 1031, in call_with_injection
private-gpt-private-gpt-ollama-1  |     dependencies = self.args_to_inject(
private-gpt-private-gpt-ollama-1  |                    ^^^^^^^^^^^^^^^^^^^^
private-gpt-private-gpt-ollama-1  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/injector/__init__.py", line 91, in wrapper
private-gpt-private-gpt-ollama-1  |     return function(*args, **kwargs)
private-gpt-private-gpt-ollama-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^
private-gpt-private-gpt-ollama-1  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/injector/__init__.py", line 1079, in args_to_inject
private-gpt-private-gpt-ollama-1  |     instance: Any = self.get(interface)
private-gpt-private-gpt-ollama-1  |                     ^^^^^^^^^^^^^^^^^^^
private-gpt-private-gpt-ollama-1  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/injector/__init__.py", line 91, in wrapper
private-gpt-private-gpt-ollama-1  |     return function(*args, **kwargs)
private-gpt-private-gpt-ollama-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^
private-gpt-private-gpt-ollama-1  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/injector/__init__.py", line 974, in get
private-gpt-private-gpt-ollama-1  |     provider_instance = scope_instance.get(interface, binding.provider)
private-gpt-private-gpt-ollama-1  |                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
private-gpt-private-gpt-ollama-1  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/injector/__init__.py", line 91, in wrapper
private-gpt-private-gpt-ollama-1  |     return function(*args, **kwargs)
private-gpt-private-gpt-ollama-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^
private-gpt-private-gpt-ollama-1  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/injector/__init__.py", line 800, in get
private-gpt-private-gpt-ollama-1  |     instance = self._get_instance(key, provider, self.injector)
private-gpt-private-gpt-ollama-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
private-gpt-private-gpt-ollama-1  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/injector/__init__.py", line 811, in _get_instance
private-gpt-private-gpt-ollama-1  |     return provider.get(injector)
private-gpt-private-gpt-ollama-1  |            ^^^^^^^^^^^^^^^^^^^^^^
private-gpt-private-gpt-ollama-1  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/injector/__init__.py", line 264, in get
private-gpt-private-gpt-ollama-1  |     return injector.create_object(self._cls)
private-gpt-private-gpt-ollama-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
private-gpt-private-gpt-ollama-1  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/injector/__init__.py", line 998, in create_object
private-gpt-private-gpt-ollama-1  |     self.call_with_injection(init, self_=instance, kwargs=additional_kwargs)
private-gpt-private-gpt-ollama-1  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/injector/__init__.py", line 1031, in call_with_injection
private-gpt-private-gpt-ollama-1  |     dependencies = self.args_to_inject(
private-gpt-private-gpt-ollama-1  |                    ^^^^^^^^^^^^^^^^^^^^
private-gpt-private-gpt-ollama-1  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/injector/__init__.py", line 91, in wrapper
private-gpt-private-gpt-ollama-1  |     return function(*args, **kwargs)
private-gpt-private-gpt-ollama-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^
private-gpt-private-gpt-ollama-1  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/injector/__init__.py", line 1079, in args_to_inject
private-gpt-private-gpt-ollama-1  |     instance: Any = self.get(interface)
private-gpt-private-gpt-ollama-1  |                     ^^^^^^^^^^^^^^^^^^^
private-gpt-private-gpt-ollama-1  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/injector/__init__.py", line 91, in wrapper
private-gpt-private-gpt-ollama-1  |     return function(*args, **kwargs)
private-gpt-private-gpt-ollama-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^
private-gpt-private-gpt-ollama-1  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/injector/__init__.py", line 974, in get
private-gpt-private-gpt-ollama-1  |     provider_instance = scope_instance.get(interface, binding.provider)
private-gpt-private-gpt-ollama-1  |                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
private-gpt-private-gpt-ollama-1  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/injector/__init__.py", line 91, in wrapper
private-gpt-private-gpt-ollama-1  |     return function(*args, **kwargs)
private-gpt-private-gpt-ollama-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^
private-gpt-private-gpt-ollama-1  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/injector/__init__.py", line 800, in get
private-gpt-private-gpt-ollama-1  |     instance = self._get_instance(key, provider, self.injector)
private-gpt-private-gpt-ollama-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
private-gpt-private-gpt-ollama-1  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/injector/__init__.py", line 811, in _get_instance
private-gpt-private-gpt-ollama-1  |     return provider.get(injector)
private-gpt-private-gpt-ollama-1  |            ^^^^^^^^^^^^^^^^^^^^^^
private-gpt-private-gpt-ollama-1  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/injector/__init__.py", line 264, in get
private-gpt-private-gpt-ollama-1  |     return injector.create_object(self._cls)
private-gpt-private-gpt-ollama-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
private-gpt-private-gpt-ollama-1  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/injector/__init__.py", line 998, in create_object
private-gpt-private-gpt-ollama-1  |     self.call_with_injection(init, self_=instance, kwargs=additional_kwargs)
private-gpt-private-gpt-ollama-1  |   File "/home/worker/app/.venv/lib/python3.11/site-packages/injector/__init__.py", line 1040, in call_with_injection
private-gpt-private-gpt-ollama-1  |     return callable(*full_args, **dependencies)
private-gpt-private-gpt-ollama-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
private-gpt-private-gpt-ollama-1  |   File "/home/worker/app/private_gpt/components/llm/llm_component.py", line 169, in __init__
private-gpt-private-gpt-ollama-1  |     raise ValueError(
private-gpt-private-gpt-ollama-1  | ValueError: Failed to connect to Ollama, check if Ollama server is running on http://ollama:11434
private-gpt-ollama-cpu-1          | [GIN] 2024/08/28 - 12:18:50 | 200 |      28.263µs |      172.22.0.2 | GET      "/"
private-gpt-private-gpt-ollama-1 exited with code 1
private-gpt-ollama-cpu-1          | [GIN] 2024/08/28 - 12:18:55 | 200 |      29.718µs |      172.22.0.2 | GET      "/"
private-gpt-ollama-cpu-1          | [GIN] 2024/08/28 - 12:19:00 | 200 |      27.973µs |      172.22.0.2 | GET      "/"
private-gpt-ollama-cpu-1          | [GIN] 2024/08/28 - 12:19:05 | 200 |      64.762µs |      172.22.0.2 | GET      "/"
private-gpt-ollama-cpu-1          | [GIN] 2024/08/28 - 12:19:10 | 200 |      63.405µs |      172.22.0.2 | GET      "/"
private-gpt-ollama-cpu-1          | [GIN] 2024/08/28 - 12:19:15 | 200 |      59.763µs |      172.22.0.2 | GET      "/"
private-gpt-ollama-cpu-1          | [GIN] 2024/08/28 - 12:19:20 | 200 |      61.337µs |      172.22.0.2 | GET      "/"
private-gpt-ollama-cpu-1          | [GIN] 2024/08/28 - 12:19:26 | 200 |      67.367µs |      172.22.0.2 | GET      "/"
private-gpt-ollama-cpu-1          | [GIN] 2024/08/28 - 12:19:31 | 200 |      63.661µs |      172.22.0.2 | GET      "/"
private-gpt-ollama-cpu-1          | [GIN] 2024/08/28 - 12:19:36 | 200 |      65.992µs |      172.22.0.2 | GET      "/"
private-gpt-ollama-cpu-1          | [GIN] 2024/08/28 - 12:19:41 | 200 |      67.017µs |      172.22.0.2 | GET      "/"
^CGracefully stopping... (press Ctrl+C again to force)
Aborting on container exit...
[+] Running 3/3
 ⠿ Container private-gpt-private-gpt-ollama-1  Stopped                                                                                                                                                                                       0.0s
 ⠿ Container private-gpt-ollama-cpu-1          Stopped                                                                                                                                                                                       0.7s
 ⠿ Container private-gpt-ollama-1              Stopped                                                                                                                                                                                       0.6s
canceled

I have followed the quick start guide, but am unable to start private-gpt. I get the following error log above. These lines seem relevant:

msg="CPU does not have minimum vector extensions, GPU inference disabled" required=avx detected="no vector extensions" [INFO ] httpx - HTTP Request: GET http://ollama:11434/api/tags "HTTP/1.1 504 Gateway Timeout" [ERROR ] private_gpt.utils.ollama - Failed to connect to Ollama: Gateway Timeout Failed to connect to Ollama, check if Ollama server is running on http://ollama:11434

I tried to access localhost:11434 but no luck. Maybe my hardware is too underpowered to run Ollama? It's a little NUC with 16 GB RAM and the following CPU:

Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         39 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  4
  On-line CPU(s) list:   0-3
Vendor ID:               GenuineIntel
  Model name:            Intel(R) Celeron(R) N5105 @ 2.00GHz
    CPU family:          6
    Model:               156
    Thread(s) per core:  1
    Core(s) per socket:  4
    Socket(s):           1
    Stepping:            0
    CPU max MHz:         2900.0000
    CPU min MHz:         800.0000
    BogoMIPS:            3993.60

Steps to Reproduce

git clone https://github.com/zylon-ai/private-gpt.git cd private-gpt docker-compose --profile ollama-cpu up

Expected Behavior

private-gpt should be accessable at localhost:8001

Actual Behavior

It does not come up, see above error message.

Environment

Intel NUC / 16 GB RAM / Intel(R) Celeron(R) N5105 @ 2.00GHz

Additional Information

No response

Version

No response

Setup Checklist

NVIDIA GPU Setup Checklist

LumenYoung commented 2 weeks ago

I'm also experiencing the same issue following the documentation's default setup. Feel free to reach out for further information. But I also experience this not only from main but also when using the 0.6.3 release.

robvandyck commented 2 weeks ago

I also experience this issue.

the-jan-peter commented 2 weeks ago

Yes, me as well. Its a pity, that the docker images in the compse file are not taged, instead they are set to :latest. Maybe one of the maintainers could tell us a set of tags (version numbers) that work? Maybe that can help us.

gabryz95 commented 2 weeks ago

Hello everyone,

I encountered the same issue and resolved it by making a few adjustments to the docker-compose.yaml file. You can find my updated version below. docker-compose.yaml.zip

darkpowerxo commented 2 weeks ago

thank you!

Hello everyone,

I encountered the same issue and resolved it by making a few adjustments to the docker-compose.yaml file. You can find my updated version below. docker-compose.yaml.zip