digitalscream
commented
6 months ago I've been checking this section of the Ollama logs, just to make sure it's using both GPUs and getting past the 8GB limit mentioned in tickets from a few months back:
[SYCL] call ggml_init_sycl
ggml_init_sycl: GGML_SYCL_DEBUG: 0
ggml_init_sycl: GGML_SYCL_F16: no
found 6 SYCL devices:
| | | |Compute |Max compute|Max work|Max sub| |
|ID| Device Type| Name|capability|units |group |group |Global mem size|
|--|------------------|---------------------------------------------|----------|-----------|--------|-------|---------------|
| 0|[level_zero:gpu:0]| Intel(R) Arc(TM) A770 Graphics| 1.3| 512| 1024| 32| 16225243136|
| 1|[level_zero:gpu:1]| Intel(R) Arc(TM) A770 Graphics| 1.3| 512| 1024| 32| 16225243136|
| 2| [opencl:gpu:0]| Intel(R) Arc(TM) A770 Graphics| 3.0| 512| 1024| 32| 16225243136|
| 3| [opencl:gpu:1]| Intel(R) Arc(TM) A770 Graphics| 3.0| 512| 1024| 32| 16225243136|
| 4| [opencl:cpu:0]|AMD Ryzen 5 3600 6-Core Processor | 3.0| 12| 8192| 64| 67349479424|
| 5| [opencl:acc:0]| Intel(R) FPGA Emulation Device| 1.2| 12|67108864| 64| 67349479424|
ggml_backend_sycl_set_mul_device_mode: true
detect 2 SYCL GPUs: [0,1] with top Max compute units:512
llm_load_tensors: ggml ctx size = 0.50 MiB
llm_load_tensors: offloading 48 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 49/49 layers to GPU
llm_load_tensors: SYCL0 buffer size = 9282.81 MiB
llm_load_tensors: SYCL1 buffer size = 8745.30 MiB
llm_load_tensors: SYCL_Host buffer size = 140.62 MiB
....................................................................................................
llama_new_context_with_model: n_ctx = 2048
llama_new_context_with_model: n_batch = 512
llama_new_context_with_model: n_ubatch = 512
llama_new_context_with_model: freq_base = 1000000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init: SYCL0 KV buffer size = 200.00 MiB
llama_kv_cache_init: SYCL1 KV buffer size = 184.00 MiB
llama_new_context_with_model: KV self size = 384.00 MiB, K (f16): 192.00 MiB, V (f16): 192.00 MiB
llama_new_context_with_model: SYCL_Host output buffer size = 0.15 MiB
llama_new_context_with_model: SYCL0 compute buffer size = 324.00 MiB
llama_new_context_with_model: SYCL1 compute buffer size = 324.00 MiB
llama_new_context_with_model: SYCL_Host compute buffer size = 20.01 MiB
llama_new_context_with_model: graph nodes = 1590
llama_new_context_with_model: graph splits = 3Is there anything else I should be looking at to figure out what's going on, or is that enough info?
yangw1234
sgwhat
System specs: 2 x A770, one on PCIE 3.0 x16, one on PCIE 3.0 x4, Ryzen 3600, 64GB RAM (EDIT: Host is Ubuntu 22.04, Docker base image is
intelanalytics/ipex-llm-xpu:latest).When running Ollama in Docker, using the setup described here:
https://ipex-llm.readthedocs.io/en/latest/doc/LLM/Quickstart/ollama_quickstart.html
...it loads large models just fine (eg Codellama 34b, with approx. 10GB on each GPU), but the response generation starts around 30t/s and gradually slows down to 5t/s within the same response.
At the same time, the GPU frequency slows from 1Ghz+ down to 100-300MHz.
The GPUs aren't power-starved, and they're running in open air (so no cooling problems, ambient temp is ~18C).
Worth noting that I'm running with
source ipex-llm-init -c -gin the container startup and--no-mmapon llama.cpp (it segfaults without the former, and hangs or outright crashes without the latter).I know it's not ideal running the second GPU on PCIE x4, but I'd have thought that'd cause general slowness rather than a gradual slowdown.
It shows the exact same behaviour with smaller models, too - eg LLama 3 8b and Mistral 7b. With those smaller models, running on a single GPU returns 50-60t/s.