PI_ERROR_OUT_OF_HOST_MEMORY Intel ARC 750

[vevilz]

I'm running on an intel arc 750, 32Gb RAM, there is more than enough disk space, what could be the problem?

sudo docker run -d \
    --device /dev/dri \
    -v /opt/ai/models/huggingface:/root/.cache/huggingface \
    -v /opt/ai/fastchat/logs:/logs \
    -p 7860:7860 \
    -p 8000:8000 \
    itlackey/ipex-arc-fastchat:latest \
    --model-path codellama/CodeLlama-7b-Instruct-hf --max-gpu-memory 7Gib

Downloading generation_config.json: 100%|██████████| 132/132 [00:00<00:00, 326kB/s]
2024-02-28 09:05:44 | ERROR | stderr |
2024-02-28 09:05:49 | ERROR | stderr | Traceback (most recent call last):
2024-02-28 09:05:49 | ERROR | stderr |   File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
2024-02-28 09:05:49 | ERROR | stderr |     return _run_code(code, main_globals, None,
2024-02-28 09:05:49 | ERROR | stderr |   File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
2024-02-28 09:05:49 | ERROR | stderr |     exec(code, run_globals)
2024-02-28 09:05:49 | ERROR | stderr |   File "/opt/venv/lib/python3.10/site-packages/fastchat/serve/model_worker.py", line 341, in <module>
2024-02-28 09:05:49 | ERROR | stderr |     args, worker = create_model_worker()
2024-02-28 09:05:49 | ERROR | stderr |   File "/opt/venv/lib/python3.10/site-packages/fastchat/serve/model_worker.py", line 315, in create_model_worker
2024-02-28 09:05:49 | ERROR | stderr |     worker = ModelWorker(
2024-02-28 09:05:49 | ERROR | stderr |   File "/opt/venv/lib/python3.10/site-packages/fastchat/serve/model_worker.py", line 74, in __init__
2024-02-28 09:05:49 | ERROR | stderr |     self.model, self.tokenizer = load_model(
2024-02-28 09:05:49 | ERROR | stderr |   File "/opt/venv/lib/python3.10/site-packages/fastchat/model/model_adapter.py", line 320, in load_model
2024-02-28 09:05:49 | ERROR | stderr |     model.to(device)
2024-02-28 09:05:49 | ERROR | stderr |   File "/opt/venv/lib/python3.10/site-packages/transformers/modeling_utils.py", line 2271, in to
2024-02-28 09:05:49 | ERROR | stderr |     return super().to(*args, **kwargs)
2024-02-28 09:05:49 | ERROR | stderr |   File "/opt/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1145, in to
2024-02-28 09:05:49 | ERROR | stderr |     return self._apply(convert)
2024-02-28 09:05:49 | ERROR | stderr |   File "/opt/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 797, in _apply
2024-02-28 09:05:49 | ERROR | stderr |     module._apply(fn)
2024-02-28 09:05:49 | ERROR | stderr |   File "/opt/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 797, in _apply
2024-02-28 09:05:49 | ERROR | stderr |     module._apply(fn)
2024-02-28 09:05:49 | ERROR | stderr |   File "/opt/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 797, in _apply
2024-02-28 09:05:49 | ERROR | stderr |     module._apply(fn)
2024-02-28 09:05:49 | ERROR | stderr |   [Previous line repeated 2 more times]
2024-02-28 09:05:49 | ERROR | stderr |   File "/opt/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 820, in _apply
2024-02-28 09:05:49 | ERROR | stderr |     param_applied = fn(param)
2024-02-28 09:05:49 | ERROR | stderr |   File "/opt/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1143, in convert
2024-02-28 09:05:49 | ERROR | stderr |     return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
2024-02-28 09:05:49 | ERROR | stderr | RuntimeError: Native API failed. Native API returns: -6 (PI_ERROR_OUT_OF_HOST_MEMORY) -6 (PI_ERROR_OUT_OF_HOST_MEMORY)

[ahhmino]

Did you ever find a solution to this?

[vevilz]

Did you ever find a solution to this?

I didn’t decide, I suspect it’s due to a lack of memory, I took arc770 for the test and everything worked.