search

Pelochus / ezrknn-llm

Easier usage of LLMs in Rockchip's NPU on SBCs like Orange Pi 5 and Radxa Rock 5 series
Other
36 stars 3 forks source link

Docker converting LLMs for Rockchip's NPUs killed #8

Closed thanhtantran closed 1 month ago

thanhtantran commented 1 month ago

Hello,

I am using WLS from a Windows 10 machine and try to convert Qwen-1_8B-Chat using docker Desktop on Windows

everything works fine until the converting process, i got killed and container stopped everytime runs on this step

root@d7769b94e50c:/ezrknn-llm/rkllm-toolkit# cd examples
root@d7769b94e50c:/ezrknn-llm/rkllm-toolkit/examples# ls
huggingface
root@d7769b94e50c:/ezrknn-llm/rkllm-toolkit/examples# cd huggingface
root@d7769b94e50c:/ezrknn-llm/rkllm-toolkit/examples/huggingface# git lfs pull
root@d7769b94e50c:/ezrknn-llm/rkllm-toolkit/examples/huggingface# ls
Qwen-1_8B-Chat  test.py
root@d7769b94e50c:/ezrknn-llm/rkllm-toolkit/examples/huggingface# cd Qwen-1_8B-Chat
root@d7769b94e50c:/ezrknn-llm/rkllm-toolkit/examples/huggingface/Qwen-1_8B-Chat# git lfs pull
root@d7769b94e50c:/ezrknn-llm/rkllm-toolkit/examples/huggingface/Qwen-1_8B-Chat# cd ..
root@d7769b94e50c:/ezrknn-llm/rkllm-toolkit/examples/huggingface# ls
Qwen-1_8B-Chat  test.py
root@d7769b94e50c:/ezrknn-llm/rkllm-toolkit/examples/huggingface# python3 test.py
rkllm-toolkit version: 1.0.1
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████| 2/2 [00:21<00:00, 11.00s/it]
Optimizing model: 100%|█████████████████████████████████████████████████████████████████| 24/24 [10:58<00:00, 27.45s/it]
Converting model: 100%|███████████████████████████████████████████████████████████| 195/195 [00:00<00:00, 712447.11it/s]
Killed
root@d7769b94e50c:/ezrknn-llm/rkllm-toolkit/examples/huggingface#

How many RAM the computer need to have to convert?

Pelochus commented 1 month ago

For Qwen 1.8B I think it was around 14-20 GB RAM free, so ideally, you should have 16 GB RAM, nothing running in the background and at least 4 GB of swap memory.

Pelochus commented 1 month ago

You can also use this one already converted for 1.0.1:

https://huggingface.co/FydeOS/Qwen-1_8B-Chat_rkLLM/tree/main