Cannot convert model file type using llama.cpp.

[JIANGTUNAN]

start command：

jetson-containers run $(autotag llama_cpp)
# The interaction pattern goes into the container
cd /opt/llama.cpp
python3 convert_hf_to_gguf.py

error message：

root@ubuntu:/opt/llama.cpp# python3 convert_hf_to_gguf.py
Traceback (most recent call last):
  File "/opt/llama-cpp-python/vendor/llama.cpp/convert_hf_to_gguf.py", line 20, in <module>
    import torch
ModuleNotFoundError: No module named 'torch'

I tried to install it manually：

root@ubuntu:/opt/llama.cpp# pip install torch torchvision torchaudio
root@ubuntu:/opt/llama.cpp# python3 -c "import torch; print(torch.__version__)"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.10/dist-packages/torch/__init__.py", line 289, in <module>
    _load_global_deps()
  File "/usr/local/lib/python3.10/dist-packages/torch/__init__.py", line 245, in _load_global_deps
    raise err
  File "/usr/local/lib/python3.10/dist-packages/torch/__init__.py", line 226, in _load_global_deps
    ctypes.CDLL(global_deps_lib_path, mode=ctypes.RTLD_GLOBAL)
  File "/usr/lib/python3.10/ctypes/__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)

I tried to use the conversion script from the llama.cpp project (convert_hf_to_gguf.py) and ran into problems converting model file types. I want to convert from safetensors togguf. Is there any better advice? Be grateful for your help.

[johnnynunez]

@dusty-nv maybe needs: sudo apt install libopenmpi3 build of pytorch that didn't have it included.

[dusty-nv]

@JIANGTUNAN yes, in the container this:

apt-get update &&  \
apt-get install -y --no-install-recommends \
    libopenblas-dev \
    libopenmpi-dev \
    openmpi-bin \
    openmpi-common \
    gfortran \
    libomp-dev

@johnnynunez I had not been including PyTorch/ect in the llama.cpp build to keep it minimal. However I see now that for the HF->GGUF conversion, it is needed. In the future, by default I think I will rebase llama.cpp on pytorch image, and have a separate llama.cpp:minimal image without it.

[johnnynunez]

@JIANGTUNAN yes, in the container this:

apt-get update &&  \
apt-get install -y --no-install-recommends \
    libopenblas-dev \
    libopenmpi-dev \
    openmpi-bin \
    openmpi-common \
    gfortran \
    libomp-dev

@johnnynunez I had not been including PyTorch/ect in the llama.cpp build to keep it minimal. However I see now that for the HF->GGUF conversion, it is needed. In the future, by default I think I will rebase llama.cpp on pytorch image, and have a separate llama.cpp:minimal image without it.

Thank you dusty