Closed xtfocus closed 6 months ago
That is because phoGPT
uses tensors with bias
parameter in addition to weight
parameter and llama.cpp
currently does not support MPT-trained model with such feature. I created a fork of llama.cpp
here some months back to make it able to run phoGPT
. Perhaps applying same method will also work with latest version of llama.cpp
Thanks. I will take a look and respond back soon.
@xtfocus I faced the same issue. Then I found @nviet's solution on Huggingface, and it worked!
cd llama-cpp-python
git checkout e9bc4c4baf3f121a178dec215770ccd0ac86c28e
cd vendor
git clone https://github.com/nviet/llama.cpp
cd ..
pip install .
Thanks @nviet and @xtfocus Here is the PR I just made for llama.cpp to address this issue, following @nviet's commit with minor modifications to make 'bias' parameters optional (i.e. now it works for both MPT and PhoGPT): https://github.com/ggerganov/llama.cpp/pull/5638
PR was merged. I will close this issue.
I successfully converted the model to the gguf format using llama.cpp
convert-hf-to-gguf.py
scriptOutput
I tried inference and the error showed up