search

marella / ctransformers

Python bindings for the Transformer models implemented in C/C++ using GGML library.
MIT License
1.76k stars 137 forks source link

Segfault with DeepSeek GGUF models #183

Closed freckletonj closed 7 months ago

freckletonj commented 7 months ago

I tried out deepseek-coder-X-.gguf with ctransformers. https://huggingface.co/TheBloke/deepseek-coder-33B-instruct-GGUF

Upon loading it, ie model = AutoModelForCausalLM.from_pretrained(...), I get the following error:

ERROR: byte not found in vocab: '
'
Segmentation fault (core dumped)

For reference, this model works with llama.cpp.

freckletonj commented 7 months ago

I found some discussions, but have not dug in sufficiently yet.

https://huggingface.co/TheBloke/deepseek-coder-33B-instruct-GGUF/discussions/2

https://github.com/oobabooga/text-generation-webui/discussions/4495

TheBloke commented 7 months ago

ctransformers hasn't been updated since early September and doesn't support models that use a BPE vocab, like Deepseek Coder

I'd recommend using llama-cpp-python until ctransformers has been updated - we're not sure if/when that will happen as marella, the developer, is not available at the moment.

freckletonj commented 7 months ago

Thanks @TheBloke, I'll switch over. Also, geez you have incredible work ethic and a positive impact on this ecosystem. Thank you!