Open Abdullah-kwl opened 3 months ago
Hi @Abdullah-kwl
Thanks for the issue ! Can you make sure you have the latest transformers installed ? pip install -U transformers
yes, I am using an updated transformer library. in collab I am using the same command to download transformers. pip install -U transformers
Thanks @Abdullah-kwl , will try to repro and report back here
Hi @Abdullah-kwl I ran successfully the code on fresh new Google Colab env: https://colab.research.google.com/drive/1rfJZp3DsbavH6IFo-rXUDNvqwFJVIbOb?usp=sharing Note we do run a bunch of tests: https://github.com/huggingface/transformers/blob/main/tests/quantization/ggml/test_ggml.py and they do all pass on our end as of today !
@younesbelkada yes i again run it now its working there was some dependencies conflict with other libraries but now its running.
but now I am facing problem that my session crashed after using all available RAM, I think its loading model on ram but when I use llama-cpp-python it does not load model on ram and we can easily inference with larger models even more then 7B. is there is a way that my session doe not crash due to ram usage.
try out using this model:
model_id = "TheBloke/WestLake-7B-v2-GGUF" filename = "westlake-7b-v2.Q2_K.gguf" model = AutoModel.from_pretrained(model_id, gguf_file=filename) tokenizer = AutoTokenizer.from_pretrained(model_id, gguf_file=filename)
Hello, I have a general question about using GGUF models with AutoModel.
Is there any difference in the implementation if compared to the ctransformers library? Until now, I used GGUF models in Python with ctransformers, and it is really fast in generating responses. Using GGUF from AutoModel, instead, significantly slow down the inference time. Any clue about this? I am not sure if this transformers compatibility with GGUF models is intended for the the same usage as for ctransformers or llama.cpp or I am doing something wrong. Also, is there any way to set up "context_length" and "gpu_layers" offloading from AutoTokenizer?
Thanks!
cc @SunMarc
Feature request
https://huggingface.co/docs/transformers/main/en/gguf in above documentation it shows that it loads the gguf model and provided the simple example
from transformers import AutoTokenizer, AutoModelForCausalLM
model_id = "TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF" filename = "tinyllama-1.1b-chat-v1.0.Q6_K.gguf"
model = AutoModelForCausalLM.from_pretrained(model_id, gguf_file=filename) tokenizer = AutoTokenizer.from_pretrained(model_id, gguf_file=filename)
but when I run the code it shows the error : OSError: TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF does not appear to have a file named pytorch_model.bin, tf_model.h5, model.ckpt or flax_model.msgpack. and some time show your transformer may be not updated
even after updating transformer it shows the same error not loading gguf model , add support to load gguf model
Motivation
it will solve to load gguf model with out the help of other library such as llama.cpp and ollama
Your contribution
I do not have a complete implementation in mind, but I suggest to start with previous method which mention in https://huggingface.co/docs/transformers/main/en/gguf