Open kai-breitbarth opened 1 year ago
I had a similar problem to you. If you installed your dependencies through conda install --file requirements.txt
then I'd suggest reinstalling the dependencies through:
pip install -r requirements.txt
and running python run_localGPT_API.py
then.
@kai-breitbarth at the moment, GGML files are only supported on CPU/MPS. If you are running this on Nvidia GPU, look for GPTQ files for the models. There is a PR I am testing which will fix this issue.
Pass device type as 'cpu' in line 228. It should fix the problem.
I just pushed a new update, that will fix it. Make sure you re-install llama-cpp if you are running it on GPU using the instructions provided in the readme for ggml models.
Thanks a lot for the fast help! @DeutscheGabanna Moin! Until now I didn't try the API. The API runs with the Wizard model on GPU! So a first success! @PromtEngineer thanks a lot for the update! I rebuild everything regarding the readme. It works now. But for some weird reasons still using the CPU. I switched it to the wizard model an then it runs on GPU like expected. So no Llama2 but a running version.
Ohhhhhhh. It's the difference between GGML and GPTQ... of course. I have to learn a lot...
can someone pls help im still getting this errorr , i dont have GPU only CPU but keep getting [OSError: Can't load tokenizer for 'TheBloke/Llama-2-7B-Chat-GGML']
@helloworld4774 what base model name are you using?
model_id="TheBloke/Llama-2-7B-Chat-GGML" model_basename = "llama-2-7b-chat.ggmlv3.q4_0.bin"
install new conda environment and try to run to python code (after some cd...) without pip installing any libraries, worked for me for the model - vilsonrodrigues/falcon-7b-instruct-sharded.. Hope it works.
@PromtEngineer I have the same issue. I am running on Window 11 with CPU. Have the same error while loading Llama2 tokenizer
@PromtEngineer , I have a Pull Request to fix this.. Please review
I didn't change anything in run_localGPT.py:
I got everything installed until I tried to start
python run_localGPT.py
The error message is:
I have Cuda 11.7, AutoGPT installed successfully:
I found this in Huggingface: https://huggingface.co/TheBloke/LLaMa-7B-GGML/discussions/2 and also tried to install ctransformers with
CT_CUBLAS=1 pip install ctransformers --no-binary ctransformers
No clue what I'm doing wrong. Any help appreciated.
Thanks,
Kai