search

Abdullahw72 / langchain-chatbot-multiple-PDF

Langchain Chatbot with Multiple PDF Support using OpenAI and Hugging Face Models
54 stars 31 forks source link

Huggingface Error #2

Closed feynmanish closed 8 months ago

feynmanish commented 1 year ago

It works with openAi, but with HuggingFace Getting "Unable to load weights from pytorch checkpoint file" when loading model from transformers error occurs. Any suggestions?

Abdullahw72 commented 1 year ago

Hi feynmanish, can you share which HuggingFace model are you trying to load? Also, please share your GPU VRAM. HuggingFace models require high VRAM as they run locally.

feynmanish commented 1 year ago

128 mb, but there is a similar issue with most capable machines: https://github.com/invoke-ai/InvokeAI/issues/2234

PinkalShah2022 commented 1 year ago

@Abdullahw72 : Great work, But wanted to know what should be bare minimum system requirement to run the code + for every user we need to have separate instance to chat. How much extra memory it will consume

Abdullahw72 commented 1 year ago

@PinkalShah2022 Thank you! It really depends on which model you want to use, to be honest. If you use a hugging face model, you'll need to look for really small models like Falcon 7B parameters, it will require at least 12 GB VRAM and a pretty decent CPU. Look for specs of each model on Huggingface under the model descriptions! Same for the memory, depends on use-case but 16 -32 GB RAM would be a good place to begin for any small/medium sized LLM.