`ChatHuggingFace` Asking for API Token Even for Locally Downloaded Model

[AnandUgale]

Checked other resources

• [X] I added a very descriptive title to this issue.
• [X] I searched the LangChain documentation with the integrated search.
• [X] I used the GitHub search to find a similar question and didn't find it.
• [X] I am sure that this is a bug in LangChain rather than my code.
• [X] The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Description

Issue

When using the ChatHuggingFace model from the langchain_huggingface library with a locally downloaded model, the system prompts for a Hugging Face API key. This behavior persists even though the model is stored locally and accessed without an internet connection. The issue does not occur when using the llm.invoke() method directly with the HuggingFacePipeline. Description

I have downloaded an LLM model locally and want to access it without an internet connection. The model works correctly when invoked directly using llm.invoke(), but when attempting to use the ChatHuggingFace model, it requires a Hugging Face API key.

Example Code

from transformers import BitsAndBytesConfig from langchain_huggingface import HuggingFacePipeline from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline

Define Quantization

quantization_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype="bfloat16", bnb_4bit_use_double_quant=True )

Initialize LLM and Parameters

model_id = "microsoft/Phi-3-mini-4k-instruct" tokenizer = AutoTokenizer.from_pretrained("../Phi-3-mini-4k-instruct/") model = AutoModelForCausalLM.from_pretrained("../Phi-3-mini-4k-instruct/", quantization_config=quantization_config)

Initialize pipeline

pipe = pipeline( task="text-generation", model=model, tokenizer=tokenizer, max_new_tokens=1024, do_sample=False, repetition_penalty=1.03, )

Initialize HuggingFace Pipeline

llm = HuggingFacePipeline( pipeline=pipe, )

Successful Invocation

llm.invoke("What is HuggingFace?") (Output: "What is HuggingFace?\n\nHuggingFace is a company and an open-source community...")

Using Chat Model

from langchain_huggingface import ChatHuggingFace

llm_engine_hf = ChatHuggingFace(llm=llm) llm_engine_hf.invoke("Hugging Face is")

Error encountered

""" Error Message LocalTokenNotFoundError: Token is required (token=True), but no token found. You need to provide a token or be logged in to Hugging Face with huggingface-cli login or huggingface_hub.login. See https://huggingface.co/settings/tokens """

Error Message and Stack Trace (if applicable)

Error Message LocalTokenNotFoundError: Token is required (token=True), but no token found. You need to provide a token or be logged in to Hugging Face with huggingface-cli login or huggingface_hub.login. See https://huggingface.co/settings/tokens

System Info

linux - ubuntu 20.04.06 LTS Python - 3.8 langchain-huggingface==0.0.3

[CrasCris]

Did you set the huggingface token in enviroment variable?

[AnandUgale]

Hi @CrasCris,

I have downloaded the Phi-3 model locally and want to access it on a machine that doesn't allow connections to third-party API tokens like Hugging Face. Let's assume I want to access the model on a machine without an internet connection. That's why I haven’t been using the Hugging Face API key. What would you suggest in this situation?

[CrasCris]

You have to load all the model in the machine and load , i only had made with TrOCR model like this processor = TrOCRProcessor.from_pretrained('microsoft/trocr-small-handwritten', cache_dir='models/trocr') or this way OCR_model = VisionEncoderDecoderModel.from_pretrained('models/trocr')