[Bug]: Query Engine expects an Open AI API Key even when using local embeddings and having set llm as None

[sherwin684]

Bug Description

I am trying to create a query_engine as follows :

index = VectorStoreIndex.from_documents(documents=documents)
query_engine = index.as_query_engine(llm=None, embed_model="local")

And I have used a custom HuggingFaceEmbedding model.

from llama_index.embeddings.huggingface import HuggingFaceEmbedding

embed_model = HuggingFaceEmbedding(
model_name="BAAI/bge-small-en-v1.5"
)

But I get an ValueError error when running, saying that I have to specify an Open AI API Key eventhough I am using a local llm and a local embedding model.

query_engine = index.as_query_engine(llm=None, embed_model="local")

As mentioned in previous issues https://github.com/run-llama/llama_index/issues/852, a ServiceContext cannot be specified as it it has been deprecated in the current version of llama_index.

Version

llama-index==0.10.20

Steps to Reproduce

Define a local HuggingFaceEmbedding class and specify the index and query engine as follows :

from llama_index.embeddings.huggingface import HuggingFaceEmbedding

embed_model = HuggingFaceEmbedding(
model_name="BAAI/bge-small-en-v1.5"
)

index = VectorStoreIndex.from_documents(documents=documents)
query_engine = index.as_query_engine(llm=None, embed_model="local")

An error arises from the line

query_engine = index.as_query_engine(llm=None, embed_model="local")

Relevant Logs/Tracbacks

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
File ~/my_data/01_python/llm-evaluation/llm-eval-env/lib/python3.10/site-packages/llama_index/core/embeddings/utils.py:59, in resolve_embed_model(embed_model, callback_manager)
     58     embed_model = OpenAIEmbedding()
---> 59     validate_openai_api_key(embed_model.api_key)
     60 except ImportError:

File ~/my_data/01_python/llm-evaluation/llm-eval-env/lib/python3.10/site-packages/llama_index/embeddings/openai/utils.py:104, in validate_openai_api_key(api_key)
    103 if not openai_api_key:
--> 104     raise ValueError(MISSING_API_KEY_ERROR_MESSAGE)

ValueError: No API key found for OpenAI.
Please set either the OPENAI_API_KEY environment variable or openai.api_key prior to initialization.
API keys can be found or created at https://platform.openai.com/account/api-keys

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
Cell In[2], line 1
----> 1 query_engine, rag_dataset = create_query_engine_rag_dataset(
      2     "./data/paul_graham"
      3 )

Cell In[1], line 24
...
API keys can be found or created at https://platform.openai.com/account/api-keys

Consider using embed_model='local'.
Visit our documentation for more embedding options: https://docs.llamaindex.ai/en/stable/module_guides/models/embeddings.html#modules
******

[dosubot[bot]]

Hey there, @sherwin684! Great to see you diving deep again with LlamaIndex. Hope you've been well! 🦙✨

Thank you for reporting this issue. It seems like you're encountering a ValueError when trying to use a local language model and a local embedding model with the as_query_engine method of the VectorStoreIndex class. This error is raised because the LlamaIndex Query Engine is designed to interact with OpenAI's API for generating completions, chat responses, and streaming completions. These operations inherently depend on OpenAI's services, which require authentication via an API key.

However, based on the context provided, there are a few potential solutions that have been suggested in similar issues:

1. Persist the index and then load it with the embed_model specified and llm set to None. This should allow the use of a local model without requiring an OpenAI API key. Here's the code snippet for reference:

index.storage_context.persist(persist_dir="./storage")
loaded_index = load_index_from_storage(StorageContext.from_defaults(persist_dir="./storage"), embed_model=embed_model, llm=None)

(Source: Issue #10051)

2. Set the OPENAI_API_KEY environment variable before importing the VectorStoreIndex. You can do this by adding the following line to your Python script before any imports:

import os
os.environ["OPENAI_API_KEY"] = 'your-api-key'

Make sure to replace 'your-api-key' with your actual OpenAI API key. (Source: Issue #11900)

3. Set llm=None when creating the ServiceContext to disable the Language Model (LLM) entirely if you are not intending to use OpenAI's models. Here's the code snippet for reference:

service_context = ServiceContext.from_defaults(
    chunk_size=2048,
    llm=None,
    embed_model=embed_model
)

And when you load the index, pass in the service context:

index = load_index_from_storage(storage_context, service_context=service_context)

(Source: Issue #10330)

Please try these solutions and let us know if any of them resolve your issue. If not, we'll continue to investigate and find a solution for you.

Sources

• [Question]: When customize the LLM, using VectorStoreIndex, it reports error
• [Question]: local HuggingFaceEmbedding returns "No API key found for OpenAI."
• [Bug]: ValueError: ** Could not load OpenAI embedding model. If you intended to use OpenAI, please check your OPENAI_API_KEY. Original error: No API key found for OpenAI. Please set either the OPENAI_API_KEY environment variable or openai.api_key prior to initialization. API keys can be found or created at https://platform.openai.com/account/api-keys Consider using embed_model='local'. Visit our documentation for more embedding options: https://docs.llamaindex.ai/en/stable/module_guides/models/embeddings.html#modules **
• [Question]: LLama-Index asking for OpenAI key while loading an index even though using a custom embedding model?
• [Question]: Why Running Llama 2 require OpenAI API Key
• llama-index-legacy/tests/llms/test_openai.py
• llama-index-core/llama_index/core/indices/vector_store/base.py

#### About Dosu

This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant. If you want to continue the conversation, start your reply with @dosu-bot.

[logan-markewich]

@sherwin684 you aren't passing in the embed model to the initial index creation

I would just do this

from llama_index.core.llms import MockLLM

index = VectorStoreIndex.from_documents(documents=documents, embed_model=embed_model)
query_engine = index.as_query_engine(llm=MockLLM())

[sherwin684]

Yes it worked thanks. It would be great if it is documented as mentioend above. I never found about MockLLM in the documentation regarding using local embedding models.

[namedgraph]

@logan-markewich if I'm using Azure and not OpenAI directly, do I pass AzureOpenAI as llm?

[logan-markewich]

@namedgraph yup 👍 (assuming AzureOpenAI was imported from llama-index of course)

[manishb27]

llm=MockLLM()

This worked for me too. Thanks