[Bug]: service context not using the suggest LLM?

[hemanth]

Bug Description

Trying to use llama_index.llms.palm as llm for service_context and it still seems to be connecting to OpenAI?

The above exception was the direct cause of the following exception:

LocalProtocolError                        Traceback (most recent call last)
LocalProtocolError: Illegal header value b'Bearer '

The above exception was the direct cause of the following exception:

APIConnectionError                        Traceback (most recent call last)
/usr/local/lib/python3.10/dist-packages/openai/_base_client.py in _request(self, cast_to, options, remaining_retries, stream, stream_cls)
    895                     stream_cls=stream_cls,
    896                 )
--> 897             raise APIConnectionError(request=request) from err
    898 
    899         return self._process_response(

APIConnectionError: Connection error.

Version

0.8.66

Steps to Reproduce

import os

from IPython.display import display
from llama_index import ServiceContext, VectorStoreIndex, download_loader
from llama_index.llms.palm import PaLM

llm = PaLM(os.environ['API_KEY'])

SimpleWebPageReader = download_loader("SimpleWebPageReader")

loader = SimpleWebPageReader()
documents = loader.load_data(urls=['https://v8.dev/blog/wasm-gc-porting'])

service_context = ServiceContext.from_defaults(llm=llm,
                                               chunk_size=800,
                                               chunk_overlap=20)

index = VectorStoreIndex.from_documents(documents,
                                        service_context=service_context)

engine = index.as_query_engine()

response = engine.query("Summarize the website in 5 points")
display((f"{response}"))

Relevant Logs/Tracbacks

No response

[dosubot[bot]]

🤖

Hello @hemanth,

Good to see you again! I hope you're doing well. Thank you for providing a detailed description of the issue you're facing. I'm currently reviewing it and will get back to you with a proper solution shortly.

[dosubot[bot]]

🤖

Thank you for providing the detailed information. It seems like you're encountering an issue where the ServiceContext is still connecting to OpenAI even though you're trying to use llama_index.llms.palm as llm.

Based on the information provided and the similar issues found in the LlamaIndex repository, it appears that the PaLM class in llama_index.llms.palm is designed to work with Google's AI platform, not OpenAI. This is why the ServiceContext is still connecting to OpenAI.

Here is the relevant code from the PaLM class that shows this:

try:
    import google.generativeai as palm
except ImportError:
    raise ValueError(
        "PaLM is not installed. "
        "Please install it with `pip install google-generativeai`."
    )
api_key = api_key or os.environ.get("PALM_API_KEY")
palm.configure(api_key=api_key)

models = palm.list_models()

In the complete method, it uses palm.generate_text to generate the text.

import google.generativeai as palm

completion = palm.generate_text(
    model=self.model_name,
    prompt=prompt,
    **kwargs,
)

You can find this code in the llama_index/llms/palm.py file.

One of the similar issues suggests that you need to provide the embedding model as well if you want everything to run locally. You can run any embedding model from huggingface by following this guide: https://gpt-index.readthedocs.io/en/latest/how_to/customization/embeddings.html#custom-embeddings

I hope this helps! If you have any other questions or need further clarification, please let me know.

Sources

• [Bug]: Setting Palm LLM Abstraction to ServiceContext still looks for OpenAI API Key
• OpenAI LLM being used even when using ServiceContext with a custom LLM
• llama_index/llms/palm.py
• llama_index/indices/service_context.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[logan-markewich]

There are two models in llama-index, an LLM and an embedding model.

You've only set an LLM, so the emebddings are still using the default of openai

a quick fix is setting something like embed_model="local:BAAI/bge-base-en-v1.5" in the service context

Or, check out all our embedding models here https://docs.llamaindex.ai/en/stable/module_guides/models/embeddings.html#modules

[hemanth]

I imported from llama_index.embeddings import GooglePaLMEmbedding and used that in the service context, but seeing the below error @logan-markewich

[/usr/local/lib/python3.10/dist-packages/llama_index/indices/query/embedding_utils.py](https://localhost:8080/#) in get_top_k_embeddings(query_embedding, embeddings, similarity_fn, similarity_top_k, embedding_ids, similarity_cutoff)
     31         similarity = similarity_fn(query_embedding_np, emb)
     32         if similarity_cutoff is None or similarity > similarity_cutoff:
---> 33             heapq.heappush(similarity_heap, (similarity, embedding_ids[i]))
     34             if similarity_top_k and (len(similarity_heap) > similarity_top_k):
     35                 heapq.heappop(similarity_heap)

ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

This looks like a 🐛

[hemanth]

@logan-markewich anything on ^ :)

[ravi03071991]

@hemanth looking into it.

[ravi03071991]

@hemanth this PR - https://github.com/run-llama/llama_index/pull/8936 fixes the issue.

[hemanth]

Thanks for the fix @ravi03071991 [I was there at the same file :D]