[Bug]: Setting Palm LLM Abstraction to ServiceContext still looks for OpenAI API Key

[cmagorian]

Bug Description

Traceback (most recent call last):
  File "C:\Users\Crispy\PycharmProjects\chatbottest\main.py", line 13, in <module>
    service_context = ServiceContext.from_defaults(llm=model)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Crispy\PycharmProjects\chatbottest\venv\Lib\site-packages\llama_index\indices\service_context.py", line 163, in from_defaults
    embed_model = embed_model or OpenAIEmbedding()
                                 ^^^^^^^^^^^^^^^^^
  File "C:\Users\Crispy\PycharmProjects\chatbottest\venv\Lib\site-packages\llama_index\embeddings\openai.py", line 248, in __init__
    validate_openai_api_key(
  File "C:\Users\Crispy\PycharmProjects\chatbottest\venv\Lib\site-packages\llama_index\llms\openai_utils.py", line 268, in validate_openai_api_key
    raise ValueError(MISSING_API_KEY_ERROR_MESSAGE)
ValueError: No API key found for OpenAI.
Please set either the OPENAI_API_KEY environment variable or openai.api_key prior to initialization.
API keys can be found or created at https://platform.openai.com/account/api-keys

Coming from:

import google.generativeai as palm
from llama_index import VectorStoreIndex, SimpleDirectoryReader, ServiceContext, set_global_service_context
from llama_index.llms.palm import PaLM
from llama_index.llms.openai import OpenAI
import gradio as gr

palm.configure(api_key=<api-key>')

model = PaLM('<api-key>')

documents = SimpleDirectoryReader('docs').load_data()
service_context = ServiceContext.from_defaults(llm=model) # Bug comes from right here, it looks like `from_defaults()` needs `embed_model` to be something or defaults to OpenAI
set_global_service_context(service_context)
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()

def chatbot(input_text):
    response = query_engine.query(input_text)
    return response

iface = gr.Interface(fn=chatbot, inputs=gr.inputs.Textbox(lines=7, label="Talk to me..."), outputs="text", title="Creepy hand")

iface.launch(share=True)

Version

0.7.16

Steps to Reproduce

Setup LLM as PaLM abstraction and pass to the service_context globally.

Relevant Logs/Tracbacks

Traceback (most recent call last):
  File "C:\Users\Crispy\PycharmProjects\chatbottest\main.py", line 13, in <module>
    service_context = ServiceContext.from_defaults(llm=model)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Crispy\PycharmProjects\chatbottest\venv\Lib\site-packages\llama_index\indices\service_context.py", line 163, in from_defaults
    embed_model = embed_model or OpenAIEmbedding()
                                 ^^^^^^^^^^^^^^^^^
  File "C:\Users\Crispy\PycharmProjects\chatbottest\venv\Lib\site-packages\llama_index\embeddings\openai.py", line 248, in __init__
    validate_openai_api_key(
  File "C:\Users\Crispy\PycharmProjects\chatbottest\venv\Lib\site-packages\llama_index\llms\openai_utils.py", line 268, in validate_openai_api_key
    raise ValueError(MISSING_API_KEY_ERROR_MESSAGE)
ValueError: No API key found for OpenAI.
Please set either the OPENAI_API_KEY environment variable or openai.api_key prior to initialization.
API keys can be found or created at https://platform.openai.com/account/api-keys

[logan-markewich]

There are two models in llamaindex, the LLM and the embed model

You have only set the LLM, so the embed model is defaulting to openai.

I don't think palm has an embedding model yet? At least not one supported yet. You can read more about setting embed models here https://gpt-index.readthedocs.io/en/latest/core_modules/model_modules/embeddings/usage_pattern.html#embedding-model-integrations

[cmagorian]

Hey @logan-markewich thanks for getting back on this.

I think PaLM API does have an embedding API: https://developers.generativeai.google/tutorials/embeddings_quickstart

I'm happy to extend it. Is there a process for submitting PRs against llama_index?

Thanks! Chris

[logan-markewich]

@cmagorian ya, basically just extend the base class, run the linting from the makefile, and off you go.

I'd you submit a PR, we usually try to get it merged within a day, unless it's a giant change (which this shouldn't be!)

[cmagorian]

@logan-markewich I got pretty far with implementing the PaLMEmbeddings class for the query_engine. But ultimately ran into a problem where the context added before synthesis made PaLM API believe that it was a language it didn't work with yet, likely because data was converted to NaN.

Not entirely sure what the best way of changing that behavior is.

[logan-markewich]

Huh, that's super weird. So the the api returns NaN for languages it doesn't support? I wonder if the api has a way to hard-coded the language 👀

[cmagorian]

Yeah checked that out. NaN isnt being returned by the PaLM embedder API, it's generated somewhere in llama_index when reading empty items from a CSV document.

[logan-markewich]

Oh weird... I feel like the only place that would happen is with a pandas query engine? If you aren't using that, feel free to send a file/code over that can reproduce that!

[hpsc3]

@cmagorian were you able to get it to work. I am facing a similar issue probably because of the same reason: https://github.com/jerryjliu/llama_index/issues/7898