[Bug]: LlamaIndex LiteLLMEmbedding is not passing api_key in its call to Voyage AI

[shashvatsinha]

What happened?

When using LlamaIndex, if I do the following:

embedding_model = "voyage/voyage-code-2"
Settings.embed_model = LiteLLMEmbedding(model_name=embedding_model,

Then api_key is not picked up from VOYAGE_API_KEY, and LiteLLM / Voyage throws a 401 error:

  File "/usr/local/lib/python3.11/site-packages/openai/_base_client.py", line 1041, in _request
    raise self._make_status_error_from_response(err.response) from None
openai.AuthenticationError: Error code: 401 - {'detail': 'Provided API key is invalid.'}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/litellm/main.py", line 3502, in embedding
    response = openai_chat_completions.embedding(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/litellm/llms/openai.py", line 1166, in embedding
    raise OpenAIError(status_code=e.status_code, message=str(e))
litellm.llms.openai.OpenAIError: Error code: 401 - {'detail': 'Provided API key is invalid.'}

But if I do:

embedding_model = "voyage-code-2"
Settings.embed_model = VoyageEmbedding(model_name = "voyage-code-2")

then it correctly picks up the API key from VOYAGE_API_KEY and the call works.

This is probably a second bug and should be filed as so, but I'm guessing that it would be the first question anyone asks, so leaving it here: If I provide the API key like so:

embedding_model = "voyage/voyage-code-2"
Settings.embed_model = LiteLLMEmbedding(model_name=embedding_model, api_key='pa-v5u...')

Then I get the following error:

LiteLLM:ERROR: cost_calculator.py:673 - litellm.cost_calculator.py::response_cost_calculator - Exception occurred - unsupported operand type(s) for *: 'float' and 'NoneType'/nTraceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/litellm/cost_calculator.py", line 660, in response_cost_calculator
    response_cost = completion_cost(
                    ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/litellm/cost_calculator.py", line 601, in completion_cost
    raise e
  File "/usr/local/lib/python3.11/site-packages/litellm/cost_calculator.py", line 583, in completion_cost
    ) = cost_per_token(
        ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/litellm/cost_calculator.py", line 173, in cost_per_token
    model_cost_ref[model]["input_cost_per_token"] * prompt_tokens
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~
TypeError: unsupported operand type(s) for *: 'float' and 'NoneType'

Relevant log output

No response

Twitter / LinkedIn details

https://www.linkedin.com/in/shashvat/

[krrishdholakia]

@shashvatsinha

LiteLLM:ERROR: cost_calculator.py:673

Is this causing the call to fail? or just a warning?

[shashvatsinha]

I rechecked. Here's how I'm using it with Qdrant and LlamaIndex:

try:
    vector_store = QdrantVectorStore(client=client, collection_name="github_prs")
    storage_context = StorageContext.from_defaults(vector_store=vector_store)
    index = VectorStoreIndex.from_documents(
        documents,
        storage_context=storage_context,
    )
except Exception as e:
    print (f"Caught exception {e}")

It works, it doesn't raise an exception.

That said, this is the secondary issue - primary issue is still that the LiteLLMEmbedding() call requires an api_key parameter to be passed for Voyage, otherwise it fails. If I need to put in if-else code for different providers when calling LiteLLMEmbedding, then I might as well implement my own factory method and use VoyageEmbedding, OpenAIEmbedding and BedrockEmbedding directly.

[krrishdholakia]

@shashvatsinha how're you setting the api key param for voyage?

[shashvatsinha]

.env file and VOYAGE_API_KEY Same as all the others (OPENAI_API_KEY etc).

[shashvatsinha]

i.e. with any other embedding model, I don't have to set api_key = os.getenv('PROVIDER_KEY') They all pick up their PROVIDER_KEY environment variable when used with LiteLLMEmbed.

But when using a voyage model, it doesn't work, and I have to specify api_key = os.getenv('VOYAGE_API_KEY').

[elkotito]

About the second bug from cost calculator, it is because prompt_tokens in the response from Voyage API is None.

CreateEmbeddingResponse(data=[Embedding(embedding=[], index=0, object='embedding')], model='voyage-code-2', object='list', usage=Usage(prompt_tokens=None, total_tokens=7))