[Bug]: prisma spendlog update error when completion_token is none

[stronk7]

What happened?

After adding a couple of ollama embeddings to the configuration to start playing with them, some errors immediately appear in the logs, after the 1st request, recurring, every few (10?) seconds.

I'm running a (just updated) ghcr.io/berriai/litellm:main-latest docker container.

And this is the definition of the 2 embeddings in config.yaml:

  - model_name: "nomic-embed-text"
    litellm_params:
      model: "ollama/nomic-embed-text"
      api_base: "http://host.docker.internal:11434"
  - model_name: "all-minilm"
    litellm_params:
      model: "ollama/all-minilm"
      api_base: "http://host.docker.internal:11434

(that are using a local/host ollama instance, I imagine that it's not important)

The endpoints seem to work ok:

curl --location 'http://0.0.0.0:4000/v1/embeddings' \
--header 'Content-Type: application/json' \
--header 'Authorization: ••••••' \
--data '{
    "model": "all-minilm",
    "input": "this is a test"
}'

And I get (correct):

{
    "model": "all-minilm",
    "data": [
        {
            "object": "embedding",
            "index": 0,
            "embedding": [
                0.16015781462192535,
                0.29886701703071594,
                ...
                ...
                0.5186416506767273,
                -0.35049015283584595
            ]
        }
    ],
    "object": "list",
    "usage": {
        "prompt_tokens": 4,
        "completion_tokens": null,
        "total_tokens": 4
    }
}

Note that I've not tried with embeddings from other providers, only with ollama.

Relevant log output

Job "update_spend (trigger: interval[0:00:12], next run at: 2024-08-07 17:54:54 UTC)" raised an exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/apscheduler/executors/base_py3.py", line 30, in run_coroutine_job
    retval = await job.func(*job.args, **job.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/litellm/proxy/utils.py", line 2777, in update_spend
    raise e
  File "/usr/local/lib/python3.11/site-packages/litellm/proxy/utils.py", line 2738, in update_spend
    await prisma_client.db.litellm_spendlogs.create_many(
  File "/usr/local/lib/python3.11/site-packages/prisma/actions.py", line 9489, in create_many
    resp = await self._client._execute(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prisma/client.py", line 525, in _execute
    return await self._engine.query(builder.build(), tx_id=self._tx_id)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prisma/engine/query.py", line 244, in query
    return await self.request(
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prisma/engine/http.py", line 141, in request
    return utils.handle_response_errors(resp, errors_data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prisma/engine/utils.py", line 180, in handle_response_errors
    raise prisma_errors.MissingRequiredValueError(error)
prisma.errors.MissingRequiredValueError: Unable to match input value to any allowed input type for the field. Parse errors: [Invalid argument type. `data` should be of any of the following types: `LiteLLM_SpendLogsCreateManyInput`, `data.completion_tokens`: A value is required but not set]

Twitter / LinkedIn details

No response

[krrishdholakia]

@stronk7 these look like prisma errors - raise prisma_errors.MissingRequiredValueError(error) prisma.errors.MissingRequiredValueError: Unable to match input value to any allowed input type for the field. Parse errors: [Invalid argument type. data should be of any of the following types: LiteLLM_SpendLogsCreateManyInput, data.completion_tokens: A value is required but not set]

i'll investigate this

[stronk7]

Hi @krrishdholakia,

for the records I've tried here with openai text-embedding-3-small and haven't been able to reproduce the problem, so it seems that it's ollama-related only.

One little detail that called my attention is that the openai embeddings request returns:

    "usage": {
        "prompt_tokens": 4,
        "completion_tokens": 0,
        "total_tokens": 4
    }

but ollama, returns null instead:

    "usage": {
        "prompt_tokens": 4,
        "completion_tokens": null,
        "total_tokens": 4
    }

I don't know if that can be the cause of the prisma problem or no, but worth sharing.

Ciao :-)

[krrishdholakia]

Looks like this is caused. by completion_tokens being 'null' vs the expected 'int' type