search

BerriAI / litellm

Python SDK, Proxy Server to call 100+ LLM APIs using the OpenAI format - [Bedrock, Azure, OpenAI, VertexAI, Cohere, Anthropic, Sagemaker, HuggingFace, Replicate, Groq]
https://docs.litellm.ai/docs/
Other
12.17k stars 1.41k forks source link

[Bug]: Cannot add callbacks due to file readonly restrictions #4940

Open sumitdas66 opened 1 month ago

sumitdas66 commented 1 month ago

What happened?

I deployed LiteLLM on AWS EKS cluster and tried to add callbacks. However, it comes back with the error:

{"error":{"message":"Authentication Error, [Errno 30] Read-only file system: '/etc/litellm/config.yaml'","type":"auth_error","param":"None","code":400}}

image

Seems like the UI calls an update api ["POST /config/update HTTP/1.1"] to modify the config.yaml which is stored in volumeMount litellm-config with mountPath /etc/litellm/ and this volume is readonly.

Is there a work-around for this, since right now the values.yaml does not allow us to change the readonly as false for the above said mount.

Note:

  1. The operation is being done by the Admin user.
  2. I have set readonlyrootfilesystem as false but that too does not work.

Relevant log output

litellm.proxy.proxy_server.update_config(): Exception occured - [Errno 30] Read-only file system: '/etc/litellm/config.yaml'
INFO:     172.16.48.0:55048 - "POST /config/update HTTP/1.1" 400 Bad Request

Twitter / LinkedIn details

No response

krrishdholakia commented 1 month ago

i think what we can do is support writing it to the db instead of the file.

Can we do a 10min call for this? I'd like to make sure i can repro this flow and solve the problem well

https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

sumitdas66 commented 1 month ago

Hey @krrishdholakia, would be happy to help. I am from IST timezone. What would be a suitable time for you to connect?

sumitdas66 commented 1 month ago

I may be completely wrong, but here is what I think is happening.

  1. Everything under ".Values.proxy_config" including the "litellm_settings" where we are supposed to add the callbacks for langfuse is converted into a config map by "/templates/configmap-litellm.yaml". https://github.com/BerriAI/litellm/blob/main/deploy/charts/litellm-helm/templates/configmap-litellm.yaml

  2. The "/templates/deployment.yaml" then mounts the configmap to the litellm config file " /etc/litellm/config.yaml". https://github.com/BerriAI/litellm/blob/main/deploy/charts/litellm-helm/templates/deployment.yaml#L211

  3. According to the Kubernetes documentation, ConfigMap is always mounted as readonly. https://kubernetes.io/docs/concepts/storage/volumes/#configmap image

  4. This might be a reason why the update_config() method fails with the above error. https://github.com/BerriAI/litellm/blob/main/litellm/proxy/proxy_server.py#L8746

Note: I also tried adding just environment variables using the update config api [POST /config/update] from swagger, still it gives me the same error. Theoretically it should just add/update environment variables that are part of the payload and not look for the config.yaml file at all.