[Bug]: unable to connect to the LiteLLM Proxy as the ChatOpenAI call leads to “SSLCertVerificationError” exception

[krrishdholakia]

What happened?

We have to run LiteLLM proxy as HTTPS/SSL endpoint instead of HTTP. Currently we run as HTTP endpoint. We have deployed EKS container in AWS with LiteLLM proxy with AWS ALB to give it that domain address. However we are hitting roadblock where we are unable to connect to the LiteLLM Proxy as the ChatOpenAI call leads to “SSLCertVerificationError” exception. We have tried to add certificate to ensure it is trusted source, but that doesn’t work and also I know one can turn off SSL verify flag such as in Python requests but just not clear where to do in the chain based on the following code and corresponding stacktrace.

Relevant log output

No response

Twitter / LinkedIn details

No response

[ishaan-jaff]

Working on this:

• our railway app is also an https endpoint, but I don't see this error there

[krrishdholakia]

I believe this is an error raised by openai when the referrer is unverified. cc: @sestinj I believe you dealt with this a while ago

How did you solve this?

[ishaan-jaff]

relevant issue: https://community.openai.com/t/how-can-i-disable-ssl-verification-when-using-openai-api-in-python/110837/4

I'm looking for a way to repro this issue, have not found one so far

[ishaan-jaff]

Potential fix from another user:

The ssl verification issue happened with me since in our organization we have Zscalar added and it did had the certificate available for openai url.

I was able to solve this issue by manually downloading the certificate and adding it to the 'REQUESTS_CA_BUNDLE' environment variable in my code

[sestinj]

verify_ssl=False I know was one of our solutions, but mostly would let people add their own certificates manually through ca_bundle_path. Not sure this is relevant here unfortunately

[ishaan-jaff]

this is fixed for anyone looking to do this with litellm proxy you can set ssl_verify=False on the OpeAI python client

import openai, httpx
client = openai.OpenAI(
    api_key="anything",
    base_url="http://0.0.0.0:8000", # litellm proxy url 
    http_client=httpx.Client(verify=False)
)

# request sent to model set on litellm proxy, `litellm --model`
response = client.chat.completions.create(
    model="azure/chatgpt-v-2",
    messages = [
        {
            "role": "user",
            "content": "this is a test request, write a short poem"
        }
    ],
)

print(response)