Misleading HTTP 200 Status Code on Network Connection Lost Error via Portkey

[Nathanpamart]

What Happened?

Issue Summary:

When making a request to OpenAI's API through Portkey, I received an HTTP 200 status code despite the response body indicating a "Network connection lost." error. This behavior is misleading and prevents automatic retry mechanisms from triggering, as they typically rely on non-200 status codes to indicate failure.

{ "status": 200, "headers": { "content-type": "application/json" }, "body": { "html-message": "Network connection lost." }, "responseTime": 108, "lastUsedOptionJsonPath": "config" }

Steps to Reproduce: Send a POST request to https://api.openai.com/v1/chat/completions via Portkey with the following headers and body:

Headers: (List only the relevant headers, especially those related to Portkey and retry logic) Content-Type: application/json X-Portkey-Retry: {"attempts": 5} (Include any other headers that are crucial to replicating the issue) Body: (Simplify to illustrate the request without exposing sensitive information)

see the request json below

{ "url": "https://api.openai.com/v1/chat/completions", "method": "POST", "headers": { "accept": "application/json", "accept-encoding": "gzip, br", "cf-visitor": "{\"scheme\":\"https\"}", "connection": "Keep-Alive", "content-length": "7325", "content-type": "application/json", "host": "api.portkey.ai", "user-agent": "OpenAI/Python 1.14.2", "x-forwarded-proto": "https", "x-portkey-retry": "{\"attempts\": 5}", "x-real-ip": "213.27.249.134", "x-stainless-arch": "x64", "x-stainless-async": "false", "x-stainless-lang": "python", "x-stainless-os": "MacOS", "x-stainless-package-version": "1.14.2", "x-stainless-runtime": "CPython", "x-stainless-runtime-version": "3.10.0" }, "body": { "messages": [ { "role": "system", "content": "<my_system_prompt>" }, { "role": "user", "content": "<my content>" } ], "model": "gpt-4-1106-preview", "n": 1, "stream": false, "temperature": 0 }, "portkeyHeaders": { "x-portkey-api-key": "rD********Hg=", "x-portkey-config": "{\"cache\":{\"mode\":\"simple\",\"maxAge\":864000}}", "x-portkey-provider": "openai", "x-portkey-trace-id": "f2fc3222-e3e5-4081-9d3f-f92cf2d312b0" } }

What Should Have Happened?

In the event of a network connection issue or similar error, I expect to receive an HTTP status code indicative of an error (e.g., 4xx or 5xx). This would allow client-side logic, such as automatic retries, to function correctly based on the status code.

Actual Behavior: Received an HTTP 200 status code, suggesting a successful response, despite the body containing an error message indicating a lost network connection.

Suggested Fix: Adjust the API or Portkey's error handling logic to return a more appropriate HTTP status code when an error occurs, especially for network-related issues.

Relevant Code Snippet

I trigger this request from a python script using langchain, as below

from langchain_community.chat_models import ChatOpenAI from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders

`headers = createHeaders( provider="openai", api_key=PORTKEY_API_KEY,

Cache Options

            config={
                "cache": {"mode": "simple", "maxAge": 864000}
            },  # in seconds (10 days)
            retry={"attempts": 5},
            # Metadata
            environment="test",
            user="<user>",
            organisation="<org>",
            prompt="test",
        )`

after initializing an LLM object

llm = ChatOpenAI( model_name=model_name, temperature=0, base_url=PORTKEY_GATEWAY_URL, default_headers=headers, )

Your Twitter/LinkedIn

https://www.linkedin.com/in/nathanpamart/

[rootAvish]

I faced this bug too. An additional side effect of this bug is Portkey's cache caches the error response (assume because of the 200 response code). You don't get a valid response till you force cache refresh.

[vrushankportkey]

Hi @Nathanpamart @rootAvish sorry for missing out on this! Tagging @narengogi who can take a look at this

[narengogi]

@rootAvish @Nathanpamart This issue has since been fixed, the gateway now handles http failures correctly