Receiving malformed responses from the providers (oai, azure-oai, ...)

bricks-cloud / BricksLLM

🔒 Enterprise-grade API gateway that helps you monitor and impose cost or rate limits per API key. Get fine-grained access control and monitoring per user, application, or environment. Supports OpenAI, Azure OpenAI, Anthropic, vLLM, and open-source LLMs.

https://trybricks.ai/

MIT License

901 stars 61 forks source link

Receiving malformed responses from the providers (oai, azure-oai, ...) #73

Closed minhoryang closed 4 months ago

minhoryang commented 4 months ago

Error logs like these:

error when parsing run steps result | invalid character "x1f' looking for beginning of value...
error when recording an event          {"error": "pq: invalid byte sequence for encoding \ "UTF8\": 0x83...

error when recording an event logs were generated when should_log_response=True in a key configuration.

minhoryang commented 4 months ago

possibly related with https://community.openai.com/t/gpt-4-1106-preview-is-not-generating-utf-8/482839/6, but it reports invalid char at the beginning of value.

minhoryang commented 4 months ago

In our usecase, we placed the load balancer in front of bricksllm instances, and LB injected accept-encoding with br or gzip automatically for the performance boost. With this PR, we don't send accept-encoding to providers.