search

microsoft / azurechat

🤖 💼 Azure Chat Solution Accelerator powered by Azure Open AI Service
MIT License
1.14k stars 954 forks source link

Specify a "Content-Type" header in response streams so that Cloudflar… #305

Open ShaunMaher opened 5 months ago

ShaunMaher commented 5 months ago

By default, the upstream "ai" module (https://github.com/vercel/ai/) is returning the response from the Azure OpenAI API with a "text/plain" Content-Type. If your azurechat instance is hosted behind a cloudflared tunnel (maybe just Cloudflare in general, I haven't tested this), the response will be buffered by cloudflare(d) and returned all at once.

Other than not being the expected experience, this can lead to timeouts (cloudflare default = 100s) if the response is long or slow to generate. If the response is streamed, there should be enough data flowing to avoid this timeout.

Cloudflared looks for the Content-Type response header and if it is "text/event-stream" it will know not to buffer the response but to send it immediately to the requestor. Reference.

ShaunMaher commented 5 months ago

@microsoft-github-policy-service agree [company="Orro"]

ShaunMaher commented 5 months ago

@microsoft-github-policy-service agree company="Orro"