Linked backends: Streaming does not stream; it arrives in a single payload

[johnnyreilly]

Describe the bug

Imagine an Azure Static Web App with a linked backend. The linked backend is an Azure Function App, but based upon our investigations; the issue does not appear to be function app related.

To Reproduce

Deploy a Static Web App and a Function App which is a linked backend to the SWA. The backend contains this streaming function named GetChatCompletionsStream:

using System;
using System.Collections.Generic;
using System.IO;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.Http;
using Microsoft.Azure.WebJobs.Extensions.OpenApi.Core.Attributes;
using Microsoft.AspNetCore.Http;
using Microsoft.Extensions.Logging;
using System.Net;
using System.Text.Json;
using Azure;
using Azure.AI.OpenAI;
using System.Linq;

namespace ZebraGptFunctionApp.Functions;

public record CompletePromptParameters(
    string prompt,
    double temperature,
    int maxTokens,
    string deploymentName,
    bool canRecord
);

public class OpenAiFunction
{
    private readonly AppSettings _appSettings;
    private readonly OpenAIClient _openAIClient;
    private readonly ILogger<OpenAiFunction> _log;

    public OpenAiFunction(
        AppSettings appSettings,
        ILogger<OpenAiFunction> log,
        OpenAIClient openAIClient
    )
    {
        _appSettings = appSettings;
        _log = log;
        _openAIClient = openAIClient;
    }

    public record ChatCompletionParameters(
        string DeploymentName,
        float Temperature,
        int MaxTokens,
        List<ChatMessageParameter> Messages,
        bool CanRecord
    );

    public record ChatMessageParameter(string Role, string Content);

    [FunctionName(nameof(OpenAiFunction.GetChatCompletionsStream))]
    public async Task<IActionResult> GetChatCompletionsStream(
        [HttpTrigger(AuthorizationLevel.Anonymous)] HttpRequest req
    )
    {
        try
        {
            var chatCompletionParameters = await GetChatCompletionParameters(req);
            var chatOptions = GetChatCompletionsOptions(chatCompletionParameters);

            var response = req.HttpContext.Response;

            response.StatusCode = (int)HttpStatusCode.OK;
            // response.ContentType = "text/plain";
            response.ContentType = "text/event-stream";

            await using var sw = new StreamWriter(response.Body);

            var streamingChatCompletionsResponse = await _openAIClient.Client.GetChatCompletionsStreamingAsync(
                deploymentOrModelName: chatCompletionParameters.DeploymentName,
                chatOptions
            );

            using StreamingChatCompletions streamingChatCompletions = streamingChatCompletionsResponse.Value;

            await foreach (StreamingChatChoice choice in streamingChatCompletions.GetChoicesStreaming())
            {
                await foreach (ChatMessage message in choice.GetMessageStreaming())
                {
                    // THIS IS STREAMING
                    await sw.WriteAsync(message.Content);
                    await sw.FlushAsync();
                }
            }
        }
        catch (Exception ex)
        {
            _log.LogError(ex, "Problem getting chat completion");
            req.HttpContext.Response.StatusCode = (int)HttpStatusCode.InternalServerError;
        }

        // Required to avoid the host trying to add headers to the response
        return new EmptyResult();
    }

    private async static Task<ChatCompletionParameters> GetChatCompletionParameters(HttpRequest req)
    {
        var content = await new StreamReader(req.Body).ReadToEndAsync();

        var chatCompletionParameters =
            JsonSerializer.Deserialize<ChatCompletionParameters>(content, new JsonSerializerOptions
            {
                PropertyNamingPolicy = JsonNamingPolicy.CamelCase,
            })!;

        return chatCompletionParameters;
    }

    private static ChatCompletionsOptions GetChatCompletionsOptions(ChatCompletionParameters chatCompletionParameters)
    {
        var chatRoleMapping = new Dictionary<string, ChatRole>()
            {
                { "system", ChatRole.System }, { "user", ChatRole.User }, { "assistant", ChatRole.Assistant }
            };

        var chatOptions = new ChatCompletionsOptions()
        {
            Temperature = chatCompletionParameters.Temperature,
            MaxTokens = chatCompletionParameters.MaxTokens,
            NucleusSamplingFactor = (float)0.95,
            FrequencyPenalty = 0,
            PresencePenalty = 0,
        };

        // it has to be added like this because by design Messages is a readonly. https://github.com/Azure/azure-sdk-for-net/issues/35096
        chatCompletionParameters.Messages.ForEach(message => 
            chatOptions.Messages.Add(new ChatMessage(chatRoleMapping[message.Role], message.Content)));

        return chatOptions;
    }
}

Eagle peeps will note we're building an Open AI chat mechanism - but what's significant here is we stream text to the caller.

On the front end we have a TypeScript function that looks like this:

        try {
            if (!userInput.trim()) {
                return;
            }

            const userMessage: ChatMessage = {
                role: "user",
                content: userInput,
            };
            setMessages((prevMessages) => [...prevMessages, userMessage]);
            setUserInput("");
            setIsLoading(true);

            const response = await fetch("/api/GetChatCompletionsStream", {
                method: "POST",
                body: JSON.stringify({
                    deploymentName: "OpenAi-gpt-35-turbo",
                    maxTokens,
                    temperature,
                    messages: [...messages, userMessage],
                } as ChatCompletionParameters),
                headers: {
                    Accept: "text/event-stream",
                    "Content-Type": "application/json",
                },
            });

            if (!response.body) {
                return;
            }

            const reader = response.body.getReader();

            let reads = 0;
            let responseMessage = "";
            // eslint-disable-next-line no-constant-condition
            while (true) {
                const { done, value } = await reader.read();

                if (done) {
                    break;
                }

                const fragment = new TextDecoder("utf-8").decode(value);
                console.log(`fragment #${++reads}`, fragment);
                responseMessage += fragment;
                setLiveResponseMessage(responseMessage);
            }
            console.log(`done in ${reads} reads`);
            setMessages((prevMessages) => [
                ...prevMessages,
                { role: "assistant", content: responseMessage },
            ]);
            setIsLoading(false);
            setLiveResponseMessage("");
        } catch (error) {
            console.error("Error sending message:", error);
        }

Expected behavior

Running locally, this works as expected: streaming. Deployed - it does not; we get a single payload in our "stream".

Screenshots

Running locally (it working):

Deployed to Azure (it not working):

Notice the 1 reads - that's streaming not working.

Additional context We have tried directly accessing the function app from the front end of the static web app and confirmed streaming from the function app directly works. However, using this approach we lose all the benefits of linked backends.

[thomasgauvin]

Thanks for reporting @johnnyreilly, let me bring this up to eng team

[thomasgauvin]

I've brought this up to our eng team and it's on our backlog

[johnnyreilly]

Thanks @thomasgauvin!

[ChouchouCendre]

Same issue here, new stream.PassThrough() is working correctly locally but it's not working on the server. Everything arrives at the end.

@thomasgauvin any idea when it'll be fixed? Are we talking days, weeks?

[thomasgauvin]

We tried reproducing this but it seems we're able to get streams working (with a .NET 6 Azure Function). For your linked backend, are you using in-process or isolated for the .NET runtime? Which .NET version?

You've mentioned that your Azure Functions can be validated to stream responses back. If that is the case, I presume you have .NET 6 in-process Azure Functions?

[johnnyreilly]

Hi @thomasgauvin,

We're using .NET 6 in process Azure Functions. I'd be happy to jump on a call and demo if you'd like. To be clear: everything arrives data wise, but it does it in a single payload.

Worth noting: running locally streaming works just fine. It is when it is deployed to Azure we have the issue. Both our Static Web App and our Azure Function are deployed to the West Europe region

This will be of limited use since it references out private Bicep registry, but our infrastructure as code for deployment looks like this:

// The actual static web app
module staticWebApp 'static-sites.bicep' = {
  name: '${deploymentPrefix}-zebra-gpt'
  params: {
    appSettings: {
      // ...
    }
    location: location
    repositoryBranch: repositoryBranch
    repositoryUrl: repositoryUrl
    staticSiteName: staticSiteName
    tags: tagsWithHiddenLinks
    customDomainName: (repositoryBranch == 'main') ? customDomainName : ''
  }
}

module appServicePlan 'br:icebox.azurecr.io/bicep/ice/providers/web/serverfarms:v1.0' = {
  name: '${deploymentPrefix}-appServicePlan'
  params: {
    tags: tags
    location: location
    appServicePlanName: appServicePlanName
    kind: 'linux'
    sku: {
      name: appServicePlanSku
      capacity: appServicePlanInstanceCount
    }
    zoneRedundant: appServicePlanZoneRedundant
  }
}

resource storageAccount 'Microsoft.Storage/storageAccounts@2021-02-01' existing = {
  name: storageAccountName
}

module zebraGptFunctionApp 'br:icebox.azurecr.io/bicep/ice/composites/function-apps/dotnet:v1.9' = {
  name: '${deploymentPrefix}-function-app'
  params: {
    location: location
    tags: tags
    os: 'linux'
    identityType: 'SystemAssigned,UserAssigned'
    userAssignedIdentities: {
      '${managedIdentityId}': {}
    }
    functionAppName: functionAppName
    appServicePlanName: appServicePlan.outputs.appServicePlanName
    appServicePlanResourceGroup: resourceGroup().name
    storageAccountName: storageAccount.name
    clientId: aadClientId
    tenantId: aadTenantId
    customDomainEnabled: false
    protectedByFrontDoor: false
    easyAuthEnabled: true
    keyVaultReferenceIdentity: managedIdentityId
    netFrameworkVersion: 'v6.0'
    functionsExtensionVersion: '~4'
    fxVersion: 'DOTNET|6.0'
    logAnalyticsWsName: logAnalyticsWsName
    logAnalyticsWsResourceGroup: resourceGroup().name
    additionalWebConfigs: {
      alwaysOn: true
    }
    additionalAppSettings: {
      //..
    }
  }
}

resource staticAppBackend 'Microsoft.Web/staticSites/linkedBackends@2022-03-01' = {
  name: '${staticSiteName}/backend'
  properties: {
    backendResourceId: zebraGptFunctionApp.outputs.functionAppResourceId
    region: location
  }
}

Where static-sites.bicep is

@description('Tags that our resources need')
param tags {
  branch: string
  owner: string
  costCenter: string
  application: string
  description: string
  repo: string
}

@description('Location for the resources')
param location string

@description('Type of managed service identity. SystemAssigned, UserAssigned. https://docs.microsoft.com/en-us/azure/templates/microsoft.web/staticsites?tabs=bicep#managedserviceidentity')
param identityType string = 'SystemAssigned'

@description('The list of user assigned identities associated with the resource.')
param userAssignedIdentities object = {}

@description('The SKU for the static site. https://docs.microsoft.com/en-us/azure/templates/microsoft.web/staticsites?tabs=bicep#skudescription')
param sku object = {
  name: 'Standard'
  tier: 'Standard'
}

@description('Lock the config file for this static web app. https://docs.microsoft.com/en-us/azure/templates/microsoft.web/staticsites?tabs=bicep#staticsite')
param allowConfigFileUpdates bool = true

@description('Where the repository lives eg https://dev.azure.com/investec/investec-cloud-experience/_git/tea-and-biscuits')
param repositoryUrl string

@description('The branch being deployed eg main')
param repositoryBranch string

@description('The name of the static site eg stapp-tab-prod-001')
param staticSiteName string

@secure()
@description('Configuration for your static site')
param appSettings object = {}

@description('Build properties for the static site.')
param buildProperties object = {}

@allowed([
  'Disabled'
  'Disabling'
  'Enabled'
  'Enabling'
])
@description('State indicating the status of the enterprise grade CDN serving traffic to the static web app.')
param enterpriseGradeCdnStatus string = 'Disabled'

@allowed([
  'Disabled'
  'Enabled'
])
@description('State indicating whether staging environments are allowed or not allowed for a static web app.')
param stagingEnvironmentPolicy string = 'Enabled'

@description('Template Options for the static site. https://docs.microsoft.com/en-us/azure/templates/microsoft.web/staticsites?tabs=bicep#staticsitetemplateoptions')
param templateProperties object = {}

@description('Custom domain name for the static side eg testing-swa.sbx.investec.io')
param customDomainName string = ''

resource staticSite 'Microsoft.Web/staticSites@2022-03-01' = { // https://docs.microsoft.com/en-us/azure/templates/microsoft.web/staticsites?tabs=bicep
  name: staticSiteName
  location: location
  tags: tags
  identity: {
    type: identityType
    userAssignedIdentities: empty(userAssignedIdentities) ? null : userAssignedIdentities
  }
  sku: sku
  properties: {
    allowConfigFileUpdates: allowConfigFileUpdates
    provider: 'DevOps' // see: https://github.com/Azure/static-web-apps/issues/516
    repositoryUrl: repositoryUrl
    branch: repositoryBranch
    buildProperties: empty(buildProperties) ? null : buildProperties
    enterpriseGradeCdnStatus: enterpriseGradeCdnStatus
    stagingEnvironmentPolicy: stagingEnvironmentPolicy
    templateProperties: empty(templateProperties) ? null : templateProperties
  }
}

resource customDomain 'Microsoft.Web/staticSites/customDomains@2022-03-01' = if(!empty(customDomainName)) {
  parent: staticSite
  name: !empty(customDomainName) ? customDomainName : 'blank1' // https://github.com/Azure/bicep/issues/1754
  properties: {}
}

resource staticSiteAppsettings 'Microsoft.Web/staticSites/config@2022-03-01' = {
  parent: staticSite
  name: 'appsettings'
  kind: 'config'
  properties: appSettings
}

output defaultHostName string = staticSite.properties.defaultHostname // eg gentle-bush-0db02ce03.azurestaticapps.net
output siteName string = staticSite.name
output siteResourceId string = staticSite.id
output siteSystemAssignedIdentityId string = (staticSite.identity.type == 'SystemAssigned') ? staticSite.identity.principalId : ''

[thomasgauvin]

Ok thanks for the details @johnnyreilly, I think we were able to repro by adding the call to OpenAI and attempting to stream the response back. We're continuing to investigate

[v1212]

@johnnyreilly, thanks for the detail, we are working on a fix in SWA to return the stream response from function app API, before the fix rolling out in SWA product, a workaround to this issue can be calling direct API of function app instead of that of SWA API path, that worked in our investigation, that need an extra CORS setting in the target function app allowing call from web page of SWA app.

[johnnyreilly]

Thanks @v1212 - I'm aware that it's possible to handle calling the function directly; but we struggled to get the Azure AD authentication working with our front end code. In the end we pivoted to using Azure Container Apps with Easy Auth. Great to hear a fix is on the way though!

[thomasgauvin]

Update: We've been investigating this and determined that this depends on our underlying infrastructure. We're working with our partner teams to see how we could support these streaming use cases.

[pietz]

@johnnyreilly, thanks for the detail, we are working on a fix in SWA to return the stream response from function app API, before the fix rolling out in SWA product, a workaround to this issue can be calling direct API of function app instead of that of SWA API path, that worked in our investigation, that need an extra CORS setting in the target function app allowing call from web page of SWA app.

Could you elaborate on that? I'm not able to setup streaming in an Azure Function (Python) in any way. Are you suggesting to separate frontend and backend, in order to have a separate Azure Function? Because I tried that and it didn't work.

[thomasgauvin]

@pietz some .NET versions supported streaming, but Python does not yet as far as I'm aware. That is why Johnny had some success with that

[martgra]

Any update here? @thomasgauvin

Experiencing same - streaming is not working between Static Web app linked to Web app backend. Frontend: react Backend: docker container + fastapi. Can provide more details on request.

[nkma1989]

Hi, I have this exact issue, we are using Azure Web App to host a docker container with a Fast API, talking to Azure OpenAI service, using streaming responses. When I run locally it's streaming and everything works as expected, but when deploying to azure web app it does not stream, but waits and then returns the full payload. We are using python 3.10 and a gunicorn server to host the app. @thomasgauvin is this also something that you are looking at?

[philipmuller]

Hello!

Experiencing the same issue when using nextjs with the Vercel AI SDK deployed on Azure SWA. Any updates or workarounds?

[kunal1054]

We are also facing same issue with angular front end and .net core 7 backend deployed on azure app service. Response is returned as a single response. Works locally. We can directly call open ai streaming chat endpoint as that would expose the api key. This has to be resolved asap

[simplecohesion]

Same here (nextjs + vercel ai sdk). Is there any hope of this being supported, as the thread is over a year old?

[hectordanielc]

Same here (nextjs + vercel ai sdk). Should we open another thread?

[lukju]

I've built both, a Python and and a Node/Javascript API. In both cases, streaming does work locally but not as a deployed SWA. Any ideas or workarounds?

[egerszu]

I've built both, a Python and and a Node/Javascript API. In both cases, streaming does work locally but not as a deployed SWA. Any ideas or workarounds?

The only workaround is that instead of using /api/, use https://your-azure-func.azurewebsites.net/api/whatever/?code=your-function-key

Streaming works this way.

[hectordanielc]

I've built both, a Python and and a Node/Javascript API. In both cases, streaming does work locally but not as a deployed SWA. Any ideas or workarounds?

The only workaround is that instead of using /api/, use https://your-azure-func.azurewebsites.net/api/whatever/?code=your-function-key

Streaming works this way.

Still not streaming for me.

[thomasgauvin]

(Disclaimer, no longer PM for SWA) I wrote https://techcommunity.microsoft.com/t5/apps-on-azure-blog/add-a-context-grounded-ai-chatbot-to-your-azure-static-web-apps/ba-p/4097223#:~:text=Create%20the%20API%20for%20our%20chatbot during my last couple of weeks on the SWA team, might be helpful? Note the text/event-stream header used in the response from the Azure Function. I don't know the current status of SWA anymore but I think that blog should be a helpful resource. Keep me updated!

[hectordanielc]

(Disclaimer, no longer PM for SWA) I wrote https://techcommunity.microsoft.com/t5/apps-on-azure-blog/add-a-context-grounded-ai-chatbot-to-your-azure-static-web-apps/ba-p/4097223#:~:text=Create%20the%20API%20for%20our%20chatbot during my last couple of weeks on the SWA team, might be helpful? Note the text/event-stream header used in the response from the Azure Function. I don't know the current status of SWA anymore but I think that blog should be a helpful resource. Keep me updated!

It didn't work for me but my azure function is made in python. I'll try it with node.

[lukju]

(Disclaimer, no longer PM for SWA) I wrote https://techcommunity.microsoft.com/t5/apps-on-azure-blog/add-a-context-grounded-ai-chatbot-to-your-azure-static-web-apps/ba-p/4097223#:~:text=Create%20the%20API%20for%20our%20chatbot during my last couple of weeks on the SWA team, might be helpful? Note the text/event-stream header used in the response from the Azure Function. I don't know the current status of SWA anymore but I think that blog should be a helpful resource. Keep me updated!

It didn't work for me but my azure function is made in python. I'll try it with node.

No need to go that extra route: the programming language doesn't matter - streaming doesn't work for all flavors of SWA APIs. The only workaround is to deploy the API as a dedicated Functions app and call those endpoint directly (do not link it with the SWA).

[lukju]

Does anybody know if Microsoft is aware of this problem? As response streaming definitely does not work in Static Web Apps, it lowers the usefullness of this service in the context of ChatBots nearly to Zero. Would be great if this could somehow be fixed. Please please, you great makers of this otherwise great service: Let's make it streaming-able :-)

[v1212]

@lukju could you set content type in the api response to "application/octet-stream", or "text/event-stream"? these are two types currently Static Web App detects as streaming content, and also the mostly common types for AI streaming response.

[egerszu]

@v1212 "application/octet-stream" stream works, but not "text/event-stream". Most library does not allow you to set the header to octet-stream as SSE requires you to set it to "text/event-stream".

Here is an example code to try:

Azure Function:

import time
import azure.functions as func
from azurefunctions.extensions.http.fastapi import Request, StreamingResponse

app = func.FunctionApp(http_auth_level=func.AuthLevel.FUNCTION)

def generate_count():
    """Generate a stream of chronological numbers."""
    yield "Bad streaming test: \n"
    for i in range(20):
        time.sleep(0.5)
        yield f"data: {i}\n\n"

@app.route(route="test_app_octet/", methods=["GET"])
async def test_app_octet(req: Request) -> StreamingResponse:
    """Streams the answer from the AI back to the user with application/octet-stream."""
    print(req.__dict__)
    return StreamingResponse(generate_count(), media_type="application/octet-stream")

@app.route(route="test_text_stream/", methods=["GET"])
async def test_text_stream(req: Request) -> StreamingResponse:
    """Streams the answer from the AI back to the user with text/event-stream."""
    print(req.__dict__)
    return StreamingResponse(generate_count(), media_type="text/event-stream")

HTML:

<!DOCTYPE html>
<html>

<head>
    <title>Streaming test</title>
</head>

<body>

    <script>
        async function getSseAppOctet() {
            const sseDivE = document.querySelector(".sse-test-app-octet > .text");
            sseDivE.innerText = "";
            const response = await fetch('/api/test_app_octet/?code=<your api key>', {
                method: 'GET'
            })
            const reader = response.body.pipeThrough(new TextDecoderStream()).getReader()
            while (true) {
                const { value, done } = await reader.read();
                if (done) {
                    break;
                }

                sseDivE.innerText += value;
                console.log('Received', value);
            }
        }

        async function getSseTextStream() {
            const sseDivE = document.querySelector(".sse-test-text-stream > .text");
            sseDivE.innerText = "";
            const response = await fetch('/api/test_text_stream/?code=<your api key>', {
                method: 'GET'
            })
            const reader = response.body.pipeThrough(new TextDecoderStream()).getReader()
            while (true) {
                const { value, done } = await reader.read();
                if (done) {
                    break;
                }

                sseDivE.innerText += value;
                console.log('Received', value);
            }
        }
    </script>

    <div>
        <div style="overflow: hidden;">
            <div class="sse-test-app-octet" style="overflow: hidden; border: 1px solid">
                <button type="button" onclick="getSseAppOctet()">Check SSE App Octet</button>
                <div class="text">
                </div>
            </div>
        </div>
        <div style="overflow: hidden;">
            <div class="sse-test-text-stream" style="overflow: hidden; border: 1px solid">
                <button type="button" onclick="getSseTextStream()">Check SSE Text Stream</button>
                <div class="text">
                </div>
            </div>
        </div>
    </div>

</body>

</html>

Make sure you upload to Azure, because locally with the emulator it works, but not in azure.

[lukju]

Wow, thanks a lot - that mime-type was the trick. Changed to media_type="application/octet-stream" and it works! Great, thanks!

[hectordanielc]

Wow, thanks a lot - that mime-type was the trick. Changed to media_type="application/octet-stream" and it works! Great, thanks!

Can you please share your code?

[lukju]

What kind of code do you need? Did you see the code posted by @egerszu last Friday or so? It pretty much shows what needs to be done

[hectordanielc]

Just to see how are you constructing the endpoint. Mine is not working with media_type="application/octet-stream".

[lukju]

example in python:

example in typescript:

[TheoMcCabe]

Same i have this issue using nuxt on an azure static web app . app works fine locally but streaming binary to my backend when deploying to SWA fails