Transfer unexpectedly closed when calling app through http://{appName}

[tamland]

Please provide us with the following information:

This issue is a: (mark with an x)

• [x] bug report -> please search issues before submitting
• [ ] documentation issue or request
• [ ] regression (a behavior that used to work and stopped in a new release)

Issue description

When calling an app through http://{appName} as described in https://github.com/microsoft/azure-container-apps/issues/473 we are experiencing that the connection is often unexpectedly closed. Here's an example calling the app with curl:

~ # curl http://test-service/test -H "authorization: bearer $TOKEN" -H "Connection: close" -v -Ss --http1.1 1> /dev/null
*   Trying 10.1.112.134:80...
* Connected to test-service (10.1.112.134) port 80 (#0)
> GET /test HTTP/1.1
> Host: test-service
> User-Agent: curl/8.0.1
> Accept: */*
> authorization: bearer *****
> Connection: close
> 
< HTTP/1.1 200 OK
< x-content-type-options: nosniff
< x-xss-protection: 1; mode=block
< cache-control: no-cache, no-store, max-age=0, must-revalidate
< pragma: no-cache
< expires: 0
< strict-transport-security: max-age=31536000 ; includeSubDomains
< x-frame-options: DENY
< content-type: application/json
< date: Mon, 24 Apr 2023 07:55:31 GMT
< connection: close
< transfer-encoding: chunked
< 
{ [13425 bytes data]
* transfer closed with outstanding read data remaining
* Closing connection 0
curl: (18) transfer closed with outstanding read data remaining
~ # 

The issue appears to be related to the chunked transfer encoding and be timing/size based as it does not happen on every call. Other clients than curl returns similar issue ("request aborted" etc)

I addition, this issue does not to occur when calling the app through the FQDN, only through http://{appName} as seen here:

~ # curl http://test-service.internal.ashy*****.westeurope.azurecontainerapps.io/test -H "authorization: bearer $TOKEN" -H "Connection: close" -v -Ss --http1.1 1> /dev/null
*   Trying 10.1.195.73:80...
* Connected to test-service.internal.ashy*****.westeurope.azurecontainerapps.io (10.1.195.73) port 80 (#0)
> GET /test HTTP/1.1
> Host: test-service.internal.ashy*****.westeurope.azurecontainerapps.io
> User-Agent: curl/8.0.1
> Accept: */*
> authorization: bearer *****
> Connection: close
> 
< HTTP/1.1 200 OK
< x-content-type-options: nosniff
< x-xss-protection: 1; mode=block
< cache-control: no-cache, no-store, max-age=0, must-revalidate
< pragma: no-cache
< expires: 0
< x-frame-options: DENY
< content-type: application/json
< date: Mon, 24 Apr 2023 07:54:59 GMT
< connection: close
< transfer-encoding: chunked
< 
{ [13482 bytes data]
* Closing connection 0
~ # 

Since this issue does not occur when using FQDN, nor on other platforms such as Azure Container Instances, we believe the issue is related to ACA.

Steps to reproduce

1. Start a server that returns a sufficiently large amount of data using transfer-encoding: chunked.
2. Call the http://{appName} endpoint with the above curl command repeatedly.
3. Some requests should fail with error "transfer closed with outstanding read data remaining"

Expected behavior [What you expected to happen.]

Actual behavior [What actually happened.]

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context

Ex. Did this issue occur in the CLI or the Portal?

[torosent]

Hi. I filed a bug item and will update soon.

[njuCZ]

Hi @tamland, I can not reproduce the issue. I have created a container app app1 returning data using transfer-encoding: chunked, and inside another container app, I tried to call http://app1, it seems everything is fine.

btw, may I confirm with you that is there any timeout settings for your app?

[tamland]

@njuCZ Your response size of 6019 bytes is probably too small to trigger this. We are seeing the issue on requests that takes anything between 0 and a few seconds, so it should not be related to timeouts in the application.

[njuCZ]

@tamland Thanks for your info, I can repro now, investigating

[tamland]

@njuCZ @torosent Did you have any update on this?

[ahmelsayed]

The last update on this was @zhenqxuMSFT investigating the issue and opening https://github.com/envoyproxy/envoy/issues/28329 as a result of what he was seeing.

[tamland]

@ahmelsayed Thank you. I'll follow the envoy issue

[akselikap]

Hey! Could we add a mention of this to the Container Apps documentation? We just spent a week trying to debug this issue and now that we found out the root cause I was able to find this issue. It would be nice to add a mention to the container apps docs that you can't use connection: close with transfer-encoding: chunked when working with container apps.

I'll add a search term I used here so maybe another unlucky soul finds this a bit faster:

org.apache.http.TruncatedChunkException: Truncated chunk (expected size: 32,760; actual size: 31,835)

AxiosError: maxContentLength size of -1 exceeded

curl: (18) transfer closed with outstanding read data remaining