JavaScript file upload to container app slower than to app service

[damiangelis]

Please provide us with the following information:

This issue is a: (mark with an x)

• [X] bug report -> please search issues before submitting
• [ ] documentation issue or request
• [ ] regression (a behavior that used to work and stopped in a new release)

Issue description

We have an Azure container app with the following specs:

• Single-revision mode
• Consumption only
• 1 CPU, 2 Gi memory
• 1 minimum replica
• 10 maximum replicas

The container app is hosting a small Python web API created with the FastAPI framework.

In our React app, hosted in a storage container, we have a file uploader that uses fetch, and an upload to the back-end (that is, to the container app) is taking too long. For example, a .tif image file of 81.4 MB is taking around 5.2 minutes.

What we tried so far:

1. We scaled out the container app by adding more replicas (5), and times didn't decrease
2. We scaled up the container app to 2 CPUs and 4 Gi memory, and times didn't decrease, either
3. We deployed the Python web API to an app service. This way, the file upload is only taking 30 seconds. This made us discard the Python code base as the root cause, believing now it's a problem with the container app infrastructure

Any ideas?

[simonjj]

Thank you for raising this with us @damiangelis. We've occasionally seen similar behavior on occasion but have never been able to pin point the exact cause. Could you share your app or a portion of your app for us to take a closer look please?

[damiangelis]

@simonjj Sure thing. Please let me know if you need more details.

Dockerfile

FROM --platform=linux/amd64 python:3.10.11

WORKDIR /code

COPY ./requirements.txt /code/requirements.txt

RUN pip install --no-cache-dir --upgrade -r requirements.txt

COPY ./src /code/src

EXPOSE 5000

ENTRYPOINT [ "uvicorn", "src.main:app", "--proxy-headers", "--host", "0.0.0.0", "--port", "5000", "--limit-concurrency", "100", "--limit-max-requests", "100", "--workers", "10" ]

docker-compose.yml

version: "3.8"
services:
  cb-api:
    container_name: oai-tools-api
    build:
      context: .
      dockerfile: Dockerfile
    image: cb-api:latest
    platform: linux/amd64
    ports:
      - "8000:8080"
    volumes:
      - ./app:/code/app
    env_file:
      - .env

React service

export const uploadLocalFile = async (
  payload: IUploadPayload,
  accessToken: string,
): Promise<APISourceUpdateResponse> => {
  const formdata = new FormData();
  formdata.append('file', payload.file);
  formdata.append('file_expiration_date', payload.file_expiration_date);

  const res = await makeRequest(
    `${config.semanticSearchApiUrl}/api/v1/files`,
    'POST',
    accessToken,
    'multipart/form-data',
    false,
    formdata,
  );

  if (!res) {
    throw new Error('Failed to make request');
  }

  const { status } = res;
  const data = await res.json();
  return { status, data };
};

FastAPI router

from fastapi import APIRouter, UploadFile

from werkzeug.utils import secure_filename

from src.config import get_main_logger

v1_router = APIRouter(prefix="/files", tags=["Files"])

logger = get_main_logger()

@v1_router.post(
    "",
)
async def upload(
    file: UploadFile,
):
    # Access the uploaded file and get its filename and content type
    filename = secure_filename(file.filename)
    content_type = file.content_type

[LianaAdaza]

Hi @damiangelis, thank you for reaching out to us! The issue has been replicated and confirmed. We shall keep you updated once we hear back from our engineering team.

[Tratcher]

Testing with a basic AspNetCore app I get consistent downloads but variable uploads:

App: 1 replica, 0.25 CPU, 0.5Gi memory

HTTP & HTTP/1.1 GET /67108864 took 1859ms, rate: 34.41 MB/s Post 67108864 took 271ms, rate: 235.54 MB/s

HTTPS & HTTP/1.1 GET /67108864 took 1967ms, rate: 32.52 MB/s Post 67108864 took 1931ms, rate: 33.13 MB/s

HTTPS & HTTP/2 GET /67108864 took 1729ms, rate: 36.99 MB/s Post 67108864 took 70732ms, rate: 0.90 MB/s

Other clients, apps, and deployment combinations I've tried have given similar results. The only combination that indicates an infrastructure issue is HTTP/2 uploads, I'll follow up on that.

@damiangelis If you test these combinations do you get similar results?

[Tratcher]

I've made configuration changes that improved HTTP/2 POST throughput from 0.9 MB/s to 7.54 MB/s in my environment. I'll let you know when this is broadly deployed for validation.

[lukemoreira]

I've made configuration changes that improved HTTP/2 POST throughput from 0.9 MB/s to 7.54 MB/s in my environment. I'll let you know when this is broadly deployed for validation.

@Tratcher Are you uploading to a blob? Any optimizations in the code? I always get a consistently slow speed of around 1 Mbps.

[Tratcher]

@lukemoreira I was uploading to a null stream in the container app, removing any business logic so I could highlight ingress issues.

If you're forwarding that data on to blob storage then I'd encourage you to try these variants: 1) upload client data to container apps 2) upload app data to blob storage 3) upload client data directly to blob storage

That way it will be clearer if you're having issues with the incoming or outgoing part of your task.

[lukemoreira]

@Tratcher, I have 2 apps and the blob service.

1 - User facing app (Javascript) 2 - API application (.net) 3 - Storage blob

the connection from the API to blob happens at a great speed, but the connection from the client to the API is where it seems to be capped at 1 Mbps.

So the problem seems to be the speed between Container Apps instances.

[Tratcher]

@lukemoreira 1Mbps matches the performance problems I saw with HTTP/2. Can you test these combinations?

• HTTP (no TLS) with HTTP/1.1
• HTTPS with HTTP/1.1
• HTTPS with HTTP/2

[lukemoreira]

@lukemoreira 1Mbps matches the performance problems I saw with HTTP/2. Can you test these combinations?

• HTTP (no TLS) with HTTP/1.1
• HTTPS with HTTP/1.1
• HTTPS with HTTP/2

By HTTP/x, do you mean at the app config or should I setup up my post requests to target a certain protocol?

[Tratcher]

If you're testing from a browser you don't get much control over the protocol, but you can at least use the browser's F12 tools to look at the requests and see which protocol was used. A tool like Curl gives you more control over the protocol version.

If you enable HTTP (not HTTPS) access to your app that will always use HTTP/1.1, HTTP/2 requires HTTPS.

[Tratcher]

This change is now available in the East US region and should reach other regions in the next few days. I can confirm 8MB/s uploads from Azure US West 2 to East US. Please re-test. If you're still having issues let us know which region, and your rough location relative to that region since this issue is significantly affected by distance/latency.