[Bug]: Not able to upload files greater than 75 mb

appsmithorg / appsmith

Platform to build admin panels, internal tools, and dashboards. Integrates with 25+ databases and any API.

https://www.appsmith.com

Apache License 2.0

34.26k stars 3.71k forks source link

[Bug]: Not able to upload files greater than 75 mb #14293

Closed ajinkyakulkarni closed 11 months ago

ajinkyakulkarni commented 2 years ago

Is there an existing issue for this?

[X] I have searched the existing issues

Description

A user on discord mentioned that he/she is not able to upload a file great than 65MB

https://discord.com/channels/725602949748752515/760761686549463060/981196657653866516

I was able to reproduce this, when a file of 95MB was uploaded

Upon more investigation, it was found thatclient_max_body_size is correctly set in nginx

But, it was observed that the incorrect content-length is being sent in the header. For example, for a file of size 78mb, the sent content-length header was 104mb. For a file of size 68mb, the sent content-length header was 90mb.

So, because of this, users are not able to upload files of 100mb even if that is what we say is allowed.

Steps To Reproduce

Upload the file of size say 80mb
Observe that the content-length header is greater than 100mb.

Public Sample App

No response

Version

Cloud

ajinkyakulkarni commented 2 years ago

To temporarily get unblocked, I could verify this approach

Open the file /etc/nginx/conf.d/app.conf
Change client_max_body_size to 200m
Restart the nginx /usr/sbin/nginx -s reload
Verify the change /usr/sbin/nginx -T

Here is the complete verification of this approach

ajinkyakulkarni commented 2 years ago

Some more investigation

I launched the Appsmith Fat Container
Then I changed client_max_body_size to 200m in /etc/nginx/sites-available/default
Restarted the niginx /usr/sbin/nginx -s reload

Then I uploaded a file of 100mb and I got the following Java OutOfMemory error

Then temporarily increased the Java Heap Size and restarted the supervisor

cd /opt/appsmith
export APPSMITH_JAVA_HEAP_ARG="-Xmx512m"
./update-and-restart-supervisor.sh

Now, when I try to upload files, the backend server shows the the actions are getting executing without the Java OutOfMemory error

ajinkyakulkarni commented 2 years ago

I am able to reproduce an error where backend restarts mentioned by the user https://discord.com/channels/725602949748752515/981196657653866516

Add APPSMITH_JAVA_HEAP_ARG="-Xmx4G" in /appsmith-stacks/configuration/docker.env
Change client_max_body_size to 200mb in /opt/appsmith/templates/nginx-app-http.conf.template.sh /opt/appsmith/templates/nginx-app-https.conf.template.sh
Restart supervisor ./update-and-restart-supervisor.sh
Upload a large file and notice the restarts in /appsmith-stacks/logs/backend/backend.log

On client I see 502 Bad Gateway error

ajinkyakulkarni commented 2 years ago

I confirmed that Java Backend Server crashes even when launched outside of supervisord

https://user-images.githubusercontent.com/800578/173202763-74a99ed1-cdf7-4c89-b11a-99deb2169209.mp4