Slow uploads with HTTP/2

[haarp]

How to use GitHub

• Please use the 👍 reaction to show that you are affected by the same issue.
• Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
• Subscribe to receive notifications on status change and new comments.

Summary

When HTTP/2 is enabled on the webserver, uploads are very slow, topping out at slightly over 1MByte/s, despite both client and server offering vastly more bandwidth. Once HTTP2 was disabled, upload speeds were improved by a factor of over x10.

All PHP/DB optimizations are in place, and more. The CPU load on server and client remains low, IO load on the DB and storage remains low. Issue is observed during large file uploads, so DB overhead isn't the issue.

This issue was observed by multiple people, e.g. here, here and here. It seems to affect both nginx and Apache.

Steps to reproduce

1. Enable HTTP/2 on webserver
2. Upload using a WebDAV client - this test used the Windows 10 client via a mapped network drive
3. Get uploadin'

Expected behaviour

>10 Mbyte/s

Actual behaviour

1-2 Mbyte/s

Server configuration

Operating system: Debian Linux 10 Web server: Apache 2.4.38-3+deb10u4 Database: MariaDB 10.3.27-0+deb10u1 PHP version: PHP FPM-CGI 7.3.19-1~deb10u1 Nextcloud version: 19.0.7 Updated from an older Nextcloud/ownCloud or fresh install: Updated Where did you install Nextcloud from: Included updater

Signing status:

Signing status

``` No errors have been found. ```

List of activated apps:

App list

``` Enabled: - accessibility: 1.5.0 - activity: 2.12.1 - calendar: 2.0.4 - camerarawpreviews: 0.7.10 - checksum: 0.4.5 - cloud_federation_api: 1.2.0 - comments: 1.9.0 - contacts: 3.4.3 - dav: 1.15.0 - duplicatefinder: 0.0.4 - federatedfilesharing: 1.9.0 - federation: 1.9.0 - files: 1.14.0 - files_3d: 0.3.3 - files_rightclick: 0.16.0 - files_sharing: 1.11.0 - files_trashbin: 1.9.0 - files_versions: 1.12.0 - logreader: 2.4.0 - lookup_server_connector: 1.7.0 - metadata: 0.12.0 - nextcloud_announcements: 1.8.0 - notes: 3.6.4 - notifications: 2.7.0 - oauth2: 1.7.0 - password_policy: 1.9.1 - photos: 1.1.0 - privacy: 1.3.0 - provisioning_api: 1.9.0 - recommendations: 0.7.0 - serverinfo: 1.9.0 - settings: 1.1.0 - sharebymail: 1.9.0 - support: 1.2.1 - survey_client: 1.7.0 - systemtags: 1.9.0 - text: 3.0.1 - theming: 1.10.0 - twofactor_backupcodes: 1.8.0 - updatenotification: 1.9.0 - viewer: 1.3.0 - workflowengine: 2.1.0 Disabled: - admin_audit - contactsinteraction - encryption - files_external - files_pdfviewer - files_videoplayer - firstrunwizard - ransomware_protection - user_ldap ```

Nextcloud configuration:

Config report

``` { "system": { "instanceid": "***REMOVED SENSITIVE VALUE***", "passwordsalt": "***REMOVED SENSITIVE VALUE***", "secret": "***REMOVED SENSITIVE VALUE***", "trusted_domains": [ "***REMOVED SENSITIVE VALUE***" ], "datadirectory": "***REMOVED SENSITIVE VALUE***", "overwrite.cli.url": "***REMOVED SENSITIVE VALUE***", "overwritewebroot": "\/", "htaccess.RewriteBase": "\/", "dbtype": "mysql", "version": "19.0.7.1", "knowledgebaseenabled": false, "apps_paths": [ { "path": "\/var\/www\/nextcloud\/apps", "url": "\/apps", "writable": true } ], "dbname": "***REMOVED SENSITIVE VALUE***", "dbhost": "***REMOVED SENSITIVE VALUE***", "dbtableprefix": "oc_", "dbuser": "***REMOVED SENSITIVE VALUE***", "dbpassword": "***REMOVED SENSITIVE VALUE***", "installed": true, "loglevel": 2, "mail_smtpmode": "sendmail", "theme": "", "maintenance": false, "trashbin_retention_obligation": "auto", "memcache.local": "\\OC\\Memcache\\APCu", "mail_from_address": "***REMOVED SENSITIVE VALUE***", "mail_domain": "***REMOVED SENSITIVE VALUE***", "session_lifetime": 86400, "mysql.utf8mb4": true, "updater.release.channel": "stable", "preview_max_x": "2048", "preview_max_y": "2048", "preview_max_filesize_image": 100, "app_install_overwrite": [ "checksum" ] } } ```

Are you using external storage, if yes which one: no Are you using encryption: no Are you using an external user-backend, if yes which one: no

Client configuration

Browser: - Client software: Windows 10 WebClient service (Microsoft-WebDAV-MiniRedir/10.0.18363) Operating system: Windows 10

Logs

Web server error log

Web server error log

``` no error logs during upload ```

Nextcloud log (data/nextcloud.log)

Nextcloud log

``` no nextcloud log during upload ```

Browser log

Browser log

``` not using a browser ```

[aledr]

I see no improvement in disabling h2 and h2c. 1.8 MB/sec maximum.

Apache 2.4.43 PHP 7.4.19 Nextcloud 21.0.2

[szaimen]

Is this Issue still valid in NC21.0.3? If not, please close this issue. Thanks! :)

[haarp]

Unfortunately I can't test currently. Anybody else?

[SimplyCorbett]

Yes this is still an issue in nextcloud 22. 2MB/s with http2 40MB/s without http2.

[Juuuuulie10]

4MB/s with http2 55MB/s without http2

[DoktorHeisenberg]

When disabling http2, I'm getting three times the upload speeds: 2MB/s with http2 6,2 MB/s without http2

[MadMakz]

Same.

HTTP2 > 2-2.5 MBytes No HTTP2 > Maxing out (Current 4.1 MBytes)

Debian 10.11 Nginx 1.14.2 PHP 7.4.26 FPM + Opcache + APCu

[ZWC-father]

nextcloud 23.0.0 Ubuntu 20.04 Nginx 1.21.5 php 7.4.3 with opcache Apcu 25mb/s without http2 2mb/s with http2

[solracsf]

This seems to be an overall issue, not a Nextcloud (only) issue.

See here for more technical informations https://blog.cloudflare.com/delivering-http-2-upload-speed-improvements/

[ghost]

I could confirm that it is not a Nextcloud issue, but rather an issue with webservers ( nginx, apache ).

Using the following configuration solved the upload bandwidth issue with nginx

client_body_buffer_size 512k;

[CarlSchwan]

https://github.com/nextcloud/documentation/pull/8323

[zijiren233]

For apache, add the H2WindowSize 1048576 parameter For nginx,add the http2_body_preread_size 1048576; parameter Both of these can be set as global parameters

[sentient-russ]

TY zijiren233! This allowed the same improvement while keeping https end to end encryption. I did the following:

If your using SSLEngine on in your VirtualHost configuration for nextcloud make sure mod http2 is enabled

sudo a2enmod http2

sudo nano /etc/apache2/mods-enabled/http2.conf

Add line H2WIndowSize 1048576

<IfModule !mpm_prefork>
    Protocols h2 h2c http/1.1
    H2WindowSize 1048576
</IfModule>

[Jookia]

Just to add the pile of data, we went from 300KiB/s to 6MiB/s by setting client_max_body_size to 5M; A smaller buffer size did not max out the connection.

[prashant3285]

TY zijiren233! This allowed the same improvement while keeping https end to end encryption. I did the following:

If your using SSLEngine on in your VirtualHost configuration for nextcloud make sure mod http2 is enabled

sudo a2enmod http2

sudo nano /etc/apache2/mods-enabled/http2.conf

Add line H2WIndowSize 1048576

<IfModule !mpm_prefork>
    Protocols h2 h2c http/1.1
    H2WindowSize 1048576
</IfModule>

Thanks @zijiren233 and @sentient-russ for the guidance. I tried H2WindowSize 1048576 but didnt see much improvement. My connection upload limit is 3.5 MB/s but I was getting 1 MB/s So I took trials and finally settled at H2WindowSize 5242880 . Now I am getting 3.5 MB/s as next cloud upload speed.

Also I disabled file chunking as it was causing severe delays in uploading Check Here sudo -u www-data php var/www/nextcloud/occ config:app:set files max_chunk_size --value 0

Edit: uploading big files was becoming an issue so moved to chunk size of 50Mb sudo -u www-data php var/www/nextcloud/occ config:app:set files max_chunk_size --value 5242880

[rs-development]

So does anyone experience issues with downloads or only uploads?

I am using haproxy in front of nginx/fpm. If I have enabled http2 for all connections downloads start of 50% and drop down to 300kB/s. With http2 terminated in haproxy and http/1.1 between haproxy and nginx the download maxes out my 50MBit/s connection.

[szaimen]

H2WindowSize

Could someone please open a PR over at https://github.com/nextcloud/documentation to document that this value needs to be adjusted when http2 is activated in Apache? Thanks!

[szaimen]

I'll move this to documentation so that it does not get lost