[Bug]: Uploading files larger than 2GB to SFTP external storage backend fails

chaosgrid commented 2 days ago

⚠️ This issue respects the following points: ⚠️

[x] This is a bug, not a question or a configuration/webserver/proxy issue.
[x] This issue is not already reported on Github OR Nextcloud Community Forum (I've searched it).
[x] Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
[x] I agree to follow Nextcloud's Code of Conduct.

Bug description

Uploading files larger than 2GB to SFTP external storage backend fails. This is on a docker install using the official Nextcloud image version 29.0.9-apache.

Steps to reproduce

Connect sftp storage backend space
Upload or copy file larger than 2GB to it.
It fails.

Expected behavior

Successful file upload.

Nextcloud Server version

29

Operating system

Other

PHP engine version

None

Web server

None

Database engine version

None

Is this bug present after an update or on a fresh install?

None

Are you using the Nextcloud Server Encryption module?

None

What user-backends are you using?

[ ] Default user-backend (database)
[ ] LDAP/ Active Directory
[ ] SSO - SAML
[ ] Other

Configuration report

List of activated Apps

Nextcloud Signing status

Nextcloud Logs

The log has these entries:

[PHP] Info: Invalid HMAC at /var/www/html/3rdparty/phpseclib/phpseclib/phpseclib/Net/SSH2.php#3717
    COPY /remote.php/dav/files/redacted/test_file_over2gb.dat

[PHP] Info: Connection closed by server at /var/www/html/3rdparty/phpseclib/phpseclib/phpseclib/Net/SSH2.php#4069
    COPY /remote.php/dav/files/redacted/test_file_over2gb.dat

[no app in context] Warning: Failed to copy stream to storage
    COPY /remote.php/dav/files/redacted/test_file_over2gb.dat

Additional info

This issue is probably related to or the same as https://github.com/nextcloud/server/issues/48231

It happens on a docker container install using the official Nextcloud image with version 29.0.9-apache

According to Claude, it might be that the packaged phpseclib version is too old. In fact, checking the composer.json in my docker container, it says: "phpseclib/phpseclib": "^2.0.45"

Claude says:

That explains the issue! You're using phpseclib 2.x which has known limitations with files larger than 2GB due to how it handles HMAC calculations with 32-bit integers.

Is this plausible?

When I connect the same storage via FTPS, uploads above 2GB succeed. I am not sure when this bug was introduced since it definitely worked in the past, we uploaded >2GB files to the storage using the SFTP backend before.

chaosgrid commented 1 day ago

Update: This also happens when downloading a file from sftp backend that is bigger than 2GB. It fails right at the 2GB mark and the Nextcloud client simply says "Connection closed"..

chaosgrid commented 1 day ago

I also found this forum post, seems related (though it is not the SFTP backend): https://help.nextcloud.com/t/unable-to-upload-files-larger-than-2gb-via-webdav/200157

chaosgrid commented 9 hours ago

Updated to Nextcloud docker image v30.0.2

Same issues. When downloading a file over 2GB from sftp backend, right around the 2GB mark it fails and the debug output is this:

nextcloud / server