[Bug]: Uploading files larger than 2GB to SFTP external storage backend fails

chaosgrid commented 1 week ago

⚠️ This issue respects the following points: ⚠️

[x] This is a bug, not a question or a configuration/webserver/proxy issue.
[x] This issue is not already reported on Github OR Nextcloud Community Forum (I've searched it).
[x] Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
[x] I agree to follow Nextcloud's Code of Conduct.

Bug description

Uploading files larger than 2GB to SFTP external storage backend fails. This is on a docker install using the official Nextcloud image version 29.0.9-apache.

Steps to reproduce

Connect sftp storage backend space
Upload or copy file larger than 2GB to it.
It fails.

Expected behavior

Successful file upload.

Nextcloud Server version

29

Operating system

Other

PHP engine version

None

Web server

None

Database engine version

None

Is this bug present after an update or on a fresh install?

None

Are you using the Nextcloud Server Encryption module?

None

What user-backends are you using?

[ ] Default user-backend (database)
[ ] LDAP/ Active Directory
[ ] SSO - SAML
[ ] Other

Configuration report

List of activated Apps

Nextcloud Signing status

Nextcloud Logs

The log has these entries:

[PHP] Info: Invalid HMAC at /var/www/html/3rdparty/phpseclib/phpseclib/phpseclib/Net/SSH2.php#3717
    COPY /remote.php/dav/files/redacted/test_file_over2gb.dat

[PHP] Info: Connection closed by server at /var/www/html/3rdparty/phpseclib/phpseclib/phpseclib/Net/SSH2.php#4069
    COPY /remote.php/dav/files/redacted/test_file_over2gb.dat

[no app in context] Warning: Failed to copy stream to storage
    COPY /remote.php/dav/files/redacted/test_file_over2gb.dat

Additional info

This issue is probably related to or the same as https://github.com/nextcloud/server/issues/48231

It happens on a docker container install using the official Nextcloud image with version 29.0.9-apache

According to Claude, it might be that the packaged phpseclib version is too old. In fact, checking the composer.json in my docker container, it says: "phpseclib/phpseclib": "^2.0.45"

Claude says:

That explains the issue! You're using phpseclib 2.x which has known limitations with files larger than 2GB due to how it handles HMAC calculations with 32-bit integers.

Is this plausible?

When I connect the same storage via FTPS, uploads above 2GB succeed. I am not sure when this bug was introduced since it definitely worked in the past, we uploaded >2GB files to the storage using the SFTP backend before.

chaosgrid commented 1 week ago

Update: This also happens when downloading a file from sftp backend that is bigger than 2GB. It fails right at the 2GB mark and the Nextcloud client simply says "Connection closed"..

chaosgrid commented 1 week ago

I also found this forum post, seems related (though it is not the SFTP backend): https://help.nextcloud.com/t/unable-to-upload-files-larger-than-2gb-via-webdav/200157

chaosgrid commented 1 week ago

Updated to Nextcloud docker image v30.0.2

Same issues. When downloading a file over 2GB from sftp backend, right around the 2GB mark it fails and the debug output is this:

joshtrichards commented 1 week ago

I see you're working the issue a bit over at phpseclib, but wanted to check on this comment:

When I connect the same storage via FTPS, uploads above 2GB succeed.

Was that a typo or did you really test using FTPS instead of SFTP?

chaosgrid commented 1 week ago

I see you're working the issue a bit over at phpseclib, but wanted to check on this comment:

When I connect the same storage via FTPS, uploads above 2GB succeed.

Was that a typo or did you really test using FTPS instead of SFTP?

Yes, not a typo. I tested the same storage server via the FTPS Nextcloud storage backend and using that, there are no issues with files over 2GB. Also, directly connecting to the storage server using an SFTP tool (like WinSCP) also does not produce these bugs.

nextcloud / server