search

nextcloud / server

☁️ Nextcloud server, a safe home for all your data
https://nextcloud.com
GNU Affero General Public License v3.0
26.9k stars 4.01k forks source link

Large files (>10MB) on S3 primary storage are lost when uploaded to group folder #48481

Open infra-monkey opened 2 days ago

infra-monkey commented 2 days ago

How to use GitHub

Steps to reproduce

  1. Upload a large file to the group folder
  2. Try to access the file
  3. Get an error saying that the object cannot be found

Expected behaviour

Object should be present

Actual behaviour

I can see that The file is written on the backend image Nextcloud logs show 

 Exception Failed to read object urn:oid:769058
Could not get object urn:oid:769058 for file __groupfolders/6/24-01-01 14-59-28 1225.mov 
fopen(httpseek://): Failed to open stream: "OC\Files\Stream\SeekableHttpStream::stream_open" call failed at /var/www/html/lib/private/Files/Stream/SeekableHttpStream.php#49 
fopen(http://s3.empire.lan:9000/nextcloud-data/urn%3Aoid%3A769058): Failed to open stream: HTTP request failed! HTTP/1.1 416 Requested Range Not Satisfiable at /var/www/html/lib/private/Files/ObjectStore/S3ObjectTrait.php#72 
ServiceUnavailable Could not open file

Uploading to a regular user folder works perfectly well. Moving a large file from a user folder to a group folder looses the file also. However, copying a large file from a user folder to a group folder works. The file is accessible from the group folder. Files <10MB don't seem to show the same behavior.

Server configuration

Operating system: docker container

Web server: nginx

Database: postgresql 15

PHP version: 8.2

Nextcloud version: (see Nextcloud admin page) 29.0.7 & 30.0.0

Group folders version: 17.0.4 & 18.0.2

Updated from an older Nextcloud/ownCloud or fresh install: Updated from older versions

Where did you install Nextcloud from:

Are you using external storage, if yes which one: local/s3/smb/sftp/... primary storage is S3 minio

Are you using encryption: no

Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/Saml/... OAuth2

Client configuration

Browser: Firefox/chrome/brave

Operating system: Linux Leap or Fedora

Logs

No error during upload, but when trying to access the file I get this :

{"reqId":"AH29rbeboMLyOpfwxiEM","level":3,"time":"2024-10-01T05:48:57+00:00","remoteAddr":"10.27.1.1","user":"mattermost-bnrda-ix5z9ksay38488pmnefdehnmer","app":"PHP","method":"GET","url":"/remote.php/dav/files/mattermost-bnrda-ix5z9ksay38488pmnefdehnmer/BNRDA%20Video%20Upload/24-01-01%2014-59-28%201225.mov","message":"fopen(http://s3.empire.lan:9000/nextcloud-data/urn%3Aoid%3A769064): Failed to open stream: HTTP request failed! HTTP/1.1 416 Requested Range Not Satisfiable\r\n at /var/www/html/lib/private/Files/ObjectStore/S3ObjectTrait.php#72","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:130.0) Gecko/20100101 Firefox/130.0","version":"30.0.0.14","data":{"app":"PHP"}}
{"reqId":"s6MMhsgognRjj6U8YDz8","level":3,"time":"2024-10-01T05:48:57+00:00","remoteAddr":"10.27.1.1","user":"mattermost-bnrda-ix5z9ksay38488pmnefdehnmer","app":"PHP","method":"GET","url":"/remote.php/dav/files/mattermost-bnrda-ix5z9ksay38488pmnefdehnmer/BNRDA%20Video%20Upload/24-01-01%2014-59-28%201225.mov","message":"fopen(http://s3.empire.lan:9000/nextcloud-data/urn%3Aoid%3A769064): Failed to open stream: HTTP request failed! HTTP/1.1 416 Requested Range Not Satisfiable\r\n at /var/www/html/lib/private/Files/ObjectStore/S3ObjectTrait.php#72","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:130.0) Gecko/20100101 Firefox/130.0","version":"30.0.0.14","data":{"app":"PHP"}}
{"reqId":"AH29rbeboMLyOpfwxiEM","level":3,"time":"2024-10-01T05:48:57+00:00","remoteAddr":"10.27.1.1","user":"mattermost-bnrda-ix5z9ksay38488pmnefdehnmer","app":"PHP","method":"GET","url":"/remote.php/dav/files/mattermost-bnrda-ix5z9ksay38488pmnefdehnmer/BNRDA%20Video%20Upload/24-01-01%2014-59-28%201225.mov","message":"fopen(httpseek://): Failed to open stream: &quot;OC\\Files\\Stream\\SeekableHttpStream::stream_open&quot; call failed at /var/www/html/lib/private/Files/Stream/SeekableHttpStream.php#49","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:130.0) Gecko/20100101 Firefox/130.0","version":"30.0.0.14","data":{"app":"PHP"}}
{"reqId":"s6MMhsgognRjj6U8YDz8","level":3,"time":"2024-10-01T05:48:57+00:00","remoteAddr":"10.27.1.1","user":"mattermost-bnrda-ix5z9ksay38488pmnefdehnmer","app":"PHP","method":"GET","url":"/remote.php/dav/files/mattermost-bnrda-ix5z9ksay38488pmnefdehnmer/BNRDA%20Video%20Upload/24-01-01%2014-59-28%201225.mov","message":"fopen(httpseek://): Failed to open stream: &quot;OC\\Files\\Stream\\SeekableHttpStream::stream_open&quot; call failed at /var/www/html/lib/private/Files/Stream/SeekableHttpStream.php#49","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:130.0) Gecko/20100101 Firefox/130.0","version":"30.0.0.14","data":{"app":"PHP"}}
{"reqId":"AH29rbeboMLyOpfwxiEM","level":3,"time":"2024-10-01T05:48:57+00:00","remoteAddr":"10.27.1.1","user":"mattermost-bnrda-ix5z9ksay38488pmnefdehnmer","app":"objectstore","method":"GET","url":"/remote.php/dav/files/mattermost-bnrda-ix5z9ksay38488pmnefdehnmer/BNRDA%20Video%20Upload/24-01-01%2014-59-28%201225.mov","message":"Could not get object urn:oid:769064 for file __groupfolders/6/24-01-01 14-59-28 1225.mov","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:130.0) Gecko/20100101 Firefox/130.0","version":"30.0.0.14","exception":{"Exception":"Exception","Message":"Failed to read object urn:oid:769064","Code":0,"Trace":[{"file":"/var/www/html/lib/private/Files/ObjectStore/ObjectStoreStorage.php","line":308,"function":"readObject","class":"OC\\Files\\ObjectStore\\S3","type":"->","args":["urn:oid:769064"]},{"file":"/var/www/html/lib/private/Files/Storage/Wrapper/Wrapper.php","line":283,"function":"fopen","class":"OC\\Files\\ObjectStore\\ObjectStoreStorage","type":"->","args":["__groupfolders/6/24-01-01 14-59-28 1225.mov","r"]},{"file":"/var/www/html/lib/private/Files/Storage/Wrapper/Jail.php","line":288,"function":"fopen","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->","args":["__groupfolders/6/24-01-01 14-59-28 1225.mov","r"]},{"file":"/var/www/html/lib/private/Files/Storage/Wrapper/Quota.php","line":153,"function":"fopen","class":"OC\\Files\\Storage\\Wrapper\\Jail","type":"->","args":["24-01-01 14-59-28 1225.mov","r"]},{"file":"/var/www/html/lib/private/Files/Storage/Wrapper/Wrapper.php","line":283,"function":"fopen","class":"OC\\Files\\Storage\\Wrapper\\Quota","type":"->","args":["24-01-01 14-59-28 1225.mov","r"]},{"file":"/var/www/html/lib/private/Files/Storage/Wrapper/PermissionsMask.php","line":97,"function":"fopen","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->","args":["24-01-01 14-59-28 1225.mov","r"]},{"file":"/var/www/html/lib/private/Files/Storage/Wrapper/Wrapper.php","line":283,"function":"fopen","class":"OC\\Files\\Storage\\Wrapper\\PermissionsMask","type":"->","args":["24-01-01 14-59-28 1225.mov","r"]},{"file":"/var/www/html/lib/private/Files/View.php","line":1134,"function":"fopen","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->","args":["24-01-01 14-59-28 1225.mov","r"]},{"file":"/var/www/html/lib/private/Files/View.php","line":962,"function":"basicOperation","class":"OC\\Files\\View","type":"->","args":["fopen","/BNRDA Video Upload/24-01-01 14-59-28 1225.mov",["read"],"r"]},{"file":"/var/www/html/apps/dav/lib/Connector/Sabre/File.php","line":457,"function":"fopen","class":"OC\\Files\\View","type":"->","args":["BNRDA Video Upload/24-01-01 14-59-28 1225.mov","r"]},{"file":"/var/www/html/3rdparty/sabre/dav/lib/DAV/CorePlugin.php","line":85,"function":"get","class":"OCA\\DAV\\Connector\\Sabre\\File","type":"->","args":[]},{"file":"/var/www/html/3rdparty/sabre/event/lib/WildcardEmitterTrait.php","line":89,"function":"httpGet","class":"Sabre\\DAV\\CorePlugin","type":"->","args":[["Sabre\\HTTP\\Request"],["Sabre\\HTTP\\Response"]]},{"file":"/var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php","line":472,"function":"emit","class":"Sabre\\DAV\\Server","type":"->","args":["method:GET",[["Sabre\\HTTP\\Request"],["Sabre\\HTTP\\Response"]]]},{"file":"/var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php","line":253,"function":"invokeMethod","class":"Sabre\\DAV\\Server","type":"->","args":[["Sabre\\HTTP\\Request"],["Sabre\\HTTP\\Response"]]},{"file":"/var/www/html/apps/dav/lib/Server.php","line":370,"function":"start","class":"Sabre\\DAV\\Server","type":"->","args":[]},{"file":"/var/www/html/apps/dav/appinfo/v2/remote.php","line":19,"function":"exec","class":"OCA\\DAV\\Server","type":"->","args":[]},{"file":"/var/www/html/remote.php","line":146,"args":["/var/www/html/apps/dav/appinfo/v2/remote.php"],"function":"require_once"}],"File":"/var/www/html/lib/private/Files/ObjectStore/S3ObjectTrait.php","Line":75,"message":"Could not get object urn:oid:769064 for file __groupfolders/6/24-01-01 14-59-28 1225.mov","exception":{},"CustomMessage":"Could not get object urn:oid:769064 for file __groupfolders/6/24-01-01 14-59-28 1225.mov"}}
{"reqId":"s6MMhsgognRjj6U8YDz8","level":3,"time":"2024-10-01T05:48:57+00:00","remoteAddr":"10.27.1.1","user":"mattermost-bnrda-ix5z9ksay38488pmnefdehnmer","app":"objectstore","method":"GET","url":"/remote.php/dav/files/mattermost-bnrda-ix5z9ksay38488pmnefdehnmer/BNRDA%20Video%20Upload/24-01-01%2014-59-28%201225.mov","message":"Could not get object urn:oid:769064 for file __groupfolders/6/24-01-01 14-59-28 1225.mov","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:130.0) Gecko/20100101 Firefox/130.0","version":"30.0.0.14","exception":{"Exception":"Exception","Message":"Failed to read object urn:oid:769064","Code":0,"Trace":[{"file":"/var/www/html/lib/private/Files/ObjectStore/ObjectStoreStorage.php","line":308,"function":"readObject","class":"OC\\Files\\ObjectStore\\S3","type":"->","args":["urn:oid:769064"]},{"file":"/var/www/html/lib/private/Files/Storage/Wrapper/Wrapper.php","line":283,"function":"fopen","class":"OC\\Files\\ObjectStore\\ObjectStoreStorage","type":"->","args":["__groupfolders/6/24-01-01 14-59-28 1225.mov","r"]},{"file":"/var/www/html/lib/private/Files/Storage/Wrapper/Jail.php","line":288,"function":"fopen","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->","args":["__groupfolders/6/24-01-01 14-59-28 1225.mov","r"]},{"file":"/var/www/html/lib/private/Files/Storage/Wrapper/Quota.php","line":153,"function":"fopen","class":"OC\\Files\\Storage\\Wrapper\\Jail","type":"->","args":["24-01-01 14-59-28 1225.mov","r"]},{"file":"/var/www/html/lib/private/Files/Storage/Wrapper/Wrapper.php","line":283,"function":"fopen","class":"OC\\Files\\Storage\\Wrapper\\Quota","type":"->","args":["24-01-01 14-59-28 1225.mov","r"]},{"file":"/var/www/html/lib/private/Files/Storage/Wrapper/PermissionsMask.php","line":97,"function":"fopen","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->","args":["24-01-01 14-59-28 1225.mov","r"]},{"file":"/var/www/html/lib/private/Files/Storage/Wrapper/Wrapper.php","line":283,"function":"fopen","class":"OC\\Files\\Storage\\Wrapper\\PermissionsMask","type":"->","args":["24-01-01 14-59-28 1225.mov","r"]},{"file":"/var/www/html/lib/private/Files/View.php","line":1134,"function":"fopen","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->","args":["24-01-01 14-59-28 1225.mov","r"]},{"file":"/var/www/html/lib/private/Files/View.php","line":962,"function":"basicOperation","class":"OC\\Files\\View","type":"->","args":["fopen","/BNRDA Video Upload/24-01-01 14-59-28 1225.mov",["read"],"r"]},{"file":"/var/www/html/apps/dav/lib/Connector/Sabre/File.php","line":457,"function":"fopen","class":"OC\\Files\\View","type":"->","args":["BNRDA Video Upload/24-01-01 14-59-28 1225.mov","r"]},{"file":"/var/www/html/3rdparty/sabre/dav/lib/DAV/CorePlugin.php","line":85,"function":"get","class":"OCA\\DAV\\Connector\\Sabre\\File","type":"->","args":[]},{"file":"/var/www/html/3rdparty/sabre/event/lib/WildcardEmitterTrait.php","line":89,"function":"httpGet","class":"Sabre\\DAV\\CorePlugin","type":"->","args":[["Sabre\\HTTP\\Request"],["Sabre\\HTTP\\Response"]]},{"file":"/var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php","line":472,"function":"emit","class":"Sabre\\DAV\\Server","type":"->","args":["method:GET",[["Sabre\\HTTP\\Request"],["Sabre\\HTTP\\Response"]]]},{"file":"/var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php","line":253,"function":"invokeMethod","class":"Sabre\\DAV\\Server","type":"->","args":[["Sabre\\HTTP\\Request"],["Sabre\\HTTP\\Response"]]},{"file":"/var/www/html/apps/dav/lib/Server.php","line":370,"function":"start","class":"Sabre\\DAV\\Server","type":"->","args":[]},{"file":"/var/www/html/apps/dav/appinfo/v2/remote.php","line":19,"function":"exec","class":"OCA\\DAV\\Server","type":"->","args":[]},{"file":"/var/www/html/remote.php","line":146,"args":["/var/www/html/apps/dav/appinfo/v2/remote.php"],"function":"require_once"}],"File":"/var/www/html/lib/private/Files/ObjectStore/S3ObjectTrait.php","Line":75,"message":"Could not get object urn:oid:769064 for file __groupfolders/6/24-01-01 14-59-28 1225.mov","exception":{},"CustomMessage":"Could not get object urn:oid:769064 for file __groupfolders/6/24-01-01 14-59-28 1225.mov"}}
{"reqId":"s6MMhsgognRjj6U8YDz8","level":3,"time":"2024-10-01T05:48:57+00:00","remoteAddr":"10.27.1.1","user":"mattermost-bnrda-ix5z9ksay38488pmnefdehnmer","app":"webdav","method":"GET","url":"/remote.php/dav/files/mattermost-bnrda-ix5z9ksay38488pmnefdehnmer/BNRDA%20Video%20Upload/24-01-01%2014-59-28%201225.mov","message":"Could not open file","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:130.0) Gecko/20100101 Firefox/130.0","version":"30.0.0.14","exception":{"Exception":"Sabre\\DAV\\Exception\\ServiceUnavailable","Message":"Could not open file","Code":0,"Trace":[{"file":"/var/www/html/3rdparty/sabre/dav/lib/DAV/CorePlugin.php","line":85,"function":"get","class":"OCA\\DAV\\Connector\\Sabre\\File","type":"->","args":[]},{"file":"/var/www/html/3rdparty/sabre/event/lib/WildcardEmitterTrait.php","line":89,"function":"httpGet","class":"Sabre\\DAV\\CorePlugin","type":"->","args":[["Sabre\\HTTP\\Request"],["Sabre\\HTTP\\Response"]]},{"file":"/var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php","line":472,"function":"emit","class":"Sabre\\DAV\\Server","type":"->","args":["method:GET",[["Sabre\\HTTP\\Request"],["Sabre\\HTTP\\Response"]]]},{"file":"/var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php","line":253,"function":"invokeMethod","class":"Sabre\\DAV\\Server","type":"->","args":[["Sabre\\HTTP\\Request"],["Sabre\\HTTP\\Response"]]},{"file":"/var/www/html/apps/dav/lib/Server.php","line":370,"function":"start","class":"Sabre\\DAV\\Server","type":"->","args":[]},{"file":"/var/www/html/apps/dav/appinfo/v2/remote.php","line":19,"function":"exec","class":"OCA\\DAV\\Server","type":"->","args":[]},{"file":"/var/www/html/remote.php","line":146,"args":["/var/www/html/apps/dav/appinfo/v2/remote.php"],"function":"require_once"}],"File":"/var/www/html/apps/dav/lib/Connector/Sabre/File.php","Line":463,"message":"Could not open file","exception":{},"CustomMessage":"Could not open file"}}
{"reqId":"AH29rbeboMLyOpfwxiEM","level":3,"time":"2024-10-01T05:48:57+00:00","remoteAddr":"10.27.1.1","user":"mattermost-bnrda-ix5z9ksay38488pmnefdehnmer","app":"webdav","method":"GET","url":"/remote.php/dav/files/mattermost-bnrda-ix5z9ksay38488pmnefdehnmer/BNRDA%20Video%20Upload/24-01-01%2014-59-28%201225.mov","message":"Could not open file","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:130.0) Gecko/20100101 Firefox/130.0","version":"30.0.0.14","exception":{"Exception":"Sabre\\DAV\\Exception\\ServiceUnavailable","Message":"Could not open file","Code":0,"Trace":[{"file":"/var/www/html/3rdparty/sabre/dav/lib/DAV/CorePlugin.php","line":85,"function":"get","class":"OCA\\DAV\\Connector\\Sabre\\File","type":"->","args":[]},{"file":"/var/www/html/3rdparty/sabre/event/lib/WildcardEmitterTrait.php","line":89,"function":"httpGet","class":"Sabre\\DAV\\CorePlugin","type":"->","args":[["Sabre\\HTTP\\Request"],["Sabre\\HTTP\\Response"]]},{"file":"/var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php","line":472,"function":"emit","class":"Sabre\\DAV\\Server","type":"->","args":["method:GET",[["Sabre\\HTTP\\Request"],["Sabre\\HTTP\\Response"]]]},{"file":"/var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php","line":253,"function":"invokeMethod","class":"Sabre\\DAV\\Server","type":"->","args":[["Sabre\\HTTP\\Request"],["Sabre\\HTTP\\Response"]]},{"file":"/var/www/html/apps/dav/lib/Server.php","line":370,"function":"start","class":"Sabre\\DAV\\Server","type":"->","args":[]},{"file":"/var/www/html/apps/dav/appinfo/v2/remote.php","line":19,"function":"exec","class":"OCA\\DAV\\Server","type":"->","args":[]},{"file":"/var/www/html/remote.php","line":146,"args":["/var/www/html/apps/dav/appinfo/v2/remote.php"],"function":"require_once"}],"File":"/var/www/html/apps/dav/lib/Connector/Sabre/File.php","Line":463,"message":"Could not open file","exception":{},"CustomMessage":"Could not open file"}}

Web server error log

Web server error log ``` Insert your webserver log here ```

Nextcloud log (data/nextcloud.log)

Nextcloud log ``` Insert your Nextcloud log here ```

Browser log

Browser log ``` Insert your browser log here, this could for example include: a) The javascript console log b) The network log c) ... ```
provokateurin commented 2 days ago

Not groupfolder specific, this is an S3 issue.

fcharlaix-opendsi commented 2 days ago

I have the same issue with Nextcloud 29.0.7 on Debian 12, PHP 8.2 without Docker, using OVH Object Storage (their backend is not MinIO as far as I know).

On a Nextcloud 28.0.9 with Group Folders 16.0.10 and MinIO, the issue is not present.

kesselb commented 2 days ago

@fcharlaix-opendsi would you mind opening a support ticket?

If the problem occurs with 29.0.7, but not 28.0.9 then a relevant change is: https://github.com/nextcloud/server/pull/46013

A way to test that could be to comment the moveFromStorage method in ObjectStoreStorage to use, as before the patch, just the default implemention. Note this will reintroduce the issue that were fixed by it.

infra-monkey commented 2 days ago

@kesselb in v30.0.0 commenting out the method and restarting the container seems to fix this issue. I'll do more thorough tests tonight