search

nextcloud / server

☁️ Nextcloud server, a safe home for all your data
https://nextcloud.com
GNU Affero General Public License v3.0
26.92k stars 4.02k forks source link

S3-compatible Object Store as Primary Storage - Downloading empty files results in 503 #22164

Closed lauhayden closed 4 years ago

lauhayden commented 4 years ago

How to use GitHub

Steps to reproduce

  1. Setup a Nextcloud instance using any s3-compatible object store as primary storage. I used DigitalOcean Spaces.
  2. Upload an empty (0-byte size) file using the Nextcloud client or web UI. The upload succeeds.
  3. Try to download that file from the web UI or sync the file to another computer. This results in a 503 Service Unavailable error.

Expected behaviour

The empty file should behave the same as a non-empty file.

Actual behaviour

The empty file cannot be downloaded via the web UI or synced via Nextcloud client to another computer. Clients that run into this error stop syncing.

Server configuration

Operating system: Debian 10

Web server: nginx/1.14.2

Database: mariadb Ver 15.1 Distrib 10.3.23-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2

PHP version: PHP 7.3.19-1~deb10u1

Nextcloud version: 19.0.1

Updated from an older Nextcloud/ownCloud or fresh install: Updated from older Nextcloud (18, i believe).

Where did you install Nextcloud from: Nextcloud was manually installed via tarball. All dependencies were installed, then the tarball was downloaded from official Nextcloud website and extracted to /var/www/nextcloud.

Signing status:

Signing status ``` No errors have been found. ```

List of activated apps:

App list ``` Enabled: - accessibility: 1.5.0 - activity: 2.12.0 - bruteforcesettings: 1.6.0 - cloud_federation_api: 1.2.0 - comments: 1.9.0 - contactsinteraction: 1.0.0 - dav: 1.15.0 - federatedfilesharing: 1.9.0 - federation: 1.9.0 - files: 1.14.0 - files_pdfviewer: 1.8.0 - files_rightclick: 0.16.0 - files_sharing: 1.11.0 - files_trashbin: 1.9.0 - files_versions: 1.12.0 - files_videoplayer: 1.8.0 - firstrunwizard: 2.8.0 - logreader: 2.4.0 - lookup_server_connector: 1.7.0 - nextcloud_announcements: 1.8.0 - notifications: 2.7.0 - oauth2: 1.7.0 - password_policy: 1.9.1 - photos: 1.1.0 - privacy: 1.3.0 - provisioning_api: 1.9.0 - recommendations: 0.7.0 - serverinfo: 1.9.0 - settings: 1.1.0 - sharebymail: 1.9.0 - spreed: 9.0.3 - support: 1.2.1 - survey_client: 1.7.0 - systemtags: 1.9.0 - text: 3.0.1 - theming: 1.10.0 - twofactor_backupcodes: 1.8.0 - updatenotification: 1.9.0 - viewer: 1.3.0 - workflowengine: 2.1.0 Disabled: - admin_audit - encryption - files_external - previewgenerator - user_ldap ```

Nextcloud configuration:

Config report ``` { "system": { "objectstore": { "class": "OC\\Files\\ObjectStore\\S3", "arguments": { "bucket": "my-domain-nextcloud", "autocreate": true, "key": "***REMOVED SENSITIVE VALUE***", "secret": "***REMOVED SENSITIVE VALUE***", "hostname": "nyc3.digitaloceanspaces.com", "port": 443, "use_ssl": true, "region": "nyc3", "use_path_style": false } }, "instanceid": "***REMOVED SENSITIVE VALUE***", "passwordsalt": "***REMOVED SENSITIVE VALUE***", "secret": "***REMOVED SENSITIVE VALUE***", "trusted_domains": [ "cloud.my-domain.com" ], "datadirectory": "***REMOVED SENSITIVE VALUE***", "dbtype": "mysql", "version": "19.0.1.1", "overwrite.cli.url": "https:\/\/cloud.my-domain.com", "dbname": "***REMOVED SENSITIVE VALUE***", "dbhost": "***REMOVED SENSITIVE VALUE***", "dbport": "", "dbtableprefix": "oc_", "mysql.utf8mb4": true, "dbuser": "***REMOVED SENSITIVE VALUE***", "dbpassword": "***REMOVED SENSITIVE VALUE***", "installed": true, "mail_smtpmode": "sendmail", "mail_sendmailmode": "smtp", "mail_from_address": "***REMOVED SENSITIVE VALUE***", "mail_domain": "***REMOVED SENSITIVE VALUE***", "memcache.local": "\\OC\\Memcache\\APCu", "maintenance": false, "theme": "", "loglevel": 2 } } ```

Are you using external storage, if yes which one: None

Are you using encryption: No

Are you using an external user-backend, if yes which one: No

Client configuration

Browser: Firefox 79.0

Operating system: Ubuntu 18.04 LTS

Logs

Web server error log

Web server error log ``` ```

Nextcloud log (data/nextcloud.log)

Nextcloud log ``` {"reqId":"Zc9ah1gG1D0D6cu58ACv","level":3,"time":"2020-08-10T00:09:00+00:00","remoteAddr":"2607:fea8:79f:fa13:7179:8f59:9e55:d1f9","user":"hayden","app":"PHP","method":"GET","url":"/remote.php/webdav/test_empty","message":"fopen(https://deltaepsilon-nextcloud.nyc3.digitaloceanspaces.com/urn%3Aoid%3A34194): failed to open stream: HTTP request failed! HTTP/1.1 416 Requested Range Not Satisfiable\r\n at /var/www/nextcloud/lib/private/Files/ObjectStore/S3ObjectTrait.php#73","userAgent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:79.0) Gecko/20100101 Firefox/79.0","version":"19.0.1.1"} {"reqId":"Zc9ah1gG1D0D6cu58ACv","level":3,"time":"2020-08-10T00:09:00+00:00","remoteAddr":"2607:fea8:79f:fa13:7179:8f59:9e55:d1f9","user":"hayden","app":"PHP","method":"GET","url":"/remote.php/webdav/test_empty","message":"fopen(httpseek://): failed to open stream: "OC\\Files\\Stream\\SeekableHttpStream::stream_open" call failed at /var/www/nextcloud/lib/private/Files/Stream/SeekableHttpStream.php#67","userAgent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:79.0) Gecko/20100101 Firefox/79.0","version":"19.0.1.1"} {"reqId":"Zc9ah1gG1D0D6cu58ACv","level":4,"time":"2020-08-10T00:09:00+00:00","remoteAddr":"2607:fea8:79f:fa13:7179:8f59:9e55:d1f9","user":"hayden","app":"webdav","method":"GET","url":"/remote.php/webdav/test_empty","message":{"Exception":"Sabre\\DAV\\Exception\\ServiceUnavailable","Message":"Could not open file","Code":0,"Trace":[{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php","line":90,"function":"get","class":"OCA\\DAV\\Connector\\Sabre\\File","type":"->","args":[]},{"file":"/var/www/nextcloud/3rdparty/sabre/event/lib/WildcardEmitterTrait.php","line":89,"function":"httpGet","class":"Sabre\\DAV\\CorePlugin","type":"->","args":[{"__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":474,"function":"emit","class":"Sabre\\DAV\\Server","type":"->","args":["method:GET",[{"__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]]},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":251,"function":"invokeMethod","class":"Sabre\\DAV\\Server","type":"->","args":[{"__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":319,"function":"start","class":"Sabre\\DAV\\Server","type":"->","args":[]},{"file":"/var/www/nextcloud/apps/dav/appinfo/v1/webdav.php","line":82,"function":"exec","class":"Sabre\\DAV\\Server","type":"->","args":[]},{"file":"/var/www/nextcloud/remote.php","line":167,"args":["/var/www/nextcloud/apps/dav/appinfo/v1/webdav.php"],"function":"require_once"}],"File":"/var/www/nextcloud/apps/dav/lib/Connector/Sabre/File.php","Line":428,"CustomMessage":"--"},"userAgent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:79.0) Gecko/20100101 Firefox/79.0","version":"19.0.1.1"} {"reqId":"Zc9ah1gG1D0D6cu58ACv","level":3,"time":"2020-08-10T00:09:00+00:00","remoteAddr":"2607:fea8:79f:fa13:7179:8f59:9e55:d1f9","user":"hayden","app":"PHP","method":"GET","url":"/remote.php/webdav/test_empty","message":"Undefined index: debugMode at /var/www/nextcloud/core/templates/exception.php#26","userAgent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:79.0) Gecko/20100101 Firefox/79.0","version":"19.0.1.1"} ```

Browser log

Browser log ``` Nothing relevant for debugging, other than the "Internal Server Error" splash page, and 503 Service Unavailable error code. ```
jdhirst commented 4 years ago

Can confirm this also happens on Wasabi. There needs to be a method defined to store these files in the database but not attempting to upload them to S3.

This is definitely a breaking feature as it stops our migration to primary object storage as sync clients will fail to sync due to these zero-byte file errors.

lauhayden commented 4 years ago

Can confirm this also happens on Wasabi.

I suspected that this was not specific to DO Spaces, but was too lazy to spin up an instance to test. Thanks for confirming!

jdhirst commented 4 years ago

@lauhayden I'm not sure if it is relating to all object storage or just S3-based (since DO Spaces and Wasabi both use S3). I can test with Swift and let you know.

jdhirst commented 4 years ago

@lauhayden I have tested this with Swift and have confirmed it works fine. This seems to be related to the S3 API implementation itself.

Config used:

  'objectstore' => 
  array (
    'class' => 'OC\\Files\\ObjectStore\\Swift',
    'arguments' => 
    array (
      'autocreate' => false,
      'user' => 
      array (
        'name' => 'nextcloud-dev',
        'password' => 'secret',
        'domain' => 
        array (
          'name' => 'Default',
        ),
      ),
      'scope' => 
      array (
        'project' => 
        array (
          'name' => 'nextcloud-dev',
          'domain' => 
          array (
            'name' => 'Default',
          ),
        ),
      ),
      'serviceName' => 'swift',
      'region' => 'eu-north',
      'url' => 'https://cloud.example.net/identity/v3',
      'bucket' => 'nextcloud-dev',
    ),
  ),

Test:

➜  ~ touch zerobyte-test
➜  ~ rclone copy zerobyte-test nextcloud-dev:
➜  ~ rm -f zerobyte-test
➜  ~ rclone ls nextcloud-dev:
        0 zerobyte-test
➜  ~ rclone copy nextcloud-dev:zerobyte-test . 
➜  ~ ls
 zerobyte-test
➜  ~

Additionally, here are my test results when using S3 (Wasabi):

➜  ~ touch zerobyte-test
➜  ~ rclone copy zerobyte-test n:
➜  ~ rm -f zerobyte-test 
➜  ~ rclone copy n:zerobyte-test .
^C (hangs here and tries multiple times

Server Logs:


{
  "reqId": "d8C7es6JaAIIsbXo7GDo",
  "level": 3,
  "time": "2020-08-10T15:19:09+00:00",
  "remoteAddr": "123.123.123.123",
  "user": "example",
  "app": "PHP",
  "method": "GET",
  "url": "/remote.php/webdav/zerobyte-test",
  "message": "fopen(https://s3.eu-central-1.wasabisys.com/somebucket/urn%3Aoid%1A4627403): failed to open stream: HTTP request failed! HTTP/1.1 416 Requested Range Not Satisfiable\r\n at /var/www/html/lib/private/Files/ObjectStore/S3ObjectTrait.php#73",
  "userAgent": "rclone/v1.50.2",
  "version": "19.0.1.1"
}
{
  "reqId": "d8C7es6JaAIIsbXo7GDo",
  "level": 3,
  "time": "2020-08-10T15:19:09+00:00",
  "remoteAddr": "123.123.123.123",
  "user": "example",
  "app": "PHP",
  "method": "GET",
  "url": "/remote.php/webdav/zerobyte-test",
  "message": "fopen(httpseek://): failed to open stream: "OC\\Files\\Stream\\SeekableHttpStream::stream_open" call failed at /var/www/html/lib/private/Files/Stream/SeekableHttpStream.php#67",
  "userAgent": "rclone/v1.50.2",
  "version": "19.0.1.1"
}
{
  "reqId": "d8C7es6JaAIIsbXo7GDo",
  "level": 4,
  "time": "2020-08-10T15:19:09+00:00",
  "remoteAddr": "123.123.123.123",
  "user": "example",
  "app": "webdav",
  "method": "GET",
  "url": "/remote.php/webdav/zerobyte-test",
  "message": {
    "Exception": "Sabre\\DAV\\Exception\\ServiceUnavailable",
    "Message": "Could not open file",
    "Code": 0,
    "Trace": [
      {
        "file": "/var/www/html/3rdparty/sabre/dav/lib/DAV/CorePlugin.php",
        "line": 90,
        "function": "get",
        "class": "OCA\\DAV\\Connector\\Sabre\\File",
        "type": "->",
        "args": []
      },
      {
        "file": "/var/www/html/3rdparty/sabre/event/lib/WildcardEmitterTrait.php",
        "line": 89,
        "function": "httpGet",
        "class": "Sabre\\DAV\\CorePlugin",
        "type": "->",
        "args": [
          {
            "__class__": "Sabre\\HTTP\\Request"
          },
          {
            "__class__": "Sabre\\HTTP\\Response"
          }
        ]
      },
      {
        "file": "/var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php",
        "line": 474,
        "function": "emit",
        "class": "Sabre\\DAV\\Server",
        "type": "->",
        "args": [
          "method:GET",
          [
            {
              "__class__": "Sabre\\HTTP\\Request"
            },
            {
              "__class__": "Sabre\\HTTP\\Response"
            }
          ]
        ]
      },
      {
        "file": "/var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php",
        "line": 251,
        "function": "invokeMethod",
        "class": "Sabre\\DAV\\Server",
        "type": "->",
        "args": [
          {
            "__class__": "Sabre\\HTTP\\Request"
          },
          {
            "__class__": "Sabre\\HTTP\\Response"
          }
        ]
      },
      {
        "file": "/var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php",
        "line": 319,
        "function": "start",
        "class": "Sabre\\DAV\\Server",
        "type": "->",
        "args": []
      },
      {
        "file": "/var/www/html/apps/dav/appinfo/v1/webdav.php",
        "line": 82,
        "function": "exec",
        "class": "Sabre\\DAV\\Server",
        "type": "->",
        "args": []
      },
      {
        "file": "/var/www/html/remote.php",
        "line": 167,
        "args": [
          "/var/www/html/apps/dav/appinfo/v1/webdav.php"
        ],
        "function": "require_once"
      }
    ],
    "File": "/var/www/html/apps/dav/lib/Connector/Sabre/File.php",
    "Line": 428,
    "CustomMessage": "--"
  },
  "userAgent": "rclone/v1.50.2",
  "version": "19.0.1.1"
}
lauhayden commented 4 years ago

@jdhirst1 thanks, updating.

solracsf commented 4 years ago

Duplicate of https://github.com/nextcloud/server/issues/22029?

lauhayden commented 4 years ago

@acsfer Yep, definitely a duplicate. Thanks for linking - I couldn't find anything on initial search.