search

nextcloud / groupfolders

📁👩‍👩‍👧‍👦 Admin-configured folders shared by everyone in a group. https://github.com/nextcloud-releases/groupfolders
https://apps.nextcloud.com/apps/groupfolders
288 stars 87 forks source link

Non-administrator accounts time out when attempting to download large folders #1529

Open BBauman opened 4 years ago

BBauman commented 4 years ago

Steps to reproduce

  1. Create a group folder
  2. Upload a large, dense folder structure to the group folder (ours is 3 layers deep, several thousand files, and just over 40 GB.
  3. Share the folder uploaded in step 2 with a normal user.
  4. Attempt to download the folder from step 2 with the normal user account, and with the administrator account.

Expected behaviour

Both users should be able to download a zip of the large, 40+ GB folder.

Actual behaviour

Only the Administrator is reliably able to download the folder, after waiting perhaps 5 seconds. The normal user will continue waiting until the web server times out with a 504 (tested up to 5 minutes). On very rare occasions the normal user may succeed.

Server configuration

Operating system: Ubuntu 18.04.1

Web server: Nginx 1.14.0

Database: MariaDB 10.3.12

PHP version: PHP-FPM 7.2

Nextcloud version: (see Nextcloud admin page) 18.0.4

Updated from an older Nextcloud/ownCloud or fresh install: Updated sequentially from 15.

Where did you install Nextcloud from: Downloaded from https://nextcloud.com/install.

Signing status:

Signing status ``` No errors have been found. ```

List of activated apps:

App list ``` Enabled: - accessibility: 1.4.0 - activity: 2.11.0 - bruteforcesettings: 1.6.0 - cloud_federation_api: 1.1.0 - comments: 1.8.0 - dav: 1.14.0 - federatedfilesharing: 1.8.0 - files: 1.13.1 - files_pdfviewer: 1.7.0 - files_rightclick: 0.15.2 - files_sharing: 1.10.1 - files_texteditor: 2.11.0 - files_trashbin: 1.8.0 - files_versions: 1.11.0 - files_videoplayer: 1.7.0 - groupfolders: 6.0.6 - logreader: 2.3.0 - lookup_server_connector: 1.6.0 - nextcloud_announcements: 1.7.0 - notifications: 2.6.0 - oauth2: 1.6.0 - password_policy: 1.8.0 - photos: 1.0.0 - privacy: 1.2.0 - provisioning_api: 1.8.0 - recommendations: 0.6.0 - serverinfo: 1.8.0 - settings: 1.0.0 - sharebymail: 1.8.0 - support: 1.1.0 - survey_client: 1.6.0 - systemtags: 1.8.0 - text: 2.0.0 - theming: 1.9.0 - twofactor_backupcodes: 1.7.0 - updatenotification: 1.8.0 - viewer: 1.2.0 - workflowengine: 2.0.0 Disabled: - admin_audit - contacts - encryption - federation - files_external - firstrunwizard - user_ldap ```

Nextcloud configuration:

Config report ``` { "system": { "instanceid": "***REMOVED SENSITIVE VALUE***", "passwordsalt": "***REMOVED SENSITIVE VALUE***", "secret": "***REMOVED SENSITIVE VALUE***", "trusted_domains": [ "***REMOVED SENSITIVE VALUE***" ], "datadirectory": "***REMOVED SENSITIVE VALUE***", "dbtype": "mysql", "version": "18.0.4.2", "overwrite.cli.url": "***REMOVED SENSITIVE VALUE***", "dbname": "***REMOVED SENSITIVE VALUE***", "dbhost": "***REMOVED SENSITIVE VALUE***", "dbport": "", "dbtableprefix": "oc_", "mysql.utf8mb4": true, "dbuser": "***REMOVED SENSITIVE VALUE***", "dbpassword": "***REMOVED SENSITIVE VALUE***", "installed": true, "mail_smtpmode": "smtp", "mail_sendmailmode": "smtp", "mail_from_address": "***REMOVED SENSITIVE VALUE***", "mail_domain": "***REMOVED SENSITIVE VALUE***", "mail_smtphost": "***REMOVED SENSITIVE VALUE***", "mail_smtpport": "465", "skeletondirectory": "", "updater.release.channel": "stable", "mail_smtpsecure": "ssl", "mail_smtpauth": 1, "mail_smtpauthtype": "LOGIN", "mail_smtpname": "***REMOVED SENSITIVE VALUE***", "mail_smtppassword": "***REMOVED SENSITIVE VALUE***", "updater.secret": "***REMOVED SENSITIVE VALUE***", "maintenance": false, "loglevel": 2 } } ```

Are you using external storage, if yes which one: local/smb/sftp/... No Are you using encryption: yes/no No Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/... No

Client configuration

Browser: Chrome, Edge, Safari Operating system: Windows 10, OSX 13 & 14

Logs

Web server error log

Web server error log ``` Sample timeout from Nginx waiting for PHP to fulfill the download request: 2020/04/24 11:54:41 [error] 22555#22555: *19226 upstream timed out (110: Connection timed out) while reading response header from upstream, client: [redacted], server: [redacted], request: "GET /index.php/apps/files/ajax/download.php?dir=%2F&files=[redacted]&downloadStartSecret=[redacted] HTTP/2.0", upstream: "fastcgi://unix:/var/run/php/php7.2-fpm.sock", host: "[redacted]", referrer: "https://[redacted]/apps/files/?dir=/&fileid=411233" Sample PHP-FPM log entry: [24-Apr-2020 14:56:05] NOTICE: child 2883 stopped for tracing [24-Apr-2020 14:56:05] NOTICE: about to trace 2883 [24-Apr-2020 14:56:05] NOTICE: finished trace of 2883 [24-Apr-2020 14:56:25] WARNING: [pool www] child 5082, script '/srv/www/nextcloud/index.php' (request: "GET /index.php?lastReqId=SlENCoyEZOV2EK7omflZ") executing too slow (13.087787 sec), logging Sample PHP-FPM slow log entry: [24-Apr-2020 14:50:09] [pool www] pid 2885 script_filename = /srv/www/nextcloud/index.php [0x00007feb18a1d690] execute() /srv/www/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOStatement.php:117 [0x00007feb18a1d610] execute() /srv/www/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php:1054 [0x00007feb18a1d530] executeUpdate() /srv/www/nextcloud/lib/private/DB/Connection.php:220 [0x00007feb18a1d4a0] executeUpdate() /srv/www/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Query/QueryBuilder.php:203 [0x00007feb18a1d430] execute() /srv/www/nextcloud/lib/private/DB/QueryBuilder/QueryBuilder.php:215 [0x00007feb18a1d370] execute() /srv/www/nextcloud/lib/private/DB/Adapter.php:142 [0x00007feb18a1d2a0] insertIgnoreConflict() /srv/www/nextcloud/lib/private/DB/Connection.php:265 [0x00007feb18a1d220] insertIgnoreConflict() /srv/www/nextcloud/lib/private/Lock/DBLockingProvider.php:141 [0x00007feb18a1d190] initLockField() /srv/www/nextcloud/lib/private/Lock/DBLockingProvider.php:187 [0x00007feb18a1d0d0] acquireLock() /srv/www/nextcloud/lib/private/Files/Storage/Common.php:747 [0x00007feb18a1cff0] acquireLock() /srv/www/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php:593 [0x00007feb18a1cf60] acquireLock() /srv/www/nextcloud/lib/private/Files/Storage/Wrapper/Jail.php:460 [0x00007feb18a1ced0] acquireLock() /srv/www/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php:593 [0x00007feb18a1ce40] acquireLock() /srv/www/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php:593 [0x00007feb18a1cdb0] acquireLock() /srv/www/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php:593 [0x00007feb18a1cd20] acquireLock() /srv/www/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php:593 [0x00007feb18a1cc90] acquireLock() /srv/www/nextcloud/apps/files_sharing/lib/SharedStorage.php:423 [0x00007feb18a1cbc0] acquireLock() /srv/www/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php:593 [0x00007feb18a1cb30] acquireLock() /srv/www/nextcloud/lib/private/Files/View.php:1947 [0x00007feb18a1ca50] lockPath() /srv/www/nextcloud/lib/private/Files/View.php:2059 ```

Nextcloud log (data/nextcloud.log)

Nextcloud log ``` None - this operation produces no logs during the 5 minutes it takes for Nginx to timeout, nor for several minutes afterwards. ```

Browser log

Browser log ``` No Javascript logs between clicking the download link and timeout Network calls: https://[redacted]/index.php/apps/files/ajax/download.php?dir=%2F&files=[redacted]&downloadStartSecret=[redacted] Then: https://[redacted]/ocs/v2.php/apps/notifications/api/v2/notifications repeats until timeout. ```
joshtrichards commented 1 year ago

Using the (optional) Zipper files_zip app might address this. It uses a background job for the archive generation:

https://github.com/nextcloud/files_zip#zipper

joshtrichards commented 6 days ago

Only the Administrator is reliably able to download the folder, after waiting perhaps 5 seconds. The normal user will continue waiting until the web server times out with a 504 (tested up to 5 minutes). On very rare occasions the normal user may succeed.

I can understand why there might be a timeout for this type of action, but I can't think of a good reason why you'd see different behavior between an admin and a user account. Was this testing performed in serial fashion (one completes then the other account is tested) and from the same client/location?