Gallery fails to load with large amount of files/folders

paulijar commented 7 years ago

Bug report

Steps to reproduce

Have lots of files and folders in your Nextcloud account. My test account had 58832 files in 8549 folders. Only two of the folders had any image files and there were less than 200 of them in total.
Try to open the Gallery app from the root of your Nextcloud account.

Expected behavior

Gallery should open and show the folders with images in them

Actual behavior

Gallery shows only the loading spinner indefinitely. After one minute, the request for endpoint /gallery/files/list?location=&mediatypes=image%2Fpng%3Bimage%2Fjpeg%3Bimage%2Fgif%3Bimage%2Ftiff&features=&etag times out (http error 504).

The gallery.js script tries to show an error message but even that fails because of another bug. The code is like this:

var result = xhr.responseJSON;
var albumPath = decodeURIComponent(currentLocation);
var message;
if (result === null) {
    message = t('gallery', 'There was a problem reading files from this album');
} else {
    message = result.message;
}

... but the result is not null but undefined. Hence, the script execution is terminated.

If I navigate to some sub-folder in the Files app and then open the Gallery app with the "grid view" button, then the application opens correctly. With a folder containing 3640 files and 533 folders, the gallery/files/list service took 12 seconds to process on the server. Assuming linear timing properties, around 18000 files and/or 2600 folders might be the maximum numbers which can be handled by the Gallery app. On more high-power server, the limits could be somewhat higher but probably not by an order of magnitude.

Obviously, the Gallery app is doing lots of unnecessary processing with the non-image files. It should be able to discard folders not containing any image files in matter of seconds. I have used the same folder structure as test content when developing the Music app. The 58k+ files are actually audio files and I can retrieve them all in around 16 seconds with Folder.searchByMime from the root folder. I also tested that retrieving the <200 image files the same way would take only some hundreds of milliseconds.

Server configuration

Operating system: Linux raspberrypi 4.9.35-v7+ #1014 SMP Fri Jun 30 14:47:43 BST 2017 armv7l

Web server: nginx/1.6.2 (fpm-fcgi)

Database: mysql 5.5.55

PHP version: 5.6.30-0+deb8u1

cloud server: Nextcloud

cloud version: 12.0.3

Updated from an older installation or fresh install: Updated

List of activated apps:

Enabled apps

``` - activity: 2.5.2 - comments: 1.2.0 - dav: 1.3.0 - federatedfilesharing: 1.2.0 - federation: 1.2.0 - files: 1.7.2 - files_external: 1.3.0 - files_pdfviewer: 1.1.1 - files_sharing: 1.4.0 - files_texteditor: 2.4.1 - files_trashbin: 1.2.0 - files_versions: 1.5.0 - files_videoplayer: 1.1.0 - firstrunwizard: 2.1 - gallery: 17.0.0 - issuetemplate: 0.2.2 - logreader: 2.0.0 - lookup_server_connector: 1.0.0 - metadata: 0.5.0 - music: 0.5.3 - nextcloud_announcements: 1.1 - notifications: 2.0.0 - oauth2: 1.0.5 - password_policy: 1.2.2 - provisioning_api: 1.2.0 - serverinfo: 1.2.0 - sharebymail: 1.2.0 - survey_client: 1.0.0 - systemtags: 1.2.0 - theming: 1.3.0 - twofactor_backupcodes: 1.1.1 - updatenotification: 1.2.0 - workflowengine: 1.2.0 ```

Disabled apps

``` - admin_audit - audioplayer - encryption - user_external - user_ldap ```

Nextcloud configuration:

Content of config/config.php

``` { "debug": true, "instanceid": "ocvemsryqekj", "passwordsalt": "***REMOVED SENSITIVE VALUE***", "secret": "***REMOVED SENSITIVE VALUE***", "trusted_domains": [ "***REMOVED SENSITIVE VALUE***" ], "datadirectory": "\/mnt\/ext_hdd\/owncloud_data", "overwrite.cli.url": "https:\/\/***REMOVED***\/nextcloud", "dbtype": "mysql", "version": "12.0.3.3", "logtimezone": "Europe\/Helsinki", "installed": true, "memcache.local": "\\OC\\Memcache\\APCu", "filelocking.enabled": true, "memcache.locking": "\\OC\\Memcache\\Redis", "redis": { "host": "localhost", "port": 6379 }, "asset-pipeline.enabled": false, "maintenance": false, "dbname": "nextcloud_db", "dbhost": "127.0.0.1", "dbuser": "***REMOVED SENSITIVE VALUE***", "dbpassword": "***REMOVED SENSITIVE VALUE***", "preview_libreoffice_path": "\/usr\/lib\/libreoffice\/program\/soffice.bin", "preview_max_scale_factor": 1, "enabledPreviewProviders": [ "OC\\Preview\\PNG", "OC\\Preview\\JPEG", "OC\\Preview\\GIF", "OC\\Preview\\TIFF", "OC\\Preview\\MP3" ], "theme": "", "loglevel": 1, "htaccess.RewriteBase": "\/nextcloud" } ```

Are you using external storage, if yes which one: no

Are you using encryption: no

Are you using custom gallery.cnf config files: no

Web server error log

``` 2017/10/21 22:06:00 [error] 19518#0: *58392 FastCGI sent in stderr: "Primary script unknown" while reading response header from upstream, client: 91.158.223.156, server: ***REMOVED***, request: "GET /nextcloud/core/vendor/DOMPurify/dist/purify.min.js.map HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "***REMOVED***" 2017/10/21 22:06:00 [error] 19518#0: *58393 FastCGI sent in stderr: "Primary script unknown" while reading response header from upstream, client: 91.158.223.156, server: ***REMOVED***, request: "GET /nextcloud/core/vendor/blueimp-md5/js/md5.min.js.map HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "***REMOVED***" 2017/10/21 22:07:01 [error] 19518#0: *58391 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 91.158.223.156, server: ***REMOVED***, request: "GET /nextcloud/index.php/apps/gallery/files/list?location=&mediatypes=image%2Fpng%3Bimage%2Fjpeg%3Bimage%2Fgif%3Bimage%2Ftiff&features=&etag HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock", host: "***REMOVED***" ```

Cloud log

Nothing printed to the log, even if log level is debug.

Client configuration

Browser: Chrome/61.0.3163.100

Operating system: Windows 10

Browser log

``` GET https://***REMOVED***/nextcloud/index.php/apps/gallery/files/list?location=&mediatypes=image%2Fpng%3Bimage%2Fjpeg%3Bimage%2Fgif%3Bimage%2Ftiff&features=&etag 504 (Gateway Time-out) Uncaught TypeError: Cannot read property 'message' of undefined ```

linuxpete commented 6 years ago

I do see this behavior too with following hardware:

Server configuration detail

Operating system: Linux 4.4.0-104-generic #127-Ubuntu SMP Mon Dec 11 12:16:42 UTC 2017 x86_64

Webserver: nginx/1.10.3 (fpm-fcgi)

Database: mysql 5.7.20

PHP version: 7.0.22-0ubuntu0.16.04.1 Modules loaded: Core, date, libxml, openssl, pcre, zlib, filter, hash, Reflection, SPL, session, standard, cgi-fcgi, mysqlnd, PDO, xml, apcu, calendar, ctype, curl, dom, mbstring, fileinfo, ftp, gd, gettext, iconv, imagick, intl, json, exif, mcrypt, mysqli, pdo_mysql, Phar, posix, readline, shmop, SimpleXML, sockets, sysvmsg, sysvsem, sysvshm, tokenizer, wddx, xmlreader, xmlwriter, xsl, zip, Zend OPcache

Nextcloud version: 13.0.0 Beta 3 - 13.0.0.8 and 12.0.3

Updated from an older Nextcloud/ownCloud or fresh install:

Where did you install Nextcloud from: 13 beta 3, tarball 12.0.3 tarball

Signing status

Array

List of activated apps

``` Enabled: - activity: 2.6.1 - admin_audit: 1.3.0 - apporder: 0.4.1 - audioplayer: 2.2.2 - bookmarks: 0.10.1 - calendar: 1.5.7 - comments: 1.3.0 - contacts: 2.0.1 - dav: 1.4.5 - encryption: 2.0.0 - federatedfilesharing: 1.3.1 - federation: 1.3.0 - files: 1.8.0 - files_external: 1.4.1 - files_pdfviewer: 1.2.0 - files_sharing: 1.5.0 - files_texteditor: 2.5.1 - files_trashbin: 1.3.0 - files_versions: 1.6.0 - files_videoplayer: 1.2.0 - firstrunwizard: 2.2.1 - gallery: 18.0.0 - issuetemplate: 0.3.0 - logreader: 2.0.0 - lookup_server_connector: 1.1.0 - music: 0.5.5 - nextcloud_announcements: 1.2.0 - notes: 2.3.2 - notifications: 2.1.2 - oauth2: 1.1.0 - passman: 2.1.4 - password_policy: 1.3.0 - previewgenerator: 1.0.9 - provisioning_api: 1.3.0 - qownnotesapi: 17.5.0 - serverinfo: 1.3.0 - sharebymail: 1.3.0 - spreedme: 0.3.11 - survey_client: 1.1.0 - systemtags: 1.3.0 - theming: 1.4.1 - twofactor_backupcodes: 1.2.3 - updatenotification: 1.3.0 - workflowengine: 1.3.0 Disabled: - bruteforcesettings - deck - files_automatedtagging - files_downloadactivity - files_opds - files_reader - gpxedit - gpxmotion - gpxpod - keeweb - ownpad - rainloop - spreed - twofactor_totp - user_external - user_ldap ```

Configuration (config/config.php)

``` { "instanceid": "***REMOVED SENSITIVE VALUE***", "passwordsalt": "***REMOVED SENSITIVE VALUE***", "secret": "***REMOVED SENSITIVE VALUE***", "trusted_domains": [ "localhost", "13.xxxxxxx.nl", "workhorse" ], "datadirectory": "***REMOVED SENSITIVE VALUE***", "overwrite.cli.url": "13.xxxxxxxxx.nl", "dbtype": "mysql", "version": "13.0.0.8", "logtimezone": "UTC", "installed": true, "memcache.local": "\\OC\\Memcache\\APCu", "loglevel": 0, "defaultapp": "apporder", "knowledgebaseenabled": "true", "custom_csp_policy": "default-src 'self'; script-src 'self' 'unsafe-inline' 'unsafe-eval'; style-src 'self' 'unsafe-inline' data: blob:; frame-src *; img-src *; font-src 'self' data: blob:; media-src *", "maintenance": false, "dbname": "***REMOVED SENSITIVE VALUE***", "dbhost": "***REMOVED SENSITIVE VALUE***", "dbuser": "***REMOVED SENSITIVE VALUE***", "dbpassword": "***REMOVED SENSITIVE VALUE***", "theme": "", "data-fingerprint": "5c3098e6911a583ef4c3c8b6ee9c84de", "mail_from_address": "***REMOVED SENSITIVE VALUE***", "mail_smtpmode": "smtp", "mail_domain": "***REMOVED SENSITIVE VALUE***", "mail_smtpauthtype": "LOGIN", "mail_smtphost": "***REMOVED SENSITIVE VALUE***", "mail_smtpauth": 1, "mail_smtpport": "465", "mail_smtpname": "***REMOVED SENSITIVE VALUE***", "mail_smtppassword": "***REMOVED SENSITIVE VALUE***", "mail_smtpsecure": "ssl", "enabledPreviewProviders": [ "OC\\Preview\\Image", "OC\\Preview\\MP3", "OC\\Preview\\TXT", "OC\\Preview\\MarkDown", "OC\\Preview\\Epub", "OC\\Preview\\PDF", "OC\\Preview\\OpenDocument", "OC\\Preview\\StarOffice", "OC\\Preview\\MSOfficeDoc", "OC\\Preview\\MSOffice2003", "OC\\Preview\\MSOffice2007", "OC\\Preview\\FB2" ], "mysql.utf8mb4": true } ```

Are you using external storage, if yes which one: local/smb/sftp/...

Are you using encryption: no

Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/...

LDAP configuration (delete this part if not used)

``` With access to your command line run e.g.: sudo -u www-data php occ ldap:show-config from within your Nextcloud installation folder Without access to your command line download the data/owncloud.db to your local computer or access your SQL server remotely and run the select query: SELECT * FROM `oc_appconfig` WHERE `appid` = 'user_ldap'; Eventually replace sensitive data as the name/IP-address of your LDAP server or groups. ```

Client configuration

Browser: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:57.0) Gecko/20100101 Firefox/57.0

Operating system:

Logs

Browser log

``` Insert your webserver log here ```

Nextcloud log

``` Insert your Nextcloud log here ```

Browser log

Insert your browser log here, this could for example include: a) The javascript console log b) The network log c) ...

linuxpete commented 6 years ago

Yes, I the see the same behaviour on this setup, annoying bug btw:

Server configuration detail

Operating system: Linux 4.4.0-109-generic #132-Ubuntu SMP Tue Jan 9 19:52:39 UTC 2018 x86_64

Webserver: nginx/1.10.3 (fpm-fcgi)

Database: mysql 5.7.21

PHP version: 7.0.22-0ubuntu0.16.04.1 Modules loaded: Core, date, libxml, openssl, pcre, zlib, filter, hash, Reflection, SPL, session, standard, cgi-fcgi, mysqlnd, PDO, xml, apcu, calendar, ctype, curl, dom, mbstring, fileinfo, ftp, gd, gettext, iconv, imagick, intl, json, exif, mcrypt, mysqli, pdo_mysql, Phar, posix, readline, shmop, SimpleXML, sockets, sysvmsg, sysvsem, sysvshm, tokenizer, wddx, xmlreader, xmlwriter, xsl, zip, Zend OPcache

Nextcloud version: 12.0.5 - 12.0.5.3

Updated from an older Nextcloud/ownCloud or fresh install:

Where did you install Nextcloud from: unknown

Signing status

Array

List of activated apps

``` Enabled: - activity: 2.5.2 - admin_audit: 1.2.0 - apporder: 0.4.1 - audioplayer: 2.2.4 - bookmarks: 0.10.1 - bruteforcesettings: 1.0.3 - calendar: 1.5.7 - checksum: 0.3.5 - comments: 1.2.0 - contacts: 2.0.1 - dav: 1.3.1 - deck: 0.3.0 - encryption: 1.6.0 - federatedfilesharing: 1.2.0 - federation: 1.2.0 - files: 1.7.2 - files_automatedtagging: 1.2.2 - files_downloadactivity: 1.1.1 - files_external: 1.3.0 - files_fulltextsearch: 0.5.0 - files_opds: 0.8.7 - files_pdfviewer: 1.1.1 - files_reader: 1.1.5 - files_sharing: 1.4.0 - files_texteditor: 2.4.1 - files_trashbin: 1.2.0 - files_versions: 1.5.0 - files_videoplayer: 1.1.0 - firstrunwizard: 2.1 - fulltextsearch: 0.5.1 - fulltextsearch_elasticsearch: 0.5.1 - gallery: 17.0.0 - gpxedit: 0.0.7 - gpxmotion: 0.0.5 - gpxpod: 2.2.2 - issuetemplate: 0.3.0 - keeweb: 0.4.0 - logreader: 2.0.0 - lookup_server_connector: 1.0.0 - music: 0.5.5 - news: 11.0.5 - nextcloud_announcements: 1.1 - notes: 2.3.2 - notifications: 2.0.0 - oauth2: 1.0.5 - passman: 2.1.4 - password_policy: 1.2.2 - previewgenerator: 1.0.9 - provisioning_api: 1.2.0 - qownnotesapi: 17.5.0 - rainloop: 5.0.6 - serverinfo: 1.2.0 - sharebymail: 1.2.0 - spreed: 2.0.2 - spreedme: 0.3.11 - survey_client: 1.0.0 - systemtags: 1.2.0 - tasks: 0.9.5 - theming: 1.3.0 - twofactor_backupcodes: 1.1.1 - twofactor_totp: 1.3.1 - updatenotification: 1.2.0 - workflowengine: 1.2.0 Disabled: - nextant - ownpad - user_external - user_ldap ```

Configuration (config/config.php)

``` { "instanceid": "xxxxxxxxx", "passwordsalt": "***REMOVED SENSITIVE VALUE***", "secret": "***REMOVED SENSITIVE VALUE***", "trusted_domains": [ "localhost", "oc.xxxxxxxx.nl" ], "datadirectory": "\/srv\/www-data\/nextcloud", "overwrite.cli.url": "oc.xxxxxxx.nl", "dbtype": "mysql", "version": "12.0.5.3", "logtimezone": "UTC", "installed": true, "memcache.local": "\\OC\\Memcache\\APCu", "loglevel": 0, "defaultapp": "apporder", "knowledgebaseenabled": "true", "custom_csp_policy": "default-src 'self'; script-src 'self' 'unsafe-inline' 'unsafe-eval'; style-src 'self' 'unsafe-inline' data: blob:; frame-src *; img-src *; font-src 'self' data: blob:; media-src *", "maintenance": false, "dbname": "nc_database", "dbhost": "127.0.0.1", "dbuser": "***REMOVED SENSITIVE VALUE***", "dbpassword": "***REMOVED SENSITIVE VALUE***", "theme": "", "data-fingerprint": "5c3098e6911a583ef4c3c8b6ee9c84de", "mail_from_address": "oc", "mail_smtpmode": "smtp", "mail_domain": "xxxxxx.nl", "mail_smtpauthtype": "LOGIN", "mail_smtphost": "mail.linuxpete.nl", "mail_smtpauth": 1, "mail_smtpport": "465", "mail_smtpname": "***REMOVED SENSITIVE VALUE***", "mail_smtppassword": "***REMOVED SENSITIVE VALUE***", "mail_smtpsecure": "ssl", "enabledPreviewProviders": [ "OC\\Preview\\Image", "OC\\Preview\\MP3", "OC\\Preview\\TXT", "OC\\Preview\\MarkDown", "OC\\Preview\\Epub", "OC\\Preview\\PDF", "OC\\Preview\\OpenDocument", "OC\\Preview\\StarOffice", "OC\\Preview\\MSOfficeDoc", "OC\\Preview\\MSOffice2003", "OC\\Preview\\MSOffice2007", "OC\\Preview\\FB2" ], "mysql.utf8mb4": true } ```

Are you using external storage, if yes which one: local/smb/sftp/... NO Are you using encryption: no

Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/... no

LDAP configuration (delete this part if not used)

``` With access to your command line run e.g.: sudo -u www-data php occ ldap:show-config from within your Nextcloud installation folder Without access to your command line download the data/owncloud.db to your local computer or access your SQL server remotely and run the select query: SELECT * FROM `oc_appconfig` WHERE `appid` = 'user_ldap'; Eventually replace sensitive data as the name/IP-address of your LDAP server or groups. ```

Client configuration

Browser: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0

Operating system:

Logs

Browser log

``` Insert your webserver log here ```

Nextcloud log

``` Insert your Nextcloud log here ```

Browser log

Insert your browser log here, this could for example include: a) The javascript console log b) The network log c) ...

ThePatzen commented 6 years ago

Same issue here :-(

lowlyocean commented 5 years ago

I am seeing this issue as well with Gallery v18.2.0 & Nextcloud 15.0.5 on Raspberry Pi 3. @oparoz @icewind1991

Chrome console output

``` merged.js?v=6684a091-0:55 Uncaught TypeError: Cannot read property 'message' of undefined at Object. (merged.js?v=6684a091-0:55) at Object. (core.js?v=6684a091-0:1) at j (core.js?v=6684a091-0:1) at Object.fireWith [as rejectWith] (core.js?v=6684a091-0:1) at x (core.js?v=6684a091-0:1) at XMLHttpRequest. (core.js?v=6684a091-0:1) (anonymous) @ merged.js?v=6684a091-0:55 (anonymous) @ core.js?v=6684a091-0:1 j @ core.js?v=6684a091-0:1 fireWith @ core.js?v=6684a091-0:1 x @ core.js?v=6684a091-0:1 (anonymous) @ core.js?v=6684a091-0:1 load (async) send @ core.js?v=6684a091-0:1 ajax @ core.js?v=6684a091-0:1 n. @ core.js?v=6684a091-0:1 getJSON @ core.js?v=6684a091-0:1 getFiles @ merged.js?v=6684a091-0:54 (anonymous) @ merged.js?v=6684a091-0:2 (anonymous) @ core.js?v=6684a091-0:1 j @ core.js?v=6684a091-0:1 fireWith @ core.js?v=6684a091-0:1 (anonymous) @ core.js?v=6684a091-0:1 j @ core.js?v=6684a091-0:1 fireWith @ core.js?v=6684a091-0:1 e. @ core.js?v=6684a091-0:1 (anonymous) @ merged.js?v=6684a091-0:490 j @ core.js?v=6684a091-0:1 fireWith @ core.js?v=6684a091-0:1 x @ core.js?v=6684a091-0:1 (anonymous) @ core.js?v=6684a091-0:1 load (async) send @ core.js?v=6684a091-0:1 ajax @ core.js?v=6684a091-0:1 n. @ core.js?v=6684a091-0:1 _getSlideshowTemplate @ merged.js?v=6684a091-0:489 init @ merged.js?v=6684a091-0:480 (anonymous) @ merged.js?v=6684a091-0:2 (anonymous) @ core.js?v=6684a091-0:1 j @ core.js?v=6684a091-0:1 fireWith @ core.js?v=6684a091-0:1 x @ core.js?v=6684a091-0:1 (anonymous) @ core.js?v=6684a091-0:1 load (async) send @ core.js?v=6684a091-0:1 ajax @ core.js?v=6684a091-0:1 n. @ core.js?v=6684a091-0:1 getJSON @ core.js?v=6684a091-0:1 (anonymous) @ merged.js?v=6684a091-0:2 j @ core.js?v=6684a091-0:1 fireWith @ core.js?v=6684a091-0:1 ready @ core.js?v=6684a091-0:1 I @ core.js?v=6684a091-0:1 ```

nextcloud / gallery