owncloud / core

:cloud: ownCloud web server core (Files, DAV, etc.)
https://owncloud.com
GNU Affero General Public License v3.0
8.38k stars 2.05k forks source link

Error when running file scan on Owncloud instance with objectstore primary storage backend #30425

Open rontrompert opened 6 years ago

rontrompert commented 6 years ago
### Steps to reproduce 1. Run a file scan on a Owncloud instance with an S3 object store storage backend as primary storage. ### Expected behaviour file scan runs normally ### Actual behaviour Get an error message: root# sudo -u apache php occ files:scan ron Starting scan for user 1 out of 1 (ron) Exception during scan: Undefined index: size #0 /var/www/owncloud10/lib/private/Files/Cache/Scanner.php(421): OCA\Files\Command\Scan->exceptionErrorHandler(8, 'Undefined index...', '/var/www/ownclo...', 421, Array) #1 /var/www/owncloud10/lib/private/Files/Cache/Scanner.php(382): OC\Files\Cache\Scanner->handleChildren('', true, 3, '12977', true, 16102413) #2 /var/www/owncloud10/lib/private/Files/Cache/Scanner.php(316): OC\Files\Cache\Scanner->scanChildren('', true, 3, '12977', true) #3 /var/www/owncloud10/lib/private/Files/Utils/Scanner.php(235): OC\Files\Cache\Scanner->scan('', true, 3) #4 /var/www/owncloud10/apps/files/lib/Command/Scan.php(159): OC\Files\Utils\Scanner->scan('/ron') #5 /var/www/owncloud10/apps/files/lib/Command/Scan.php(228): OCA\Files\Command\Scan->scanFiles('ron', '/ron', false, Object(Symfony\Component\Console\Output\ConsoleOutput), false) #6 /var/www/owncloud10/lib/composer/symfony/console/Command/Command.php(262): OCA\Files\Command\Scan->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) #7 /var/www/owncloud10/core/Command/Base.php(159): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) #8 /var/www/owncloud10/lib/composer/symfony/console/Application.php(826): OC\Core\Command\Base->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) #9 /var/www/owncloud10/lib/composer/symfony/console/Application.php(189): Symfony\Component\Console\Application->doRunCommand(Object(OCA\Files\Command\Scan), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) #10 /var/www/owncloud10/lib/composer/symfony/console/Application.php(120): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) #11 /var/www/owncloud10/lib/private/Console/Application.php(160): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) #12 /var/www/owncloud10/console.php(106): OC\Console\Application->run() #13 /var/www/owncloud10/occ(11): require_once('/var/www/ownclo...') #14 {main} ### Server configuration **Operating system**: CentOS7.3 **Web server:** apache2.4.6 **Database:** mariadb-5.5 **PHP version:** 7.0 **ownCloud version:** (see ownCloud admin page) 10.0.3 enterprise **Updated from an older ownCloud or fresh install:** Fresh install **Where did you install ownCloud from:** tar ball **Signing status (ownCloud 9.0 and above):** ``` Login as admin user into your ownCloud and access http://example.com/index.php/settings/integrity/failed paste the results into https://gist.github.com/ and puth the link here. ``` No errors have been found. **The content of config/config.php:** ``` Log in to the web-UI with an administrator account and click on 'admin' -> 'Generate Config Report' -> 'Download ownCloud config report' This report includes the config.php settings, the list of activated apps and other details in a well sanitized form. or If you have access to your command line run e.g.: sudo -u www-data php occ config:list system from within your ownCloud installation folder *ATTENTION:* Do not post your config.php file in public as is. Please use one of the above methods whenever possible. Both, the generated reports from the web-ui and from occ config:list consistently remove sensitive data. You still may want to review the report before sending. If done manually then it is critical for your own privacy to dilligently remove *all* host names, passwords, usernames, salts and other credentials before posting. You should assume that attackers find such information and will use them against your systems. ``` { "system": { "instanceid": "oco1leooohkc", "passwordsalt": "REMOVED SENSITIVE VALUE", "secret": "REMOVED SENSITIVE VALUE", "trusted_domains": [ "test.data.surfsara.nl" ], "license-key": "REMOVED SENSITIVE VALUE", "overwrite.cli.url": "https://test.data.surfsara.nl", "overwritehost": "test.data.surfsara.nl", "overwriteprotocol": "https", "forwarded_for_headers": [ "HTTP_X_FORWARDED_FOR", "HTTP_FORWARDED_FOR" ], "memcache.local": "\OC\Memcache\APCu", "memcache.distributed": "\OC\Memcache\Redis", "memcache.locking": "\OC\Memcache\Redis", "redis": { "host": "145.100.5.230", "port": 6379, "timeout": 0, "password": "REMOVED SENSITIVE VALUE", "dbindex": 0 }, "filelocking.enabled": "true", "trusted_proxies": [ "145.100.5.230" ], "logfile": "/var/www/owncloud10/data/owncloud.log", "dbtype": "mysql", "version": "10.0.3.3", "dbname": "owncloud10", "dbhost": "145.100.5.230", "dbtableprefix": "oc_", "dbuser": "REMOVED SENSITIVE VALUE", "dbpassword": "REMOVED SENSITIVE VALUE", "logtimezone": "UTC", "installed": true, "objectstore_multibucket": { "class": "OCA\ObjectStore\S3", "arguments": { "autocreate": true, "options": { "version": "2006-03-01", "region": "", "credentials": { "key": "REMOVED SENSITIVE VALUE", "secret": "REMOVED SENSITIVE VALUE" }, "endpoint": "https://proxy.swift.surfsara.nl:443", "command.params": { "PathStyle": true } } } } } } **List of activated apps:** ``` If you have access to your command line run e.g.: sudo -u www-data php occ app:list from within your ownCloud installation folder. ``` admin_audit: 0.8.1 comments: 0.3.0 configreport: 0.1.1 dav: 0.3.0 enterprise_key: 0.1.3 federatedfilesharing: 0.3.1 federation: 0.1.0 files: 1.5.1 files_external: 0.7.1 files_sharing: 0.10.1 files_trashbin: 0.9.1 files_versions: 1.3.0 files_videoplayer: 0.9.8 firewall: 2.4.2 firstrunwizard: 1.1 market: 0.2.2 notifications: 0.3.1 objectstore: 0.4 provisioning_api: 0.5.0 systemtags: 0.3.0 systemtags_management: 0.2.0 templateeditor: 0.1 theme-enterprise: 2.0.0 updatenotification: 0.2.1 windows_network_drive: 0.6.1 workflow: 0.2.5 Disabled: encryption external files_antivirus files_ldap_home guests password_policy sharepoint theme-example user_external user_shibboleth **Are you using external storage, if yes which one:** local/smb/sftp/... No **Are you using encryption:** yes/no No **Are you using an external user-backend, if yes which one:** LDAP/ActiveDirectory/Webdav/... No #### LDAP configuration (delete this part if not used) ``` With access to your command line run e.g.: sudo -u www-data php occ ldap:show-config from within your ownCloud installation folder Without access to your command line download the data/owncloud.db to your local computer or access your SQL server remotely and run the select query: SELECT * FROM `oc_appconfig` WHERE `appid` = 'user_ldap'; Eventually replace sensitive data as the name/IP-address of your LDAP server or groups. ``` ### Client configuration **Browser:** Firefox **Operating system:** ### Logs #### Web server error log ``` Insert your webserver log here ``` #### ownCloud log (data/owncloud.log) ``` Insert your ownCloud log here ``` #### Browser log ``` Insert your browser log here, this could for example include: a) The javascript console log b) The network log c) ... ```
PVince81 commented 6 years ago

Files scan doesn't make sense when using primary object store: the object store only contains a flat list of file ids as keys and file contents as objects, so there is nothing useful to scan.

Please note that you cannot use primary object store mode with ownCloud if you uploaded files there using the usual S3/Swift uploaders that would use absolute paths as keys.

cc @DeepDiver1975

rontrompert commented 6 years ago

Using a file system we have seen in practice that situations can occur that there at some point an inconsistencies can develop between database and storage system for whatever reason. This is understandable because writing stuff to a database and write the stuff to storage is not an atomic action and at each stage things can fail.

On a normal file system the file scan can help here to bring things into sync. But there is no equivalent for an object store.

SURFsara-e-infra commented 6 years ago

Another thing is that in a disaster recovery scenario where there are separate backups of database and storage, the database is by definition out of sync with the storage.

PVince81 commented 6 years ago

I'll reopen this for discussion but I don't think it is possible to rescan a primary object store as the scanner cannot find out what to put into oc_filecache, beacause all objects inside the object store are mapped by file id and file contents. There isn't any other information there. So by scanning, the scanner cannot know what to do with fileid + contents: no way to find in what path this object is supposed to be.

Now if we do consider recovery scenarios where the backups aren't from the same moment, it would require storing more metadata into the object store like file path, etc.

@DeepDiver1975 @butonic

TrurlMcByte commented 6 years ago

I see additional problem in it, during scan external S3-like storage error happens on first subdirectory, but verbose output show this folder as "File"

UPD: S3 don't require to create "folders" (also s3fs not compatible with "folders" at all, it's work only with files-with-full-path-inside). When "folders" (as empty objects) is created files:scan have no errors in this case