search

nextcloud / server

☁️ Nextcloud server, a safe home for all your data
https://nextcloud.com
GNU Affero General Public License v3.0
26.89k stars 4.02k forks source link

`OC\Files\View\getDirectoryContents` returns directories/files that do not exist on disk #26562

Closed kaysond closed 3 years ago

kaysond commented 3 years ago

How to use GitHub

Steps to reproduce

  1. Modify https://github.com/nextcloud/server/blob/f031dd61c14a093effeb5773323f578abd65d54f/core/Command/Encryption/MigrateKeyStorage.php#L184 and add the following lines below: 
    if ( $content === false) {
var_dump($folder); var_dump($path); var_dump($content);
}
  2. Run occ encryption:migrate-key-storage-format
  3. Observe the following output 
    string(88) "/aram/files_encryption/keys/files/Photos/637533018_20210315_133033.jpg/OC_DEFAULT_MODULE"
string(96) "/aram/files_encryption/keys/files/Photos/637533018_20210315_133033.jpg/OC_DEFAULT_MODULE/fileKey"
bool(false)
An unhandled exception has been thrown:
TypeError: Argument 1 passed to OC\Security\Crypto::decrypt() must be of the type string, bool given, called in /var/www/nubcloud/core/Command/Encryption/MigrateKeyStorage.php on line 188 and defined in /var/www/nubcloud/lib/private/Security/Crypto.php:115
Stack trace:
#0 /var/www/nubcloud/core/Command/Encryption/MigrateKeyStorage.php(188): OC\Security\Crypto->decrypt()
#1 /var/www/nubcloud/core/Command/Encryption/MigrateKeyStorage.php(166): OC\Core\Command\Encryption\MigrateKeyStorage->traverseFileKeys()
#2 /var/www/nubcloud/core/Command/Encryption/MigrateKeyStorage.php(166): OC\Core\Command\Encryption\MigrateKeyStorage->traverseFileKeys()
#3 /var/www/nubcloud/core/Command/Encryption/MigrateKeyStorage.php(166): OC\Core\Command\Encryption\MigrateKeyStorage->traverseFileKeys()
#4 /var/www/nubcloud/core/Command/Encryption/MigrateKeyStorage.php(166): OC\Core\Command\Encryption\MigrateKeyStorage->traverseFileKeys()
#5 /var/www/nubcloud/core/Command/Encryption/MigrateKeyStorage.php(259): OC\Core\Command\Encryption\MigrateKeyStorage->traverseFileKeys()
#6 /var/www/nubcloud/core/Command/Encryption/MigrateKeyStorage.php(235): OC\Core\Command\Encryption\MigrateKeyStorage->updateUserKeys()
#7 /var/www/nubcloud/core/Command/Encryption/MigrateKeyStorage.php(101): OC\Core\Command\Encryption\MigrateKeyStorage->updateUsersKeys()
#8 /var/www/nubcloud/core/Command/Encryption/MigrateKeyStorage.php(83): OC\Core\Command\Encryption\MigrateKeyStorage->updateKeys()
#9 /var/www/nubcloud/3rdparty/symfony/console/Command/Command.php(255): OC\Core\Command\Encryption\MigrateKeyStorage->execute()
#10 /var/www/nubcloud/3rdparty/symfony/console/Application.php(1009): Symfony\Component\Console\Command\Command->run()
#11 /var/www/nubcloud/3rdparty/symfony/console/Application.php(273): Symfony\Component\Console\Application->doRunCommand()
#12 /var/www/nubcloud/3rdparty/symfony/console/Application.php(149): Symfony\Component\Console\Application->doRun()
#13 /var/www/nubcloud/lib/private/Console/Application.php(215): Symfony\Component\Console\Application->run()
#14 /var/www/nubcloud/console.php(100): OC\Console\Application->run()
#15 /var/www/nubcloud/occ(11): require_once('/var/www/nubclo...')
  4. Observe that the directory in question (/aram/files_encryption/keys/files/Photos/637533018_20210315_133033.jpg/) doesn't exist: 
    administrator@fserver - /v/w/nubcloud > ls $NEXTCLOUD_DATA_ROOT/aram/files_encryption/keys/files/Photos/637533018_20210315_133033.jpg/
ls: cannot access '/var/www/files/Nubcloud/aram/files_encryption/keys/files/Photos/637533018_20210315_133033.jpg/': No such file or directory

Expected behaviour

OC\Files\View\getDirectoryContents https://github.com/nextcloud/server/blob/f031dd61c14a093effeb5773323f578abd65d54f/lib/private/Files/View.php#L1434 does not return directories/files that do not exist on disk

Actual behaviour

It returns directories/files that do not exist on disk

Server configuration

Operating system: Ubuntu 20.04.1 LTS

Web server: Apache 2.4.41-4ubuntu3.1

Database: mariadb Ver 15.1 Distrib 10.3.25-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2

PHP version: 7.4

Nextcloud version: (see Nextcloud admin page) 21.0.1

Updated from an older Nextcloud/ownCloud or fresh install: Updating since about Nextcloud version 17

Where did you install Nextcloud from: Nextcloud website

Signing status:

Signing status ``` No errors have been found. ```

List of activated apps:

App list ``` Enabled: - accessibility: 1.7.0 - activity: 2.14.3 - admin_audit: 1.11.0 - cloud_federation_api: 1.4.0 - comments: 1.11.0 - contacts: 3.5.1 - contactsinteraction: 1.2.0 - dav: 1.17.1 - encryption: 2.9.0 - federatedfilesharing: 1.11.0 - federation: 1.11.0 - files: 1.16.0 - files_external: 1.12.0 - files_pdfviewer: 2.1.0 - files_rightclick: 1.0.0 - files_sharing: 1.13.1 - files_trashbin: 1.11.0 - files_versions: 1.14.0 - files_videoplayer: 1.10.0 - firstrunwizard: 2.10.0 - hsts: 0.8.0 - impersonate: 1.8.0 - logreader: 2.6.0 - lookup_server_connector: 1.9.0 - nextcloud_announcements: 1.10.0 - notes: 4.0.4 - notifications: 2.9.0 - oauth2: 1.9.0 - password_policy: 1.11.0 - photos: 1.3.0 - privacy: 1.5.0 - provisioning_api: 1.11.0 - recommendations: 1.0.0 - serverinfo: 1.11.0 - settings: 1.3.0 - sharebymail: 1.11.0 - support: 1.4.0 - survey_client: 1.9.0 - systemtags: 1.11.0 - text: 3.2.0 - theming: 1.12.0 - twofactor_backupcodes: 1.10.0 - updatenotification: 1.11.0 - user_status: 1.1.1 - viewer: 1.5.0 - weather_status: 1.1.0 - workflowengine: 2.3.0 Disabled: - dashboard - user_ldap ```

Nextcloud configuration:

Config report ``` { "system": { "instanceid": "***REMOVED SENSITIVE VALUE***", "passwordsalt": "***REMOVED SENSITIVE VALUE***", "secret": "***REMOVED SENSITIVE VALUE***", "trusted_domains": [ "REDACTED" ], "datadirectory": "***REMOVED SENSITIVE VALUE***", "overwrite.cli.url": "REDACTED", "htaccess.RewriteBase": "\/", "dbtype": "mysql", "version": "21.0.1.1", "dbname": "***REMOVED SENSITIVE VALUE***", "dbhost": "***REMOVED SENSITIVE VALUE***", "dbport": "", "dbtableprefix": "oc_", "dbuser": "***REMOVED SENSITIVE VALUE***", "dbpassword": "***REMOVED SENSITIVE VALUE***", "installed": true, "mail_from_address": "***REMOVED SENSITIVE VALUE***", "mail_smtpmode": "sendmail", "mail_smtpauthtype": "LOGIN", "mail_domain": "***REMOVED SENSITIVE VALUE***", "maintenance": false, "theme": "", "loglevel": 2, "memcache.local": "\\OC\\Memcache\\Redis", "memcache.distributed": "\\OC\\Memcache\\Redis", "memcache.locking": "\\OC\\Memcache\\Redis", "redis": { "host": "***REMOVED SENSITIVE VALUE***", "port": 0, "timeout": 1.5 }, "mail_sendmailmode": "smtp", "mysql.utf8mb4": true, "updater.release.channel": "stable", "encryption.legacy_format_support": true, "encryption.key_storage_migrated": false, "updater.secret": "***REMOVED SENSITIVE VALUE***", "default_phone_region": "US" } } ```

Are you using external storage, if yes which one: data directory is nfs mounted

Are you using encryption: yes

Are you using an external user-backend, if yes which one: no

Client configuration

Not using web client

Logs

Web server error log

Web server error log ``` Not using web server ```

Nextcloud log (data/nextcloud.log)

Nextcloud log ``` {"reqId":"6YROnokQnyfvYJLZ5mIk","level":3,"time":"2021-04-14T18:16:18+00:00","remoteAddr":"","user":"--","app":"PHP","method":"","url":"--","message":{"Exception":"Error","Message":"file_get_contents(/var/www/files/Nubcloud/aram/files_encryption/keys/files/Photos/637533018_20210315_133033.jpg/OC_DEFAULT_MODULE/fileKey): failed to open stream: No such file or directory at /var/www/nubcloud/lib/private/Files/Storage/Local.php#274","Code":0,"Trace":[{"function":"onError","class":"OC\\Log\\ErrorHandler","type":"::"},{"file":"/var/www/nubcloud/lib/private/Files/Storage/Local.php","line":274,"function":"file_get_contents"},{"file":"/var/www/nubcloud/lib/private/Files/Storage/Wrapper/Encryption.php","line":228,"function":"file_get_contents","class":"OC\\Files\\Storage\\Local","type":"->"},{"file":"/var/www/nubcloud/lib/private/Files/Storage/Wrapper/Wrapper.php","line":248,"function":"file_get_contents","class":"OC\\Files\\Storage\\Wrapper\\Encryption","type":"->"},{"file":"/var/www/nubcloud/lib/private/Files/View.php","line":1168,"function":"file_get_contents","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->"},{"file":"/var/www/nubcloud/lib/private/Files/View.php","line":598,"function":"basicOperation","class":"OC\\Files\\View","type":"->"},{"file":"/var/www/nubcloud/core/Command/Encryption/MigrateKeyStorage.php","line":183,"function":"file_get_contents","class":"OC\\Files\\View","type":"->"},{"file":"/var/www/nubcloud/core/Command/Encryption/MigrateKeyStorage.php","line":166,"function":"traverseFileKeys","class":"OC\\Core\\Command\\Encryption\\MigrateKeyStorage","type":"->"},{"file":"/var/www/nubcloud/core/Command/Encryption/MigrateKeyStorage.php","line":166,"function":"traverseFileKeys","class":"OC\\Core\\Command\\Encryption\\MigrateKeyStorage","type":"->"},{"file":"/var/www/nubcloud/core/Command/Encryption/MigrateKeyStorage.php","line":166,"function":"traverseFileKeys","class":"OC\\Core\\Command\\Encryption\\MigrateKeyStorage","type":"->"},{"file":"/var/www/nubcloud/core/Command/Encryption/MigrateKeyStorage.php","line":166,"function":"traverseFileKeys","class":"OC\\Core\\Command\\Encryption\\MigrateKeyStorage","type":"->"},{"file":"/var/www/nubcloud/core/Command/Encryption/MigrateKeyStorage.php","line":259,"function":"traverseFileKeys","class":"OC\\Core\\Command\\Encryption\\MigrateKeyStorage","type":"->"},{"file":"/var/www/nubcloud/core/Command/Encryption/MigrateKeyStorage.php","line":235,"function":"updateUserKeys","class":"OC\\Core\\Command\\Encryption\\MigrateKeyStorage","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nubcloud/core/Command/Encryption/MigrateKeyStorage.php","line":101,"function":"updateUsersKeys","class":"OC\\Core\\Command\\Encryption\\MigrateKeyStorage","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nubcloud/core/Command/Encryption/MigrateKeyStorage.php","line":83,"function":"updateKeys","class":"OC\\Core\\Command\\Encryption\\MigrateKeyStorage","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nubcloud/3rdparty/symfony/console/Command/Command.php","line":255,"function":"execute","class":"OC\\Core\\Command\\Encryption\\MigrateKeyStorage","type":"->"},{"file":"/var/www/nubcloud/3rdparty/symfony/console/Application.php","line":1009,"function":"run","class":"Symfony\\Component\\Console\\Command\\Command","type":"->"},{"file":"/var/www/nubcloud/3rdparty/symfony/console/Application.php","line":273,"function":"doRunCommand","class":"Symfony\\Component\\Console\\Application","type":"->"},{"file":"/var/www/nubcloud/3rdparty/symfony/console/Application.php","line":149,"function":"doRun","class":"Symfony\\Component\\Console\\Application","type":"->"},{"file":"/var/www/nubcloud/lib/private/Console/Application.php","line":215,"function":"run","class":"Symfony\\Component\\Console\\Application","type":"->"},{"file":"/var/www/nubcloud/console.php","line":100,"function":"run","class":"OC\\Console\\Application","type":"->"},{"file":"/var/www/nubcloud/occ","line":11,"args":["/var/www/nubcloud/console.php"],"function":"require_once"}],"File":"/var/www/nubcloud/lib/private/Log/ErrorHandler.php","Line":92,"CustomMessage":"--"},"userAgent":"--","version":"21.0.1.1"} ```
kaysond commented 3 years ago

This issue causes the legacy encryption migration to fail (https://github.com/nextcloud/server/issues/24240#issuecomment-819264419). It seems like the View class might be pulling the file list from the database, as turning off mariadb causes the encryption migration to fail immediately, though it happens in OC\Init so maybe its too early. I haven't had a chance yet to dig through the rest of the code or the db to see what exactly is going on... It's pretty dense.

I also tried restarting redis to make sure its not a caching issue - no luck there.

eNTi commented 3 years ago

Yes please let us know how to resync the db with the actual files and get rid of the

does not have a proper header

issue.

kaysond commented 3 years ago

So the issue here is indeed that there were database entries for files that have been deleted. Not sure how it got into that state, but apparently its a common issue. In my case running occ files:scan --all seems to have fixed the issue.