search

nextcloud / server

☁️ Nextcloud server, a safe home for all your data
https://nextcloud.com
GNU Affero General Public License v3.0
26.63k stars 3.99k forks source link

external storage: local: do not stop scanning on permission failure #29514

Open alanswanson opened 2 years ago

alanswanson commented 2 years ago

How to use GitHub

Steps to reproduce

  1. Add a local external storage which includes directories with no access permissions for web server user.
  2. NC file scan fails at first directory permission denied.

Expected behaviour

Skip directory and carry on.

Actual behaviour

Stops with exception.

Comments

I had noticed that NC stopped updating any file additions/removals to a local external storage shortly after updating from 21.0.4 to 21.0.5 but had also changed the local external storage from an ext4 filesystem mountpoint to an overlayfs lowerdirs only mountpoint (a merged JBOD including the original ext4 filesystem).

Running occ files:scan shows it flaking at a permission denied with the "lost+found" directory. Since "lost+found" directories permission have always been root 700 not sure why it started failing when it was scanning fine before. The overlayfs doesn't cause any user process visible difference and I can't see any obvious related changes in 21.0.5. Bit odd.

As a workaround I've added "lost+found" to the isIgnoredDir list in lib/private/Files/Filesystem.php file but since directory permissions issues on local storage are quite likely: for any local external storage all directory permission failures should be a warning and carry on not an exception and stop.

Reopen of #28277 since it got closed by stale bot without original reporter updating.

Server configuration

Operating system: Linux Web server: Lighttpd Database: MariaDB PHP version: 7.4.25 Nextcloud version: (see Nextcloud admin page) 21.0.5

Where did you install Nextcloud from: Nextcloud web tar download.

Are you using external storage, if yes which one: local/smb/sftp/... local

Logs

OCC server error log

$ php occ -vvv files:scan --all
Starting scan for user 1 out of 5 (admin)
        Folder  /admin/
        Folder  /admin/cache
        Folder  /admin/files
        Folder  /admin/files/Media/
        Folder  /admin/files/Media/lost+found/
Exception during scan: opendir(/usr/local/media/lost+found): failed to open dir: Permission denied
#0 [internal function]: OCA\Files\Command\Scan->exceptionErrorHandler()
#1 /var/www/localhost/htdocs/cloud/lib/private/Files/Storage/Local.php(135): opendir()
#2 /var/www/localhost/htdocs/cloud/lib/private/Files/Storage/Common.php(878): OC\Files\Storage\Local->opendir()
#3 /var/www/localhost/htdocs/cloud/lib/private/Files/Storage/Wrapper/PermissionsMask.php(158): OC\Files\Storage\Common->getDirectoryContent()
#4 /var/www/localhost/htdocs/cloud/lib/private/Files/Storage/Wrapper/PermissionsMask.php(158): OC\Files\Storage\Wrapper\PermissionsMask->getDirectoryContent()
#5 [internal function]: OC\Files\Storage\Wrapper\PermissionsMask->getDirectoryContent()
#6 /var/www/localhost/htdocs/cloud/lib/private/Files/Cache/Scanner.php(409): iterator_to_array()
#7 /var/www/localhost/htdocs/cloud/lib/private/Files/Cache/Scanner.php(389): OC\Files\Cache\Scanner->handleChildren()
#8 /var/www/localhost/htdocs/cloud/lib/private/Files/Cache/Scanner.php(392): OC\Files\Cache\Scanner->scanChildren()
#9 /var/www/localhost/htdocs/cloud/lib/private/Files/Cache/Scanner.php(341): OC\Files\Cache\Scanner->scanChildren()
#10 /var/www/localhost/htdocs/cloud/lib/private/Files/Utils/Scanner.php(260): OC\Files\Cache\Scanner->scan()
#11 /var/www/localhost/htdocs/cloud/apps/files/lib/Command/Scan.php(158): OC\Files\Utils\Scanner->scan()
#12 /var/www/localhost/htdocs/cloud/apps/files/lib/Command/Scan.php(214): OCA\Files\Command\Scan->scanFiles()
#13 /var/www/localhost/htdocs/cloud/3rdparty/symfony/console/Command/Command.php(255): OCA\Files\Command\Scan->execute()
#14 /var/www/localhost/htdocs/cloud/core/Command/Base.php(169): Symfony\Component\Console\Command\Command->run()
#15 /var/www/localhost/htdocs/cloud/3rdparty/symfony/console/Application.php(1009): OC\Core\Command\Base->run()
#16 /var/www/localhost/htdocs/cloud/3rdparty/symfony/console/Application.php(273): Symfony\Component\Console\Application->doRunCommand()
#17 /var/www/localhost/htdocs/cloud/3rdparty/symfony/console/Application.php(149): Symfony\Component\Console\Application->doRun()
#18 /var/www/localhost/htdocs/cloud/lib/private/Console/Application.php(215): Symfony\Component\Console\Application->run()
#19 /var/www/localhost/htdocs/cloud/console.php(100): OC\Console\Application->run()
#20 /var/www/localhost/htdocs/cloud/occ(11): require_once('/var/www/localh...')
#21 {main}

And repeat for each user sharing the local external storage from /usr/local/media made available under Media in this example.

dschrempf commented 2 years ago

I also found:

I don't think this is or has been fixed. I keep having this problem for years now.

szaimen commented 1 year ago

Hi, please update to 24.0.9 or better 25.0.3 and report back if it fixes the issue. Thank you!

My goal is to add a label like e.g. 25-feedback to this ticket of an up-to-date major Nextcloud version where the bug could be reproduced. However this is not going to work without your help. So thanks for all your effort!

If you don't manage to reproduce the issue in time and the issue gets closed but you can reproduce the issue afterwards, feel free to create a new bug report with up-to-date information by following this link: https://github.com/nextcloud/server/issues/new?assignees=&labels=bug%2C0.+Needs+triage&template=BUG_REPORT.yml&title=%5BBug%5D%3A+

dschrempf commented 1 year ago

Same error on Nextcloud 25.0.3:

Starting scan for user 1 out of 3 (...)
Exception during scan: opendir(/path/to/file): Failed to open directory: Permission denied
#0 [internal function]: OCA\Files\Command\Scan->exceptionErrorHandler(2, 'opendir(/mnt/ho...', '/nix/store/sj7j...', 152)
#1 /nix/store/sj7jjky7jpczf7q1w1sk8ayg1rkkwclp-nextcloud-25.0.3/lib/private/Files/Storage/Local.php(152): opendir('/mnt/home/.cach...')
#2 /nix/store/sj7jjky7jpczf7q1w1sk8ayg1rkkwclp-nextcloud-25.0.3/lib/private/Files/Storage/Common.php(890): OC\Files\Storage\Local->opendir('.cache/mesa_sha...')
#3 [internal function]: OC\Files\Storage\Common->getDirectoryContent('.cache/mesa_sha...')
#4 /nix/store/sj7jjky7jpczf7q1w1sk8ayg1rkkwclp-nextcloud-25.0.3/lib/private/Files/Cache/Scanner.php(412): iterator_to_array(Object(Generator))
#5 /nix/store/sj7jjky7jpczf7q1w1sk8ayg1rkkwclp-nextcloud-25.0.3/lib/private/Files/Cache/Scanner.php(391): OC\Files\Cache\Scanner->handleChildren('.cache/mesa_sha...', true, 3, 9965255, true, 0)
#6 /nix/store/sj7jjky7jpczf7q1w1sk8ayg1rkkwclp-nextcloud-25.0.3/lib/private/Files/Cache/Scanner.php(394): OC\Files\Cache\Scanner->scanChildren('.cache/mesa_sha...', true, 3, 9965255, true)
#7 /nix/store/sj7jjky7jpczf7q1w1sk8ayg1rkkwclp-nextcloud-25.0.3/lib/private/Files/Cache/Scanner.php(394): OC\Files\Cache\Scanner->scanChildren('.cache/mesa_sha...', true, 3, 8799792, true)
#8 /nix/store/sj7jjky7jpczf7q1w1sk8ayg1rkkwclp-nextcloud-25.0.3/lib/private/Files/Cache/Scanner.php(394): OC\Files\Cache\Scanner->scanChildren('.cache', true, 3, 8616359, true)
#9 /nix/store/sj7jjky7jpczf7q1w1sk8ayg1rkkwclp-nextcloud-25.0.3/lib/private/Files/Cache/Scanner.php(342): OC\Files\Cache\Scanner->scanChildren('', true, 3, 8616358, true, Array)
#10 /nix/store/sj7jjky7jpczf7q1w1sk8ayg1rkkwclp-nextcloud-25.0.3/lib/private/Files/Utils/Scanner.php(256): OC\Files\Cache\Scanner->scan('', true, 3)
#11 /nix/store/sj7jjky7jpczf7q1w1sk8ayg1rkkwclp-nextcloud-25.0.3/apps/files/lib/Command/Scan.php(161): OC\Files\Utils\Scanner->scan('/dominik', true, NULL)
#12 /nix/store/sj7jjky7jpczf7q1w1sk8ayg1rkkwclp-nextcloud-25.0.3/apps/files/lib/Command/Scan.php(217): OCA\Files\Command\Scan->scanFiles('dominik', '/dominik', false, Object(Symfony\Component\Console\Output\ConsoleOutput), false, true, false)
#13 /nix/store/sj7jjky7jpczf7q1w1sk8ayg1rkkwclp-nextcloud-25.0.3/3rdparty/symfony/console/Command/Command.php(255): OCA\Files\Command\Scan->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#14 /nix/store/sj7jjky7jpczf7q1w1sk8ayg1rkkwclp-nextcloud-25.0.3/core/Command/Base.php(177): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#15 /nix/store/sj7jjky7jpczf7q1w1sk8ayg1rkkwclp-nextcloud-25.0.3/3rdparty/symfony/console/Application.php(1009): OC\Core\Command\Base->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#16 /nix/store/sj7jjky7jpczf7q1w1sk8ayg1rkkwclp-nextcloud-25.0.3/3rdparty/symfony/console/Application.php(273): Symfony\Component\Console\Application->doRunCommand(Object(OCA\Files\Command\Scan), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#17 /nix/store/sj7jjky7jpczf7q1w1sk8ayg1rkkwclp-nextcloud-25.0.3/3rdparty/symfony/console/Application.php(149): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#18 /nix/store/sj7jjky7jpczf7q1w1sk8ayg1rkkwclp-nextcloud-25.0.3/lib/private/Console/Application.php(213): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#19 /nix/store/sj7jjky7jpczf7q1w1sk8ayg1rkkwclp-nextcloud-25.0.3/console.php(100): OC\Console\Application->run()
#20 /nix/store/sj7jjky7jpczf7q1w1sk8ayg1rkkwclp-nextcloud-25.0.3/occ(11): require_once('/nix/store/sj7j...')
#21 {main}
Starting scan for user 2 out of 3 (...)
lckarssen commented 1 year ago

Same on Nextcloud 25.0.4:

Starting scan for user 13 out of 14 (REDACTED)
Exception during scan: opendir(/PATH/TO/FILE/ON/EXTERNALSTORAGE): Failed to open directory: Permission denied
#0 [internal function]: OCA\Files\Command\Scan->exceptionErrorHandler()
#1 /var/www/nextcloud/lib/private/Files/Storage/Local.php(153): opendir()
#2 /var/www/nextcloud/lib/private/Files/Storage/Common.php(890): OC\Files\Storage\Local->opendir()
#3 /var/www/nextcloud/lib/private/Files/Storage/Wrapper/PermissionsMask.php(157): OC\Files\Storage\Common->getDirectoryContent()
#4 [internal function]: OC\Files\Storage\Wrapper\PermissionsMask->getDirectoryContent()
#5 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(412): iterator_to_array()
#6 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(391): OC\Files\Cache\Scanner->handleChildren()
#7 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(394): OC\Files\Cache\Scanner->scanChildren()
#8 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(394): OC\Files\Cache\Scanner->scanChildren()
#9 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(394): OC\Files\Cache\Scanner->scanChildren()
#10 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(394): OC\Files\Cache\Scanner->scanChildren()
#11 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(342): OC\Files\Cache\Scanner->scanChildren()
#12 /var/www/nextcloud/lib/private/Files/Utils/Scanner.php(256): OC\Files\Cache\Scanner->scan()
#13 /var/www/nextcloud/apps/files/lib/Command/Scan.php(161): OC\Files\Utils\Scanner->scan()
#14 /var/www/nextcloud/apps/files/lib/Command/Scan.php(217): OCA\Files\Command\Scan->scanFiles()
#15 /var/www/nextcloud/apps/recognize/vendor/symfony/console/Command/Command.php(298): OCA\Files\Command\Scan->execute()
#16 /var/www/nextcloud/core/Command/Base.php(177): Symfony\Component\Console\Command\Command->run()
#17 /var/www/nextcloud/apps/recognize/vendor/symfony/console/Application.php(1040): OC\Core\Command\Base->run()
#18 /var/www/nextcloud/apps/recognize/vendor/symfony/console/Application.php(301): Symfony\Component\Console\Application->doRunCommand()
#19 /var/www/nextcloud/apps/recognize/vendor/symfony/console/Application.php(171): Symfony\Component\Console\Application->doRun()
#20 /var/www/nextcloud/lib/private/Console/Application.php(213): Symfony\Component\Console\Application->run()
#21 /var/www/nextcloud/console.php(100): OC\Console\Application->run()
#22 /var/www/nextcloud/occ(11): require_once('...')
#23 {main}
megapearl commented 6 months ago

I'm having the same problem with a WebDAV synology mount which has directories like "#recycle". Is there any way to ignore files or directires with "@" or "#" or "." in front of it?

root@cloud:/# occ files:scan -vvv -- donald
Starting scan for user 1 out of 1 (donald)
        Folder  /donald/files/Synology_mountpoint/movies/#recycle
Exception during scan: Sabre\HTTP\ClientHttpException:Forbidden
#0 /app/www/public/lib/private/Files/Storage/DAV.php(931): OC\Files\Storage\DAV->convertException()
#1 /app/www/public/lib/private/Files/Storage/Wrapper/PermissionsMask.php(157): OC\Files\Storage\DAV->getDirectoryContent()
#2 [internal function]: OC\Files\Storage\Wrapper\PermissionsMask->getDirectoryContent()
#3 /app/www/public/lib/private/Files/Cache/Scanner.php(427): iterator_to_array()
#4 /app/www/public/lib/private/Files/Cache/Scanner.php(397): OC\Files\Cache\Scanner->handleChildren()
#5 /app/www/public/lib/private/Files/Cache/Scanner.php(400): OC\Files\Cache\Scanner->scanChildren()
#6 /app/www/public/lib/private/Files/Cache/Scanner.php(400): OC\Files\Cache\Scanner->scanChildren()
#7 /app/www/public/lib/private/Files/Cache/Scanner.php(347): OC\Files\Cache\Scanner->scanChildren()
#8 /app/www/public/lib/private/Files/Utils/Scanner.php(273): OC\Files\Cache\Scanner->scan()
#9 /config/www/nextcloud/apps/files/lib/Command/Scan.php(180): OC\Files\Utils\Scanner->scan()
#10 /config/www/nextcloud/apps/files/lib/Command/Scan.php(241): OCA\Files\Command\Scan->scanFiles()
#11 /app/www/public/3rdparty/symfony/console/Command/Command.php(298): OCA\Files\Command\Scan->execute()
#12 /app/www/public/core/Command/Base.php(177): Symfony\Component\Console\Command\Command->run()
#13 /app/www/public/3rdparty/symfony/console/Application.php(1040): OC\Core\Command\Base->run()
#14 /app/www/public/3rdparty/symfony/console/Application.php(301): Symfony\Component\Console\Application->doRunCommand()
#15 /app/www/public/3rdparty/symfony/console/Application.php(171): Symfony\Component\Console\Application->doRun()
#16 /app/www/public/lib/private/Console/Application.php(206): Symfony\Component\Console\Application->run()
#17 /app/www/public/console.php(100): OC\Console\Application->run()
#18 /app/www/public/occ(11): require_once('...')
#19 {main}

+---------+-------+-----+---------+---------+--------+--------------+
| Folders | Files | New | Updated | Removed | Errors | Elapsed time |
+---------+-------+-----+---------+---------+--------+--------------+
| 145     | 288   | 0   | 0       | 0       | 1      | 00:00:04     |
+---------+-------+-----+---------+---------+--------+--------------+
root@cloud:/#
HansAndreManfredson commented 5 months ago

Same problem over here... I was happy to find this problem because I thought it would be a problem since I use LXD/Incus... I also mounted an NFS share on the host and mounted it to the system container ..... What is weird is that I can read and write to this folder from OS... 

Error during scan: opendir(/mnt/h2c/BackupiPhone/): Failed to open directory: Permission denied
#0 /snap/nextcloud/40887/htdocs/apps/files/lib/Command/Scan.php(246): OCA\Files\Command\Scan->exceptionErrorHandler()
#1 [internal function]: OCA\Files\Command\Scan->OCA\Files\Command\{closure}()
#2 /snap/nextcloud/40887/htdocs/lib/private/Files/Storage/Local.php(157): opendir()
#3 /snap/nextcloud/40887/htdocs/lib/private/Files/Storage/Common.php(896): OC\Files\Storage\Local->opendir()
#4 /snap/nextcloud/40887/htdocs/lib/private/Files/Storage/Wrapper/PermissionsMask.php(157): OC\Files\Storage\Common->getDirectoryContent()
#5 [internal function]: OC\Files\Storage\Wrapper\PermissionsMask->getDirectoryContent()
#6 /snap/nextcloud/40887/htdocs/lib/private/Files/Cache/Scanner.php(447): iterator_to_array()
#7 /snap/nextcloud/40887/htdocs/lib/private/Files/Cache/Scanner.php(404): OC\Files\Cache\Scanner->handleChildren()
#8 /snap/nextcloud/40887/htdocs/lib/private/Files/Cache/Scanner.php(354): OC\Files\Cache\Scanner->scanChildren()
#9 /snap/nextcloud/40887/htdocs/lib/private/Files/Utils/Scanner.php(273): OC\Files\Cache\Scanner->scan()
#10 /snap/nextcloud/40887/htdocs/apps/files/lib/Command/Scan.php(164): OC\Files\Utils\Scanner->scan()
#11 /snap/nextcloud/40887/htdocs/apps/files/lib/Command/Scan.php(219): OCA\Files\Command\Scan->scanFiles()
#12 /snap/nextcloud/40887/htdocs/3rdparty/symfony/console/Command/Command.php(298): OCA\Files\Command\Scan->execute()
#13 /snap/nextcloud/40887/htdocs/core/Command/Base.php(177): Symfony\Component\Console\Command\Command->run()
#14 /snap/nextcloud/40887/htdocs/3rdparty/symfony/console/Application.php(1040): OC\Core\Command\Base->run()
#15 /snap/nextcloud/40887/htdocs/3rdparty/symfony/console/Application.php(301): Symfony\Component\Console\Application->doRunCommand()
#16 /snap/nextcloud/40887/htdocs/3rdparty/symfony/console/Application.php(171): Symfony\Component\Console\Application->doRun()
#17 /snap/nextcloud/40887/htdocs/lib/private/Console/Application.php(218): Symfony\Component\Console\Application->run()
#18 /snap/nextcloud/40887/htdocs/console.php(100): OC\Console\Application->run()
#19 /snap/nextcloud/40887/htdocs/occ(11): require_once('...')
#20 {main}
root@lxd-nxc-01:~# nextcloud.occ status
  - installed: true
  - version: 27.1.7.2
  - versionstring: 27.1.7
  - edition:
  - maintenance: false
  - needsDbUpgrade: false
  - productname: Nextcloud
  - extendedSupport: false

  ---
root@lxd-nxc-01:~# cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.6 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.6 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
FlattusBlastus commented 5 months ago

Still broken in 28.0.4

cbean commented 1 month ago

Still broken in 29.04

The log has logged more than 130 entries a day cause of that.

{"reqId":"jXS1syzK8wRkNvlWJKll","level":3,"time":"2024-07-20T10:00:01+00:00","remoteAddr":"","user":"--","app":"PHP","method":"","url":"--","message":"opendir(/Public/lost+found): Failed to open directory: Permission denied at /Private/my.example.com/nextcloud/lib/private/Files/Storage/Local.php#164","userAgent":"--","version":"29.0.4.1","data":{"app":"PHP"},"id":"669b8bc35074e"}