occ files:scan generates an exception for .htaccess file

bluesky-ca commented 2 years ago

⚠️ This issue respects the following points: ⚠️

[X] This is a bug, not a question or a configuration/webserver/proxy issue.
[X] This issue is not already reported on Github (I've searched it).
[X] Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
[X] Nextcloud Server is running on 64bit capable CPU, PHP and OS.
[X] I agree to follow Nextcloud's Code of Conduct.

Bug description

occ files:scan someuser generates an exception when .htaccess file is located - in this case the user copied a directory containing valid public_html/.htaccess - is there is a good reason to block user's from having .htaccess file under their own storage - even if the directory was to be shared I assume the Nextcloud code would ignore the file. If that is not the case occ files:scan should generate a proper error, and not an exception.

occ files:scan someuser
Starting scan for user 1 out of 1 (someuser)
Exception during scan: Invalid path: someuser/public_html/courses/.htaccess
#0 /var/www/nextcloud/lib/private/Files/Storage/Common.php(890): OC\Files\Storage\Common->getMetaData()
#1 /var/www/nextcloud/lib/private/Files/Storage/Wrapper/Encoding.php(544): OC\Files\Storage\Common->getDirectoryContent()
#2 [internal function]: OC\Files\Storage\Wrapper\Encoding->getDirectoryContent()
#3 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(410): iterator_to_array()
#4 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(390): OC\Files\Cache\Scanner->handleChildren()
#5 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(393): OC\Files\Cache\Scanner->scanChildren()
#6 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(393): OC\Files\Cache\Scanner->scanChildren()
#7 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(393): OC\Files\Cache\Scanner->scanChildren()
#8 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(393): OC\Files\Cache\Scanner->scanChildren()
#9 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(393): OC\Files\Cache\Scanner->scanChildren()
#10 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(342): OC\Files\Cache\Scanner->scanChildren()
#11 /var/www/nextcloud/lib/private/Files/Utils/Scanner.php(256): OC\Files\Cache\Scanner->scan()
#12 /var/www/nextcloud/apps/files/lib/Command/Scan.php(144): OC\Files\Utils\Scanner->scan()
#13 /var/www/nextcloud/apps/files/lib/Command/Scan.php(200): OCA\Files\Command\Scan->scanFiles()
#14 /var/www/nextcloud/3rdparty/symfony/console/Command/Command.php(255): OCA\Files\Command\Scan->execute()
#15 /var/www/nextcloud/core/Command/Base.php(168): Symfony\Component\Console\Command\Command->run()
#16 /var/www/nextcloud/3rdparty/symfony/console/Application.php(1009): OC\Core\Command\Base->run()
#17 /var/www/nextcloud/3rdparty/symfony/console/Application.php(273): Symfony\Component\Console\Application->doRunCommand()
#18 /var/www/nextcloud/3rdparty/symfony/console/Application.php(149): Symfony\Component\Console\Application->doRun()
#19 /var/www/nextcloud/lib/private/Console/Application.php(211): Symfony\Component\Console\Application->run()
#20 /var/www/nextcloud/console.php(100): OC\Console\Application->run()
#21 /var/www/nextcloud/occ(11): require_once('...')
#22 {main}
+---------+-------+--------------+
| Folders | Files | Elapsed time |
+---------+-------+--------------+
| 1499    | 25174 | 00:00:23     |
+---------+-------+--------------+

Steps to reproduce

occ files:scan

Expected behavior

occ should not generate an exception for .htaccess under user's storage directory.

Installation method

No response

Operating system

RHEL/CentOS

PHP engine version

PHP 8.0

Web server

Apache (supported)

Database engine version

MariaDB

Is this bug present after an update or on a fresh install?

No response

Are you using the Nextcloud Server Encryption module?

Encryption is Disabled

What user-backends are you using?

[X] Default user-backend (database)
[ ] LDAP/ Active Directory
[ ] SSO - SAML
[ ] Other

Configuration report

No response

List of activated Apps

Enabled:
  - accessibility: 1.10.0
  - activity: 2.16.0
  - admin_audit: 1.14.0
  - announcementcenter: 6.3.1
  - apporder: 0.15.0
  - bruteforcesettings: 2.4.0
  - calendar: 3.5.0
  - circles: 24.0.1
  - cloud_federation_api: 1.7.0
  - contacts: 4.2.1
  - contactsinteraction: 1.5.0
  - dav: 1.22.0
  - extract: 1.3.5
  - federatedfilesharing: 1.14.0
  - federation: 1.14.0
  - files: 1.19.0
  - files_external: 1.16.1
  - files_linkeditor: 1.1.11
  - files_markdown: 2.3.6
  - files_pdfviewer: 2.5.0
  - files_rightclick: 1.3.0
  - files_sharing: 1.16.2
  - files_trashbin: 1.14.0
  - files_versions: 1.17.0
  - files_videoplayer: 1.13.0
  - firstrunwizard: 2.13.0
  - groupfolders: 12.0.1
  - login_notes: 1.1.0
  - logreader: 2.9.0
  - lookup_server_connector: 1.12.0
  - metadata: 0.16.0
  - notes: 4.5.1
  - notifications: 2.12.0
  - oauth2: 1.12.0
  - onlyoffice: 7.5.4
  - password_policy: 1.14.0
  - photos: 1.6.0
  - polls: 3.8.0
  - privacy: 1.8.0
  - provisioning_api: 1.14.0
  - quota_warning: 1.14.0
  - recommendations: 1.3.0
  - serverinfo: 1.14.0
  - settings: 1.6.0
  - sharebymail: 1.14.0
  - tasks: 0.14.4
  - text: 3.5.1
  - theming: 1.15.0
  - twofactor_backupcodes: 1.13.0
  - twofactor_totp: 6.4.0
  - updatenotification: 1.14.0
  - video_converter: 1.0.5
  - viewer: 1.8.0
  - workflowengine: 2.6.0

Nextcloud Signing status

No response

Nextcloud Logs

No response

Additional info

No response

szaimen commented 1 year ago

Hi, please update to 24.0.9 or better 25.0.3 and report back if it fixes the issue. Thank you!

My goal is to add a label like e.g. 25-feedback to this ticket of an up-to-date major Nextcloud version where the bug could be reproduced. However this is not going to work without your help. So thanks for all your effort!

If you don't manage to reproduce the issue in time and the issue gets closed but you can reproduce the issue afterwards, feel free to create a new bug report with up-to-date information by following this link: https://github.com/nextcloud/server/issues/new?assignees=&labels=bug%2C0.+Needs+triage&template=BUG_REPORT.yml&title=%5BBug%5D%3A+

soccerdmon11 commented 1 year ago

Hi, I have ran into this issue as well. I have even deleted the file and verified it doesn't exist, but each manual scan errors at that same file.

I am running NextCloud v25.0.3 stable branch

sudo -u www-data php /var/www/nextcloud/occ files:scan --all
Starting scan for user 1 out of 2 (ncadmin)
Starting scan for user 2 out of 2 (soccerdmon)
Exception during scan: Invalid path: files/files/Downloads/D_Downloads/Plex Add-ons/Organizr-master/Organizr-master/config/.htaccess
#0 /var/www/nextcloud/lib/private/Files/Storage/Local.php(265): OC\Files\Storage\Local->getSourcePath()
#1 /var/www/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php(227): OC\Files\Storage\Local->file_exists()
#2 /var/www/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php(227): OC\Files\Storage\Wrapper\Wrapper->file_exists()
#3 /var/www/nextcloud/lib/private/Files/View.php(1349): OC\Files\Storage\Wrapper\Wrapper->file_exists()
#4 /var/www/nextcloud/lib/private/Files/View.php(1394): OC\Files\View->getCacheEntry()
#5 /var/www/nextcloud/lib/private/Metadata/FileEventListener.php(83): OC\Files\View->getFileInfo()
#6 /var/www/nextcloud/lib/private/EventDispatcher/ServiceEventListener.php(87): OC\Metadata\FileEventListener->handle()
#7 /var/www/nextcloud/3rdparty/symfony/event-dispatcher/EventDispatcher.php(251): OC\EventDispatcher\ServiceEventListener->__invoke()
#8 /var/www/nextcloud/3rdparty/symfony/event-dispatcher/EventDispatcher.php(73): Symfony\Component\EventDispatcher\EventDispatcher->callListeners()
#9 /var/www/nextcloud/lib/private/EventDispatcher/EventDispatcher.php(88): Symfony\Component\EventDispatcher\EventDispatcher->dispatch()
#10 /var/www/nextcloud/lib/private/EventDispatcher/EventDispatcher.php(100): OC\EventDispatcher\EventDispatcher->dispatch()
#11 /var/www/nextcloud/lib/private/Files/Utils/Scanner.php(235): OC\EventDispatcher\EventDispatcher->dispatchTyped()
#12 [internal function]: OC\Files\Utils\Scanner->OC\Files\Utils\{closure}()
#13 /var/www/nextcloud/lib/private/Hooks/EmitterTrait.php(106): call_user_func_array()
#14 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(273): OC\Hooks\BasicEmitter->emit()
#15 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(481): OC\Files\Cache\Scanner->removeFromCache()
#16 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(391): OC\Files\Cache\Scanner->handleChildren()
#17 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(394): OC\Files\Cache\Scanner->scanChildren()
#18 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(394): OC\Files\Cache\Scanner->scanChildren()
#19 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(394): OC\Files\Cache\Scanner->scanChildren()
#20 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(394): OC\Files\Cache\Scanner->scanChildren()
#21 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(394): OC\Files\Cache\Scanner->scanChildren()
#22 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(394): OC\Files\Cache\Scanner->scanChildren()
#23 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(394): OC\Files\Cache\Scanner->scanChildren()
#24 /var/www/nextcloud/lib/private/Files/Cache/Scanner.php(342): OC\Files\Cache\Scanner->scanChildren()
#25 /var/www/nextcloud/lib/private/Files/Utils/Scanner.php(256): OC\Files\Cache\Scanner->scan()
#26 /var/www/nextcloud/apps/files/lib/Command/Scan.php(161): OC\Files\Utils\Scanner->scan()
#27 /var/www/nextcloud/apps/files/lib/Command/Scan.php(217): OCA\Files\Command\Scan->scanFiles()
#28 /var/www/nextcloud/3rdparty/symfony/console/Command/Command.php(255): OCA\Files\Command\Scan->execute()
#29 /var/www/nextcloud/core/Command/Base.php(177): Symfony\Component\Console\Command\Command->run()
#30 /var/www/nextcloud/3rdparty/symfony/console/Application.php(1009): OC\Core\Command\Base->run()
#31 /var/www/nextcloud/3rdparty/symfony/console/Application.php(273): Symfony\Component\Console\Application->doRunCommand()
#32 /var/www/nextcloud/3rdparty/symfony/console/Application.php(149): Symfony\Component\Console\Application->doRun()
#33 /var/www/nextcloud/lib/private/Console/Application.php(213): Symfony\Component\Console\Application->run()
#34 /var/www/nextcloud/console.php(100): OC\Console\Application->run()
#35 /var/www/nextcloud/occ(11): require_once('...')
#36 {main}
+---------+--------+--------------+
| Folders | Files  | Elapsed time |
+---------+--------+--------------+
| 21093   | 125255 | 00:02:26     |
+---------+--------+--------------+

Currently enable apps:

Activity    2.17.0
Announcement center 6.4.0
Auto Groups 1.5.1
Automated PDF conversion    1.10.0
Bookmarks   12.0.0
Calendar    4.2.3
Camera RAW Previews 0.8.1
Circles 25.0.0
Collabora Online - Built-in CODE Server 22.5.802
Collaborative tags  1.15.0
Comments    1.15.0
Contacts    5.1.0
Contacts Interaction    1.6.0
Cookbook    0.10.1
Custom menu 3.5.2
Dashboard   7.5.0
Deck    1.8.3
Deleted files   1.15.0
Dropbox integration 1.0.6
External sites  5.0.0
Federation  1.15.0
File sharing    1.17.0
First run wizard    2.14.0
Forms   3.0.4
GitHub integration  1.0.15
Google integration  1.0.9
Group folders   13.1.1
HEIC/HEIF Image Converter   1.3.4
Log Reader  2.10.0
Mail    2.2.2
Maps    0.2.4
Metadata    0.17.0
Monitoring  1.15.0
Music   1.8.1
News    20.0.1
Nextcloud announcements 1.14.0
Nextcloud Office    7.1.0
Notes   4.6.0
Notifications   2.13.1
Notifications for calendar event updates    2.0.0
OneDrive integration    1.1.4
Password policy 1.15.0
Pax Fax 1.0.9
PDF viewer  2.6.0
Photo Sphere Viewer 1.25.2
Photos  2.0.1
Polls   4.1.2
Preview Generator   5.1.1
Privacy 1.9.0
Quota warning   1.15.0
Recommendations 1.4.0
Reddit integration  1.0.5
Registration    2.0.0
Related Resources   1.0.3
Right click 1.4.0
Share by mail   1.15.0
SnappyMail  2.25.4
Splash  2.1.1
Support 1.8.0
Talk    15.0.3
Tasks   0.14.5
Text    3.6.0
Twitter integration 1.0.3
Update notification 1.15.0
Usage survey    1.13.0
User status 1.5.0
Versions    1.18.0
Weather status  1.5.0
Welcome 1.0.8

joshtrichards commented 1 year ago

.htaccess is, by default, on the blacklisted files list in NC.

The blacklist is configurable:

https://docs.nextcloud.com/server/latest/admin_manual/configuration_server/config_sample_php_parameters.html?highlight=blacklist#blacklisted-files

It's generally not desirable to allow that filename since otherwise any NC user could upload it and (under some circumstances) override Apache/web server configuration (and, in turn, bypass NC security authentication to access your data).

I do agree the Invalid path: output is overly vague. I need to poke around a bit more to see what else calls that function to make sure there isn't a good reason for keeping it vague.

joshtrichards commented 9 months ago

So looks like two things need to happen here:

Message should change from "Invalid path:" to something like "Blacklisted path detected:"
Exception should be caught and treated as a warning rather than an error so that the scan continue to run

@soccerdmon11 -

I have ran into this issue as well. I have even deleted the file and verified it doesn't exist, but each manual scan errors at that same file.

That's weird. Almost sounds like the file is already in the filecache. Sort of a catch-22 situation since you're trying to get it removed from the cache. I didn't even realize that was possible. :-) Can you verify this by temporarily setting your blacklisted_files value to an empty array: 'blacklisted_files' => [''], and re-running the scan?

nextcloud / server