search

nextcloud / server

☁️ Nextcloud server, a safe home for all your data
https://nextcloud.com
GNU Affero General Public License v3.0
27.36k stars 4.06k forks source link

[Bug]: occ files:cleanup is not working -- doesn't remove any of the million+ stale entries #43964

Open brianjmurrell opened 8 months ago

brianjmurrell commented 8 months ago

⚠️ This issue respects the following points: ⚠️

Bug description

I have over 1.4M stale entries in my oc_filecache table under just a single path files/DesktopHome:

# echo 'select count(*) from oc_filecache where storage = 1 and path like "files/DesktopHome/%";' | mysql -t nextclouddb
+----------+
| count(*) |
+----------+
|  1450799 |
+----------+

Here is the oc_storages table for reference:

+---------------------------------+------------+-----------+--------------+
| id                              | numeric_id | available | last_checked |
+---------------------------------+------------+-----------+--------------+
| home::brian                     |          1 |         1 |         NULL |
| local::/usr/src/owncloud/data/  |          2 |         1 |         NULL |
| home::jennifer                  |          3 |         1 |         NULL |
| home::brianm                    |          4 |         1 |         NULL |
| local::/var/lib/owncloud/data/  |          5 |         1 |         NULL |
| local::/var/lib/nextcloud/data/ |          6 |         1 |         NULL |
+---------------------------------+------------+-----------+--------------+

But under that path on the actual filesystem there is nothing:

[root@server ~]# ls -l /var/lib/nextcloud/data/brian/files/DesktopHome/
total 0

And yet occ files:cleanup does nothing to clean that mess up:

# sudo -u apache php /usr/share/nextcloud/occ files:cleanup
0 orphaned file cache entries deleted
0 orphaned mount entries deleted

Additionally such file appear in the NC files app but when you try to download or get details on them you get errors about them being non-available. The server also logs:

Mar  3 11:14:41 server Nextcloud[318606]: {"reqId":"ZeSh8KYbHCAExqsm5DVOlQAAABg","level":3,"time":"2024-03-03T16:14:41+00:00","remoteAddr":"10.75.22.192","user":"brian","app":"webdav","method":"GET","url":"/remote.php/dav/files/brian/DesktopHome/3","message":"{\"Exception\":\"Sabre\\\\DAV\\\\Exception\\\\ServiceUnavailable\",\"Message\":\"Could not open file\",\"Code\":0,\"Trace\":[{\"file\":\"/usr/share/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php\",\"line\":85,\"function\":\"get\",\"class\":\"OCA\\\\DAV\\\\Connector\\\\Sabre\\\\File\",\"type\":\"->\",\"args\":[]},{\"file\":\"/usr/share/nextcloud/3rdparty/sabre/event/lib/WildcardEmitterTrait.php\",\"line\":89,\"function\":\"httpGet\",\"class\":\"Sabre\\\\DAV\\\\CorePlugin\",\"type\":\"->\",\"args\":[[\"Sabre\\\\HTTP\\\\Request\"],[\"Sabre\\\\HTTP\\\\Response\"]]},{\"file\":\"/usr/share/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php\",\"line\":472,\"function\":\"emit\",\"class\":\"Sabre\\\\DAV\\\\Server\",\"type\":\"->\",\"args\":[\"method:GET\",[[\"Sabre\\\\HTTP\\\\Request\"],[\"Sabre\\\\HTTP\\\\Response\"]]]},{\"file\":\"/usr/share/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php\",\"line\":253,\"function\":\"invokeMethod\",\"class\":\"Sabre\\\\DAV\\\\Server\",\"type\":\"->\",\"args\":[[\"Sabre\\\\HTTP\\\\Request\"],[\"Sabre\\\\HTTP\\\\Response\"]]},{\"file\":\"/usr/share/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php\",\"line\":321,\"function\":\"start\",\"class\":\"Sabre\\\\DAV\\\\Server\",\"type\":\"->\",\"args\":[]},{\"file\":\"/usr/share/nextcloud/apps/dav/lib/Server.php\",\"line\":370,\"function\":\"exec\",\"class\":\"Sabre\\\\DAV\\\\Server\",\"type\":\"->\",\"args\":[]},{\"file\":\"/usr/share/nextcloud/apps/dav/appinfo/v2/remote.php\",\"line\":35,\"function\":\"exec\",\"class\":\"OCA\\\\DAV\\\\Server\",\"type\":\"->\",\"args\":[]},{\"file\":\"/usr/share/nextcloud/remote.php\",\"line\":172,\"args\":[\"/usr/share/nextcloud/apps/dav/appinfo/v2/remote.php\"],\"function\":\"require_once\"}],\"File\":\"/usr/share/nextcloud/apps/dav/lib/Connector/Sabre/File.php\",\"Line\":505,\"message\":\"Could not open file\",\"exception\":{},\"CustomMessage\":\"Could not open file\"}","userAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36","version":"28.0.2.5"}

Steps to reproduce

Probably pretty obvious give the description above.

Expected behavior

occ files:cleanup should remove those 1.4M or so stale oc_filecache entries.

Installation method

Other Community project

Nextcloud Server version

29

Operating system

RHEL/CentOS

PHP engine version

PHP 8.0

Web server

Apache (supported)

Database engine version

MariaDB

Is this bug present after an update or on a fresh install?

None

Are you using the Nextcloud Server Encryption module?

None

What user-backends are you using?

Configuration report

{
    "system": {
        "log_type": "syslog",
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "updatechecker": false,
        "check_for_working_htaccess": false,
        "asset-pipeline.enabled": false,
        "assetdirectory": "\/var\/lib\/nextcloud",
        "preview_libreoffice_path": "\/usr\/bin\/libreoffice",
        "apps_paths": [
            {
                "path": "\/usr\/share\/nextcloud\/apps",
                "url": "\/apps",
                "writable": false
            },
            {
                "path": "\/var\/lib\/nextcloud\/apps",
                "url": "\/apps-appstore",
                "writable": true
            }
        ],
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "owncloud.example.com",
            "www.example.com"
        ],
        "overwrite.cli.url": "http:\/\/owncloud.example.com\/",
        "dbtype": "mysql",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "version": "29.0.0.19",
        "installed": true,
        "theme": "",
        "maintenance": false,
        "forcessl": false,
        "loglevel": 1,
        "log_authfailip": true,
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpmode": "smtp",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "trashbin_retention_obligation": "auto",
        "htaccess.RewriteBase": "\/",
        "appstore.experimental.enabled": true,
        "mysql.utf8mb4": true
    }
}

List of activated Apps

Enabled:
  - activity: 2.21.1
  - calendar: 4.7.3
  - circles: 29.0.0-dev
  - cloud_federation_api: 1.12.0
  - comments: 1.19.0
  - contacts: 6.0.0
  - contactsinteraction: 1.10.0
  - dashboard: 7.9.0
  - dav: 1.30.1
  - federatedfilesharing: 1.19.0
  - federation: 1.19.0
  - files: 2.1.0
  - files_downloadlimit: 2.0.0
  - files_pdfviewer: 2.10.0
  - files_reminders: 1.2.0
  - files_sharing: 1.21.0
  - files_trashbin: 1.19.0
  - files_versions: 1.22.0
  - firstrunwizard: 2.18.0
  - logreader: 2.14.0
  - lookup_server_connector: 1.17.0
  - nextcloud_announcements: 1.18.0
  - notes: 4.10.0
  - notifications: 2.17.0
  - oauth2: 1.17.0
  - password_policy: 1.19.0
  - photos: 2.5.0
  - privacy: 1.13.0
  - provisioning_api: 1.19.0
  - recommendations: 2.1.0
  - related_resources: 1.4.0
  - serverinfo: 1.19.0
  - settings: 1.12.0
  - sharebymail: 1.19.0
  - support: 1.12.0
  - survey_client: 1.17.0
  - systemtags: 1.19.0
  - text: 3.10.0
  - theming: 2.4.0
  - twofactor_backupcodes: 1.18.0
  - updatenotification: 1.19.1
  - user_status: 1.9.0
  - viewer: 2.3.0
  - weather_status: 1.9.0
  - workflowengine: 2.11.0
Disabled:
  - admin_audit: 1.19.0
  - bruteforcesettings: 2.9.0
  - encryption: 2.17.0
  - files_external: 1.21.0
  - files_rightclick: 0.15.1 (installed 1.6.0)
  - files_texteditor: 2.15.1 (installed 2.15.1)
  - suspicious_login: 7.0.0
  - twofactor_totp: 11.0.0-dev
  - user_ldap: 1.20.0 (installed 1.14.1)

Nextcloud Signing status

No response

Nextcloud Logs

No response

Additional info

No response

solracsf commented 8 months ago

Try to scan instead with occ files:scan --path='brian/files/DesktopHome'.

kesselb commented 8 months ago

local::/usr/src/owncloud/data/ local::/var/lib/owncloud/data/ local::/var/lib/nextcloud/data/

Did you move the datadirectory maybe?

Similar to https://github.com/nextcloud/server/issues/43861

brianjmurrell commented 8 months ago

Try to scan instead with occ files:scan --path='brian/files/DesktopHome'.

Seems an odd thing to need to do when there is a specific command (files:cleanup) that appears to be for removing stale/orphaned entries. Is my understanding of files:cleanup incorrect and you are suggesting files:scan as a work-around or is files:scan actually the correct command for this situation? If that's the case what is files:cleanup supposed to do?

In any case, this failed:

Starting scan for user 1 out of 1 (brian)
PHP Fatal error:  Allowed memory size of 536870912 bytes exhausted (tried to allocate 33554440 bytes) in /usr/share/nextcloud/lib/private/Files/Cache/Cache.php on line 596

Did you move the datadirectory maybe?

Not to my knowledge. I did migrate to NC from OC, a long long time ago however.

Is this relevant in some way?

Doesn't the storage field in oc_filecache relate to the numeric_id of oc_storages? And given that all of the stale data in oc_filecache is in storage 1, are these:

+---------------------------------+------------+-----------+--------------+
| id                              | numeric_id | available | last_checked |
+---------------------------------+------------+-----------+--------------+
| local::/usr/src/owncloud/data/  |          2 |         1 |         NULL |
| local::/var/lib/owncloud/data/  |          5 |         1 |         NULL |
| local::/var/lib/nextcloud/data/ |          6 |         1 |         NULL |
+---------------------------------+------------+-----------+--------------+

Relevant?

How do they relate to the home::* entries?

Clearly there is some old/stale cruft there also. Should I fix oc_storages prior to trying to clean up oc_filecache? If so, how? Can I simply use the mysql CLI to delete the unwanted rows? If so, which rows are wanted and which are unwanted?

Similar to #43861

Except I didn't move my data directory since migrating from OC.

solracsf commented 8 months ago

Well the memory exhaustion shows that your "to-be-deleted" rows are that big and can't fit into the array_merge, otherwise they would be deleted.

Instead of:

$deletedIds = array_merge($deletedIds, $childIds);
$deletedPaths = array_merge($deletedPaths, $childPaths);

change to:

foreach ($childIds as $childId) {
    $deletedIds[] = $childId;
}

foreach ($childPaths as $childPath) {
    $deletedPaths[] = $childPath;
}

Should do the trick, at /usr/share/nextcloud/lib/private/Files/Cache/Cache.php on line 596

brianjmurrell commented 8 months ago

Thanks! That fix goes a bit further:

# sudo -u apache php /usr/share/nextcloud/occ files:scan --path='brian/files/DesktopHome'                                                                           
Starting scan for user 1 out of 1 (brian)
+---------+-------+-----+---------+---------+--------+--------------+
| Folders | Files | New | Updated | Removed | Errors | Elapsed time |
+---------+-------+-----+---------+---------+--------+--------------+
| 1       | 0     | 0   | 0       | 598     | 0      | 00:04:50     |
+---------+-------+-----+---------+---------+--------+--------------+

But as you can see, it's only but a tiny drop in the massive bucket of ~1.4M orphaned/stale oc_filecache entries.

Running it a second time:

# sudo -u apache php /usr/share/nextcloud/occ files:scan --path='brian/files/DesktopHome'
Starting scan for user 1 out of 1 (brian)
+---------+-------+-----+---------+---------+--------+--------------+
| Folders | Files | New | Updated | Removed | Errors | Elapsed time |
+---------+-------+-----+---------+---------+--------+--------------+
| 1       | 0     | 0   | 0       | 0       | 0      | 00:00:01     |
+---------+-------+-----+---------+---------+--------+--------------+

Any further advise or ideas?

solracsf commented 8 months ago

Try to scan the entire instance instead occ files:scan --all? Otherwise no, besides manual cleaning of DB (if you know what you are doing) no other ideas...

brianjmurrell commented 8 months ago

Try to scan the entire instance instead occ files:scan --all?

So what is the relationship of the oc_storages and the oc_filecache tables?

What is home::brian vs. local::/var/lib/nextcloud/data/ for example?

When I run occ files:scan --path='brian/files/DesktopHome' how does it know which oc_storages to be scanning in.

Otherwise no, besides manual cleaning of DB (if you know what you are doing) no other ideas...

Yes, exactly. Manual cleaning if you know what you are doing. Clearly I am not a NextCloud DB expert.

Clearly, if this situation can exist and the only way to fix it is for some kind of internal NC DB expert manual cleaning something is broken. The expertise to do this cleaning needs to exist in a tool so that everyone doesn't need to be an NC DB expert.

Why is occ files:cleanup not fixing this?

Ultimately this situation is hammering the crap out of my NC database because I am getting a constant stream of:

Mar  4 15:54:01 server Nextcloud[290301]: {"reqId":"4iEvJApBswAez31QD9B9","level":2,"time":"2024-03-04T20:54:01+00:00","remoteAddr":"","user":"--","app":"no app in context","method":"","url":"--","message":"{\"Exception\":\"OCP\\\\Files\\\\GenericFileException\",\"Message\":\"\",\"Code\":0,\"Trace\":[{\"file\":\"/usr/share/nextcloud/apps/photos/lib/Listener/SizeMetadataProvider.php\",\"line\":64,\"function\":\"getContent\",\"class\":\"OC\\\\Files\\\\Node\\\\File\",\"type\":\"->\",\"args\":[]},{\"file\":\"/usr/share/nextcloud/lib/private/EventDispatcher/ServiceEventListener.php\",\"line\":86,\"function\":\"handle\",\"class\":\"OCA\\\\Photos\\\\Listener\\\\SizeMetadataProvider\",\"type\":\"->\",\"args\":[[\"OCP\\\\FilesMetadata\\\\Event\\\\MetadataLiveEvent\"]]},{\"file\":\"/usr/share/nextcloud/3rdparty/symfony/event-dispatcher/EventDispatcher.php\",\"line\":230,\"function\":\"__invoke\",\"class\":\"OC\\\\EventDispatcher\\\\ServiceEventListener\",\"type\":\"->\",\"args\":[[\"OCP\\\\FilesMetadata\\\\Event\\\\MetadataLiveEvent\"],\"OCP\\\\FilesMetadata\\\\Event\\\\MetadataLiveEvent\",[\"Symfony\\\\Component\\\\EventDispatcher\\\\EventDispatcher\"]]},{\"file\":\"/usr/share/nextcloud/3rdparty/symfony/event-dispatcher/EventDispatcher.php\",\"line\":59,\"function\":\"callListeners\",\"class\":\"Symfony\\\\Component\\\\EventDispatcher\\\\EventDispatcher\",\"type\":\"->\",\"args\":[[[\"Closure\"],[\"Closure\"],[\"Closure\"],[\"Closure\"]],\"OCP\\\\FilesMetadata\\\\Event\\\\MetadataLiveEvent\",[\"OCP\\\\FilesMetadata\\\\Event\\\\MetadataLiveEvent\"]]},{\"file\":\"/usr/share/nextcloud/lib/private/EventDispatcher/EventDispatcher.php\",\"line\":94,\"function\":\"dispatch\",\"class\":\"Symfony\\\\Component\\\\EventDispatcher\\\\EventDispatcher\",\"type\":\"->\",\"args\":[[\"OCP\\\\FilesMetadata\\\\Event\\\\MetadataLiveEvent\"],\"OCP\\\\FilesMetadata\\\\Event\\\\MetadataLiveEvent\"]},{\"file\":\"/usr/share/nextcloud/lib/private/EventDispatcher/EventDispatcher.php\",\"line\":106,\"function\":\"dispatch\",\"class\":\"OC\\\\EventDispatcher\\\\EventDispatcher\",\"type\":\"->\",\"args\":[\"OCP\\\\FilesMetadata\\\\Event\\\\MetadataLiveEvent\",[\"OCP\\\\FilesMetadata\\\\Event\\\\MetadataLiveEvent\"]]},{\"file\":\"/usr/share/nextcloud/lib/private/FilesMetadata/FilesMetadataManager.php\",\"line\":115,\"function\":\"dispatchTyped\",\"class\":\"OC\\\\EventDispatcher\\\\EventDispatcher\",\"type\":\"->\",\"args\":[[\"OCP\\\\FilesMetadata\\\\Event\\\\MetadataLiveEvent\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":109,\"function\":\"refreshMetadata\",\"class\":\"OC\\\\FilesMetadata\\\\FilesMetadataManager\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\File\"],3]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":88,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\LazyUserFolder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":74,\"function\":\"scanFilesForUser\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[\"brian\"]},{\"file\":\"/usr/share/nextcloud/lib/public/BackgroundJob/Job.php\",\"line\":81,\"function\":\"run\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[null]},{\"file\":\"/usr/share/nextcloud/lib/public/BackgroundJob/TimedJob.php\",\"line\":102,\"function\":\"start\",\"class\":\"OCP\\\\BackgroundJob\\\\Job\",\"type\":\"->\",\"args\":[[\"OC\\\\BackgroundJob\\\\JobList\"]]},{\"file\":\"/usr/share/nextcloud/lib/public/BackgroundJob/TimedJob.php\",\"line\":92,\"function\":\"start\",\"class\":\"OCP\\\\BackgroundJob\\\\TimedJob\",\"type\":\"->\",\"args\":[[\"OC\\\\BackgroundJob\\\\JobList\"]]},{\"file\":\"/usr/share/nextcloud/cron.php\",\"line\":152,\"function\":\"execute\",\"class\":\"OCP\\\\BackgroundJob\\\\TimedJob\",\"type\":\"->\",\"args\":[[\"OC\\\\BackgroundJob\\\\JobList\"],[\"OC\\\\Log\"]]}],\"File\":\"/usr/share/nextcloud/lib/private/Files/Node/File.php\",\"Line\":56,\"message\":\"Error while generating metadata for fileid 3639031\",\"exception\":{},\"CustomMessage\":\"Error while generating metadata for fileid 3639031\"}","userAgent":"--","version":"28.0.2.5"}
Mar  4 15:54:29 server Nextcloud[290301]: {"reqId":"4iEvJApBswAez31QD9B9","level":3,"time":"2024-03-04T20:54:29+00:00","remoteAddr":"","user":"--","app":"PHP","method":"","url":"--","message":"file_get_contents(/var/lib/nextcloud/data/brian/files/DesktopHome/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/dosdevices/e:/.wine.old/dosdevices/z:/net/pc/home/brian/google-earth/res/pal2/icon13.png): Failed to open stream: No such file or directory at /usr/share/nextcloud/lib/private/Files/Storage/Local.php#327","userAgent":"--","version":"28.0.2.5","data":{"app":"PHP"}}
Mar  4 15:54:34 server Nextcloud[290301]: {"reqId":"4iEvJApBswAez31QD9B9","level":2,"time":"2024-03-04T20:54:34+00:00","remoteAddr":"","user":"--","app":"no app in context","method":"","url":"--","message":"{\"Exception\":\"OCP\\\\Files\\\\GenericFileException\",\"Message\":\"\",\"Code\":0,\"Trace\":[{\"file\":\"/usr/share/nextcloud/apps/photos/lib/Listener/SizeMetadataProvider.php\",\"line\":64,\"function\":\"getContent\",\"class\":\"OC\\\\Files\\\\Node\\\\File\",\"type\":\"->\",\"args\":[]},{\"file\":\"/usr/share/nextcloud/lib/private/EventDispatcher/ServiceEventListener.php\",\"line\":86,\"function\":\"handle\",\"class\":\"OCA\\\\Photos\\\\Listener\\\\SizeMetadataProvider\",\"type\":\"->\",\"args\":[[\"OCP\\\\FilesMetadata\\\\Event\\\\MetadataLiveEvent\"]]},{\"file\":\"/usr/share/nextcloud/3rdparty/symfony/event-dispatcher/EventDispatcher.php\",\"line\":230,\"function\":\"__invoke\",\"class\":\"OC\\\\EventDispatcher\\\\ServiceEventListener\",\"type\":\"->\",\"args\":[[\"OCP\\\\FilesMetadata\\\\Event\\\\MetadataLiveEvent\"],\"OCP\\\\FilesMetadata\\\\Event\\\\MetadataLiveEvent\",[\"Symfony\\\\Component\\\\EventDispatcher\\\\EventDispatcher\"]]},{\"file\":\"/usr/share/nextcloud/3rdparty/symfony/event-dispatcher/EventDispatcher.php\",\"line\":59,\"function\":\"callListeners\",\"class\":\"Symfony\\\\Component\\\\EventDispatcher\\\\EventDispatcher\",\"type\":\"->\",\"args\":[[[\"Closure\"],[\"Closure\"],[\"Closure\"],[\"Closure\"]],\"OCP\\\\FilesMetadata\\\\Event\\\\MetadataLiveEvent\",[\"OCP\\\\FilesMetadata\\\\Event\\\\MetadataLiveEvent\"]]},{\"file\":\"/usr/share/nextcloud/lib/private/EventDispatcher/EventDispatcher.php\",\"line\":94,\"function\":\"dispatch\",\"class\":\"Symfony\\\\Component\\\\EventDispatcher\\\\EventDispatcher\",\"type\":\"->\",\"args\":[[\"OCP\\\\FilesMetadata\\\\Event\\\\MetadataLiveEvent\"],\"OCP\\\\FilesMetadata\\\\Event\\\\MetadataLiveEvent\"]},{\"file\":\"/usr/share/nextcloud/lib/private/EventDispatcher/EventDispatcher.php\",\"line\":106,\"function\":\"dispatch\",\"class\":\"OC\\\\EventDispatcher\\\\EventDispatcher\",\"type\":\"->\",\"args\":[\"OCP\\\\FilesMetadata\\\\Event\\\\MetadataLiveEvent\",[\"OCP\\\\FilesMetadata\\\\Event\\\\MetadataLiveEvent\"]]},{\"file\":\"/usr/share/nextcloud/lib/private/FilesMetadata/FilesMetadataManager.php\",\"line\":115,\"function\":\"dispatchTyped\",\"class\":\"OC\\\\EventDispatcher\\\\EventDispatcher\",\"type\":\"->\",\"args\":[[\"OCP\\\\FilesMetadata\\\\Event\\\\MetadataLiveEvent\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":109,\"function\":\"refreshMetadata\",\"class\":\"OC\\\\FilesMetadata\\\\FilesMetadataManager\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\File\"],3]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":99,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\Folder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":88,\"function\":\"scanFolder\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[[\"OC\\\\Files\\\\Node\\\\LazyUserFolder\"]]},{\"file\":\"/usr/share/nextcloud/core/BackgroundJobs/GenerateMetadataJob.php\",\"line\":74,\"function\":\"scanFilesForUser\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[\"brian\"]},{\"file\":\"/usr/share/nextcloud/lib/public/BackgroundJob/Job.php\",\"line\":81,\"function\":\"run\",\"class\":\"OC\\\\Core\\\\BackgroundJobs\\\\GenerateMetadataJob\",\"type\":\"->\",\"args\":[null]},{\"file\":\"/usr/share/nextcloud/lib/public/BackgroundJob/TimedJob.php\",\"line\":102,\"function\":\"start\",\"class\":\"OCP\\\\BackgroundJob\\\\Job\",\"type\":\"->\",\"args\":[[\"OC\\\\BackgroundJob\\\\JobList\"]]},{\"file\":\"/usr/share/nextcloud/lib/public/BackgroundJob/TimedJob.php\",\"line\":92,\"function\":\"start\",\"class\":\"OCP\\\\BackgroundJob\\\\TimedJob\",\"type\":\"->\",\"args\":[[\"OC\\\\BackgroundJob\\\\JobList\"]]},{\"file\":\"/usr/share/nextcloud/cron.php\",\"line\":152,\"function\":\"execute\",\"class\":\"OCP\\\\BackgroundJob\\\\TimedJob\",\"type\":\"->\",\"args\":[[\"OC\\\\BackgroundJob\\\\JobList\"],[\"OC\\\\Log\"]]}],\"File\":\"/usr/share/nextcloud/lib/private/Files/Node/File.php\",\"Line\":56,\"message\":\"Error while generating metadata for fileid 3639021\",\"exception\":{},\"CustomMessage\":\"Error while generating metadata for fileid 3639021\"}","userAgent":"--","version":"28.0.2.5"}
Mar  4 15:55:01 server Nextcloud[290301]: {"reqId":"4iEvJApBswAez31QD9B9","level":3,"time":"2024-03-04T20:55:01+00:00","remoteAddr":"","user":"--","app":"PHP","method":"","url":"--","message":"file_get_contents(/var/lib/nextcloud/data/brian/files/DesktopHome/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/dosdevices/e:/.wine.old/dosdevices/z:/net/pc/home/brian/google-earth/res/pal2/icon13s.png): Failed to open stream: No such file or directory at /usr/share/nextcloud/lib/private/Files/Storage/Local.php#327","userAgent":"--","version":"28.0.2.5","data":{"app":"PHP"}}

which seems to be some job hell-bent on trying to touch every single one of those 1.4M orphaned oc_filecache entries.

brianjmurrell commented 8 months ago

Try to scan the entire instance instead occ files:scan --all?

# sudo -u apache php /usr/share/nextcloud/occ files:scan --all
Starting scan for user 1 out of 3 (brian)
…
+---------+-------+-----+---------+---------+--------+--------------+
| Folders | Files | New | Updated | Removed | Errors | Elapsed time |
+---------+-------+-----+---------+---------+--------+--------------+
| 972     | 21262 | 0   | 376     | 2       | 1      | 17:04:16     |
+---------+-------+-----+---------+---------+--------+--------------+

So nothing near the ~1.4M orphaned oc_filecache entries:

# echo 'select count(*) from oc_filecache where storage = 1 and path like "files/DesktopHome/%";' | mysql -t nextclouddb                                            
+----------+
| count(*) |
+----------+
|  1427878 |
+----------+

So still no solution and ultimately occ files:cleanup seems broken as it seems that is what should be able do this cleanup.

joshtrichards commented 7 months ago

occ files:cleanup only cleans up filecache entries that reference storage ids that no longer exist. It doesn't look at what's on-disk at all. Since storage id 1 very much still exists (it's your home directory) there's nothing for it to do in your case:

+---------------------------------+------------+-----------+--------------+
| id                              | numeric_id | available | last_checked |
+---------------------------------+------------+-----------+--------------+
 | home::brian                     |          1 |         1 |         NULL |

The bottom line is that you're trying to update the Nextcloud Server filecache so that its content matches what you have on-disk, correct?

I presume the cause of this is that you bypassed Nextcloud and removed these from your server's disk directly or something?

For this, occ files:scan is what you want.

Clearly, if this situation can exist and the only way to fix it is for some kind of internal NC DB expert manual cleaning something is broken. The expertise to do this cleaning needs to exist in a tool so that everyone doesn't need to be an NC DB expert.

Agreed. Let's figure out what's going on here. A files:scan should catch this, since these files have been removed from disk. Something is going on here causing it not to behave as would otherwise be expected.

# sudo -u apache php /usr/share/nextcloud/occ files:scan --all
Starting scan for user 1 out of 3 (brian)
…
+---------+-------+-----+---------+---------+--------+--------------+
| Folders | Files | New | Updated | Removed | Errors | Elapsed time |
+---------+-------+-----+---------+---------+--------+--------------+
| 972     | 21262 | 0   | 376     | 2       | 1      | 17:04:16     |
+---------+-------+-----+---------+---------+--------+--------------+

Try running with -vvv for verbose mode to get a bit more info.

But as you can see, it's only but a tiny drop in the massive bucket of ~1.4M orphaned/stale oc_filecache entries.

Running it a second time:

Can you share a sampling of a few of the apparently remaining entries? e.g.

SELECT * FROM oc_filecache WHERE storage=1 AND path LIKE "files/DesktopHome/%" LIMIT 5;

/var/lib/nextcloud/data/brian/files/DesktopHome/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/dosdevices/e:/.wine.old/dosdevices/z:/net/pc/home/brian/google-earth/res/pal2/icon13.png

That is one seriously strange file path. How'd this happen to start with? :)

brianjmurrell commented 7 months ago

occ files:cleanup only cleans up filecache entries that reference storage ids that no longer exist. It doesn't look at what's on-disk at all.

I see.

The bottom line is that you're trying to update the Nextcloud Server filecache so that its content matches what you have on-disk, correct?

Correct.

I presume the cause of this is that you bypassed Nextcloud and removed these from your server's disk directly or something?

Kinda. It was an experiment with symlinking my ~ into the Nextcloud files namespace to see if I could get access to my ~ through NC.

Agreed. Let's figure out what's going on here. A files:scan should catch this, since these files have been removed from disk. Something is going on here causing it not to behave as would otherwise be expected.

Indeed.

Try running with -vvv for verbose mode to get a bit more info.

Sure. As a point of reference before running that:

# echo 'SELECT count(*) FROM oc_filecache WHERE storage=1 AND path LIKE "files/DesktopHome/%";' | mysql -t nextclouddb                                              
+----------+                                                                                                                                                                       
| count(*) |
+----------+
|  1357041 |
+----------+

But with the -vvv I am getting a line of output per file. Not going to be able to paste that all here. But where it gets interesting is with:

…
        File    /brian/files/.SeedVaultAndroidBackup/1651847576164/kv/@pm@/Y29tLmFuZHJvaWQudGhlbWUuaWNvbl9wYWNrLnNhbS5zZXR0aW5ncw
        File    /brian/files/.SeedVaultAndroidBackup/1651847576164/kv/@pm@/Y29tLmdvb2dsZS5hbmRyb2lkLmFwcHMuYWRt
        File    /brian/files/.SeedVaultAndroidBackup/1651847576164/kv/@pm@/Y29tLmdvb2dsZS5hbmRyb2lkLmFwcHMuZG9jcy5lZGl0b3JzLnNoZWV0cw
        File    /brian/files/.SeedVaultAndroidBackup/1651847576164/kv/@pm@/Y29tLmFuZHJvaWQucHJveHloYW5kbGVy
        File    /brian/files/.SeedVaultAndroidBackup/1651847576164/kv/@pm@/Y29tLmFuZHJvaWQuZG9jdW1lbnRzdWk
        File    /brian/files/.SeedVaultAndroidBackup/1651847576164/kv/@pm@/Y29tLnF1YWxjb21tLnF0aS5xbXMuc2VydmljZS50ZWxlbWV0cnk
        File    /brian/files/.SeedVaultAndroidBackup/1651847576164/kv/@pm@/Y29tLmFuZHJvaWQudGhlbWUuY29sb3Iuc3BhY2U
Exception during scan: An exception occurred while executing a query: SQLSTATE[40001]: Serialization failure: 1213 Deadlock found when trying to get lock; try restarting transaction
#0 /usr/share/nextcloud/lib/private/DB/QueryBuilder/QueryBuilder.php(328): OC\DB\Exceptions\DbalException::wrap(Object(Doctrine\DBAL\Exception\DeadlockException))
#1 /usr/share/nextcloud/lib/private/DB/Adapter.php(146): OC\DB\QueryBuilder\QueryBuilder->executeStatement()
#2 /usr/share/nextcloud/lib/private/DB/Connection.php(360): OC\DB\Adapter->insertIgnoreConflict('file_locks', Array)
#3 /usr/share/nextcloud/lib/private/DB/ConnectionAdapter.php(110): OC\DB\Connection->insertIgnoreConflict('file_locks', Array)
#4 /usr/share/nextcloud/lib/private/Lock/DBLockingProvider.php(92): OC\DB\ConnectionAdapter->insertIgnoreConflict('file_locks', Array)
#5 /usr/share/nextcloud/lib/private/Lock/DBLockingProvider.php(130): OC\Lock\DBLockingProvider->initLockField('files/3c07078e8...', 1)
#6 /usr/share/nextcloud/lib/private/Files/Storage/Common.php(766): OC\Lock\DBLockingProvider->acquireLock('files/3c07078e8...', 1, 'home::brian::fi...')
#7 /usr/share/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php(616): OC\Files\Storage\Common->acquireLock('files/.SeedVaul...', 1, Object(OC\Lock\DBLockingProvider))
#8 /usr/share/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php(616): OC\Files\Storage\Wrapper\Wrapper->acquireLock('files/.SeedVaul...', 1, Object(OC\Lock\DBLockingProvider))
#9 /usr/share/nextcloud/lib/private/Files/Cache/Scanner.php(153): OC\Files\Storage\Wrapper\Wrapper->acquireLock('files/.SeedVaul...', 1, Object(OC\Lock\DBLockingProvider))
#10 /usr/share/nextcloud/lib/private/Files/Cache/Scanner.php(480): OC\Files\Cache\Scanner->scanFile('files/.SeedVaul...', 3, 4408283, Object(OC\Files\Cache\CacheEntry), true, Array)
#11 /usr/share/nextcloud/lib/private/Files/Cache/Scanner.php(404): OC\Files\Cache\Scanner->handleChildren('files/.SeedVaul...', true, 3, 4408283, true, 17062, false)
#12 /usr/share/nextcloud/lib/private/Files/Cache/Scanner.php(409): OC\Files\Cache\Scanner->scanChildren('files/.SeedVaul...', true, 3, 4408283, true, 40760, false)
#13 /usr/share/nextcloud/lib/private/Files/Cache/Scanner.php(409): OC\Files\Cache\Scanner->scanChildren('files/.SeedVaul...', true, 3, 4408280, true, 511169, 0)
#14 /usr/share/nextcloud/lib/private/Files/Cache/Scanner.php(409): OC\Files\Cache\Scanner->scanChildren('files/.SeedVaul...', true, 3, 4393435, true, 4696945144, 0)
#15 /usr/share/nextcloud/lib/private/Files/Cache/Scanner.php(409): OC\Files\Cache\Scanner->scanChildren('files/.SeedVaul...', true, 3, 4393175, true, 27684575322, 0)
#16 /usr/share/nextcloud/lib/private/Files/Cache/Scanner.php(409): OC\Files\Cache\Scanner->scanChildren('files', true, 3, 2, true, 38935159624, 0)
#17 /usr/share/nextcloud/lib/private/Files/Cache/Scanner.php(354): OC\Files\Cache\Scanner->scanChildren('', true, 3, 1, true, 38935159624)
#18 /usr/share/nextcloud/lib/private/Files/Utils/Scanner.php(277): OC\Files\Cache\Scanner->scan('', true, 3)
#19 /usr/share/nextcloud/apps/files/lib/Command/Scan.php(180): OC\Files\Utils\Scanner->scan('/brian', true, NULL)
#20 /usr/share/nextcloud/apps/files/lib/Command/Scan.php(241): OCA\Files\Command\Scan->scanFiles('brian', '/brian', NULL, Object(Symfony\Component\Console\Output\ConsoleOutput), false, true, false)
#21 /usr/share/nextcloud/3rdparty/symfony/console/Command/Command.php(298): OCA\Files\Command\Scan->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#22 /usr/share/nextcloud/core/Command/Base.php(177): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#23 /usr/share/nextcloud/3rdparty/symfony/console/Application.php(1040): OC\Core\Command\Base->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#24 /usr/share/nextcloud/3rdparty/symfony/console/Application.php(301): Symfony\Component\Console\Application->doRunCommand(Object(OCA\Files\Command\Scan), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#25 /usr/share/nextcloud/3rdparty/symfony/console/Application.php(171): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#26 /usr/share/nextcloud/lib/private/Console/Application.php(213): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#27 /usr/share/nextcloud/console.php(100): OC\Console\Application->run()
#28 /usr/share/nextcloud/occ(11): require_once('/usr/share/next...')
#29 {main}

Starting scan for user 2 out of 3 ([redacted])
…
+---------+-------+-----+---------+---------+--------+--------------+
| Folders | Files | New | Updated | Removed | Errors | Elapsed time |
+---------+-------+-----+---------+---------+--------+--------------+
| 752     | 13699 | 0   | 0       | 0       | 2      | 00:45:09     |
+---------+-------+-----+---------+---------+--------+--------------+

So it seems that the scan for that user bombs out there which is probably why it doesn't get to removing the gazillion entries for the missing files. That seems to happen after about ~5400 files and folders of the way through that user. That said, the second user has no files but the third user's ~9050 files are all scanned.

Can you share a sampling of a few of the apparently remaining entries? e.g.

+---------+---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------+---------+----------------------------------------+----------+----------+--------+------------+---------------+-----------+------------------+----------------------------------+-------------+----------+
| fileid  | storage | path                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 | path_hash                        | parent  | name                                   | mimetype | mimepart | size   | mtime      | storage_mtime | encrypted | unencrypted_size | etag                             | permissions | checksum |
+---------+---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------+---------+----------------------------------------+----------+----------+--------+------------+---------------+-----------+------------------+----------------------------------+-------------+----------+
| 3293261 |       1 | files/DesktopHome/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/dosdevices/h:/.wine/dosdevices/z:/net/pc/home/brian/.ccache/0/b/f7c379f96657096bbc400863924e0a-1620503   | 00000b5a5b3fa2c14badd40d366297e6 | 3293078 | f7c379f96657096bbc400863924e0a-1620503 |       14 |        3 | 224464 | 1158718487 |    1158718487 |         0 |                0 | 054c84bdfad0a6185209dd54384e1f6f |          17 | NULL     |
| 3456617 |       1 | files/DesktopHome/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/dosdevices/h:/memstats/1168735663                                                                        | 000012b3b4872dbc4ce57a912d01af57 | 2767903 | 1168735663                             |       14 |        3 |   3498 | 1168735664 |    1168735664 |         0 |                0 | 16700755be86cb3eeaf11918125c4bfb |          17 | NULL     |
| 3381747 |       1 | files/DesktopHome/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/dosdevices/h:/.wine/dosdevices/z:/net/linux/usr/share/pyshared/oauth-1.0.1.egg-info/dependency_links.txt | 000015f0a682c0d9c9ea6ed1ac5612d5 | 3381436 | dependency_links.txt                   |       25 |       24 |      1 | 1325300566 |    1325300566 |         0 |                0 | 2f31bdaeb1bbf2c505f636edd223f890 |          17 | NULL     |
| 2949014 |       1 | files/DesktopHome/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/dosdevices/h:/.wine.old/dosdevices/z:/net/pc/home/brian/.xbmc-old/userdata/Thumbnails/1/189b137e.jpg     | 000018863faeef23ef7d6e9fae2c270f | 2947062 | 189b137e.jpg                           |        9 |        8 |  60717 | 1411430002 |    1411430002 |         0 |                0 | 28f96134f1a91606fc911d0d39692360 |          17 | NULL     |
| 3185421 |       1 | files/DesktopHome/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/dosdevices/h:/.wine/dosdevices/z:/proc/72/task/72/gid_map                                                | 000027cf290c372b98e54b315c46bc29 | 3185394 | gid_map                                |       14 |        3 |      0 | 1442238562 |    1442238562 |         0 |                0 | 33206ed6c8041decb90d06aa41018bc8 |          17 | NULL     |
+---------+---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------+---------+----------------------------------------+----------+----------+--------+------------+---------------+-----------+------------------+----------------------------------+-------------+----------+

That is one seriously strange file path. How'd this happen to start with? :)

Indeed, it is. It happened as described above. My guess would further be that there was some kind of circular symlinking going on somewhere in there.

brianjmurrell commented 6 months ago

Was anything in my previous comment helpful in resolving this?

brianjmurrell commented 5 months ago

Given all of the information I have provided earlier, including the Exception being thrown above, is there any way to make some progress on this?

joshtrichards commented 5 months ago

Well you did something somewhat unconventional. :-)

It was an experiment with symlinking my ~ into the Nextcloud files namespace to see if I could get access to my ~ through NC.

It's going to likely require manual database intervention if you want to fix your existing installation from the looks of it. And this isn't a general support channel. You might be able to find some help on the Community Help Forum (https://help.nextcloud.com) to figure out some db queries to clear out the appropriate entries manually. At least all the entries seem to have a prefix to work with to identify them (DesktopHome) at least in the db.

There is always room for improving the robustness of the existing tools, but there's only so much that can be done w/o touching the database in a situation like this.

Your best bet is probably go restore from a backup before the experiment.

As for the last scan errors that are seedvault related, see their own docs which tell you what to do with Nextcloud:

https://github.com/seedvault-app/seedvault/wiki/FAQ#where-is-my-backup-stored

brianjmurrell commented 5 months ago

It's going to likely require manual database intervention if you want to fix your existing installation from the looks of it.

But why? That should not be necessary. There should be functional and working tools to resync the database with what's on the disk -- removing entries from the database that no longer exist on disk, as one of the operations.

And this isn't a general support channel.

I'm not looking for "support". I am looking for a tool that [re-]syncs the database and disk. You have suggested that occ files:scan is the tool for this, but clearly it's not working.

Except that potentially the scan would fix it if it didn't bail out with an exception as previously reported Having now updated to NextCloud 29, I no longer get the exception and the occ files:scan runs to completion but does not remove the entries from the database that are no longer present on disk. How is this not a bug?

kesselb commented 5 months ago

The occ command info:file can print some details about files. Could you run it on some of your stale files and share the output?

My understanding is as well that files:scan should remove non-existing files. Could you update your initial post with a recent system report for completness?

kesselb commented 5 months ago

Please make sure to have a working backup just in case.

Another command is files:repair-tree. Run it with verbose and dry-run first.

brianjmurrell commented 5 months ago

The occ command info:file can print some details about files. Could you run it on some of your stale files and share the output?

Interesting!

$ occ info:file '/brian/files/DesktopHome/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/windows/system32/gecko/1.0.0/wine_gecko/nssutil3.dll'                                                                                                                                          
nssutil3.dll
  fileid: 2739821
  mimetype: application/octet-stream
  modified: August 4, 2009, 2:42:14 AM UTC
  not encrypted
  size: 65 KB
  etag: 5fc24f3754edc7b89989eb31e584a84b
  warning: file not found in storage

So, NC properly recognizes that it doesn't exist on the underlying storage.

My understanding is as well that files:scan should remove non-existing files. Could you update your initial post with a recent system report for completness?

Everything there is still relevant except that I am using NC 29 now.

kesselb commented 5 months ago

Everything there is still relevant except that I am using NC 29 now.

The configuration report and list of activated apps are missing.

brianjmurrell commented 5 months ago

Everything there is still relevant except that I am using NC 29 now.

The configuration report and list of activated apps are missing.

Are those really relevant to this kind of issue? What exactly are you looking for?

brianjmurrell commented 5 months ago

files:repair-tree.

Doesn't seem to be doing much:

$ php /usr/share/nextcloud/occ files:repair-tree --dry-run 
Found 8 file entries with an invalid path
kesselb commented 5 months ago

Are those really relevant to this kind of issue? What exactly are you looking for?

The configuration report and list of activated apps help us to get a better overview of your system. I'm not aware that there is an configuration flag like "keep_missing_files_in_filecache" but you never know ;)

For example, we often take a look at the apps_install_overwrite section to see if incompatible apps are running. However, asking all those questions is usually a time-consuming process, and therefore we ask for the information right away, to make our and your life easier. It's understandable that you don't want to share the production URL or other information, that you consider confidential, and thus it's totally fine to remove / replace them.

Found 8 file entries with an invalid path

It's start, no? ;)

Sorry, I forgot to mention that you need to use --verbose to see which files.

Exception during scan: An exception occurred while executing a query: SQLSTATE[40001]: Serialization failure: 1213 Deadlock found when trying to get lock; try restarting transaction

You shared a stack trace in https://github.com/nextcloud/server/issues/43964#issuecomment-2038075310. The scanner sets a lock for the folder to prevent other components in Nextcloud from modifying it in between. In your case, that did not work because another process (likely also Nextcloud) tried to set a lock as well and the result was a deadlock. Is there a chance that the SeedVaultBackup was started/running when you executed files:scan? ;)

The stack trace shows that you are using the database locking (also something we look in the system report for) which is slow. We recommend configuring redis nowadays for the file locking. I don't think it's related to your ghost files, just something to keep in mind.

Did you have any luck with running files:scan --all -vvv or does it still crash? I would like you to run the command again if possible, redirect the output into a file and have a look then if the ghost files are also scanned.

If that does not lead to something, we have to add some debugging code here: https://github.com/nextcloud/server/blob/9735b5d61f8577fea01810b5de3e1dab2ec6c77e/lib/private/Files/Cache/Scanner.php#L457-L461

That's where the magic happens or does not in your case ;) existingChildren is the data from filecache, newChildNames is the data found on disk.

brianjmurrell commented 5 months ago

The configuration report and list of activated apps help us to get a better overview of your system. I'm not aware that there is an configuration flag like "keep_missing_files_in_filecache" but you never know ;)

I've added the configuration report and app list.

It's start, no? ;)

I don't think so. They are not interesting for the reasons of this bug report.

Sorry, I forgot to mention that you need to use --verbose to see which files.

Found 8 file entries with an invalid path
Path of file 3075514 is files/DesktopHome/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/dosdevices/h:/.wine.old/dosdevices/z:/net/pc/home/mythtv/.dvdcss/`-524353908cff8e79-0000000000 but should be files/DesktopHome/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/dosdevices/h:/.wine.old/dosdevices/z:/net/pc/home/mythtv/.dvdcss/-524353908cff8e79-0000000000 based on its parent
Path of file 3075522 is files/DesktopHome/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/dosdevices/h:/.wine.old/dosdevices/z:/net/pc/home/mythtv/.dvdcss/`25-bb985a43437746f8-0000000000 but should be files/DesktopHome/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/dosdevices/h:/.wine.old/dosdevices/z:/net/pc/home/mythtv/.dvdcss/25-bb985a43437746f8-0000000000 based on its parent
Path of file 3349891 is files/DesktopHome/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/dosdevices/h:/.wine/dosdevices/z:/net/pc/home/mythtv/.dvdcss/`-524353908cff8e79-0000000000 but should be files/DesktopHome/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/dosdevices/h:/.wine/dosdevices/z:/net/pc/home/mythtv/.dvdcss/-524353908cff8e79-0000000000 based on its parent
Path of file 3349899 is files/DesktopHome/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/dosdevices/h:/.wine/dosdevices/z:/net/pc/home/mythtv/.dvdcss/`25-bb985a43437746f8-0000000000 but should be files/DesktopHome/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/dosdevices/h:/.wine/dosdevices/z:/net/pc/home/mythtv/.dvdcss/25-bb985a43437746f8-0000000000 based on its parent
Path of file 3726018 is files/DesktopHome/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/dosdevices/e:/.wine.old/dosdevices/z:/net/pc/home/mythtv/.dvdcss/`-524353908cff8e79-0000000000 but should be files/DesktopHome/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/dosdevices/e:/.wine.old/dosdevices/z:/net/pc/home/mythtv/.dvdcss/-524353908cff8e79-0000000000 based on its parent
Path of file 3726026 is files/DesktopHome/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/dosdevices/e:/.wine.old/dosdevices/z:/net/pc/home/mythtv/.dvdcss/`25-bb985a43437746f8-0000000000 but should be files/DesktopHome/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/dosdevices/e:/.wine.old/dosdevices/z:/net/pc/home/mythtv/.dvdcss/25-bb985a43437746f8-0000000000 based on its parent
Path of file 4125450 is files/DesktopHome/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/dosdevices/e:/.wine/dosdevices/z:/net/pc/home/mythtv/.dvdcss/`-524353908cff8e79-0000000000 but should be files/DesktopHome/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/dosdevices/e:/.wine/dosdevices/z:/net/pc/home/mythtv/.dvdcss/-524353908cff8e79-0000000000 based on its parent
Path of file 4125458 is files/DesktopHome/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/dosdevices/e:/.wine/dosdevices/z:/net/pc/home/mythtv/.dvdcss/`25-bb985a43437746f8-0000000000 but should be files/DesktopHome/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/dosdevices/e:/.wine/dosdevices/z:/net/pc/home/mythtv/.dvdcss/25-bb985a43437746f8-0000000000 based on its parent

Did you have any luck with running files:scan --all -vvv or does it still crash?

Yes, as noted previously, now on NC 29, that does run without exception. But doesn't find any files to remove.

I would like you to run the command again if possible, redirect the output into a file and have a look then if the ghost files are also scanned.

Given the number of orphanded files (nearly 1.5M) I cannot be sure that not a single one exists in the scan but a spotcheck leaves me pretty confident that they are not. Not surprising since I'd assume the scan is driven by a search of the disk, so files that are not there are not part of the "find the files" part of the scanning process.

That's where the magic happens or does not in your case ;) existingChildren is the data from filecache, newChildNames is the data found on disk.

I'm happy to apply patches that help narrow down the cause of this bug, to be sure!

kesselb commented 5 months ago

I'm not aware that there is an configuration flag like "keep_missing_files_in_filecache" but you never know ;)

Fun fact, I just learned that such a configuration flag does exist 😲 It's called filesystem_cache_readonly and I'm not sure if I understand why it's there, but that doesn't matter ;)

Could you run the commands below to check that the folder is really not there?

ls -al /var/lib/nextcloud/data/brian/files/DesktopHome/

ls -al /var/lib/nextcloud/data/brian/files/DesktopHome/.wine.old/

brianjmurrell commented 5 months ago

It's called filesystem_cache_readonly and I'm not sure if I understand why it's there, but that doesn't matter ;)

But it seems to default to false and I don't have it set to true in my config.php.

Could you run the commands below to check that the folder is really not there?

# ls -al /var/lib/nextcloud/data/brian/files/DesktopHome/
total 8
drwxr-xr-x.  2 root   root   4096 Oct  9  2018 .
drwxr-xr-x. 12 apache apache 4096 Jan 24 14:38 ..
# ls -al /var/lib/nextcloud/data/brian/files/DesktopHome/.wine.old/
ls: cannot access '/var/lib/nextcloud/data/brian/files/DesktopHome/.wine.old/': No such file or directory
joshtrichards commented 5 months ago

So is DesktopHome/ still a symbolic link within your files/ folder?

brianjmurrell commented 5 months ago

No. It's an empty directory.

kesselb commented 5 months ago

drwxr-xr-x. 2 root root 4096 Oct 9 2018 .

Why is DesktopHome owned by root and the parent by apache?

brianjmurrell commented 5 months ago

It was done so long ago that I could not possibly say at this point. But it's empty, regardless.

Could/should the ownership on the directory really prevent orphan cleanup of all of the database entries for the missing files below that empty directory?

I've fixed the ownership and am running files:scan again on the off chance.

kesselb commented 5 months ago 
select *
from oc_filecache
where fileid in (3075514, 3075522, 3349891, 3349899, 3726018, 3726026, 4125450, 4125458);

Could you share the result for the above sql query, please. The ids are from the repair-tree output.

Did you run the repair-tree command without dry-run already, and if yes, is the output empty if you run repair-tree a second time?

brianjmurrell commented 5 months ago

Could you share the result for the above sql query, please.

+---------+---------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------+---------+--------------------------------+----------+----------+------+------------+---------------+-----------+------------------+---------------+-------------+----------+
| fileid  | storage | path                                                                                                                                                         

                                       | path_hash                        | parent  | name                           | mimetype | mimepart | size | mtime      | storage_mtime | encrypted | unencrypted_size | etag          | permissions | checksum |
+---------+---------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------+---------+--------------------------------+----------+----------+------+------------+---------------+-----------+------------------+---------------+-------------+----------+
| 3075514 |       1 | files/DesktopHome/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/dosdevices/h:/.wine.old/dosdevices/z:/net/pc/home/mythtv/.dvdcss/`-524353908cff8e79-0000000000   | 6d67672fb0c90c08a57b29cc70d61889 | 3075114 | -524353908cff8e79-0000000000   |        2 |        1 |    0 | 1432796506 |    1432796506 |   
     0 |                0 | 55f6dbdc38fbe |          17 | NULL     |
| 3075522 |       1 | files/DesktopHome/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/dosdevices/h:/.wine.old/dosdevices/z:/net/pc/home/mythtv/.dvdcss/`25-bb985a43437746f8-0000000000 | ab3be515f7af89f047498b02df37edbe | 3075114 | 25-bb985a43437746f8-0000000000 |        2 |        1 |    0 | 1430647301 |    1430647301 |   
     0 |                0 | 55f6dbdc4fed1 |          17 | NULL     |
| 3349891 |       1 | files/DesktopHome/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/dosdevices/h:/.wine/dosdevices/z:/net/pc/home/mythtv/.dvdcss/`-524353908cff8e79-0000000000       | 2901f704362a712325b6b58350dc279c | 3349491 | -524353908cff8e79-0000000000   |        2 |        1 |    0 | 1432796506 |    1432796506 |   
     0 |                0 | 55f6ed4040f4f |          17 | NULL     |
| 3349899 |       1 | files/DesktopHome/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/dosdevices/h:/.wine/dosdevices/z:/net/pc/home/mythtv/.dvdcss/`25-bb985a43437746f8-0000000000     | 91448ac593ad3d97e42e00bf873314d1 | 3349491 | 25-bb985a43437746f8-0000000000 |        2 |        1 |    0 | 1430647301 |    1430647301 |   
     0 |                0 | 55f6ed40572a0 |          17 | NULL     |
| 3726018 |       1 | files/DesktopHome/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/dosdevices/e:/.wine.old/dosdevices/z:/net/pc/home/mythtv/.dvdcss/`-524353908cff8e79-0000000000   | 435d079ed08181558d3edc28527f4d7a | 3725618 | -524353908cff8e79-0000000000   |        2 |        1 |    0 | 1432796506 |    1432796506 |   
     0 |                0 | 55f704da42144 |          17 | NULL     |
| 3726026 |       1 | files/DesktopHome/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/dosdevices/e:/.wine.old/dosdevices/z:/net/pc/home/mythtv/.dvdcss/`25-bb985a43437746f8-0000000000 | 95ec3c1753bd20a877e59d4522a4105d | 3725618 | 25-bb985a43437746f8-0000000000 |        2 |        1 |    0 | 1430647301 |    1430647301 |   
     0 |                0 | 55f704da5bf3f |          17 | NULL     |
| 4125450 |       1 | files/DesktopHome/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/dosdevices/e:/.wine/dosdevices/z:/net/pc/home/mythtv/.dvdcss/`-524353908cff8e79-0000000000       | 4a23c5673f5e1358a438b07616b842fc | 4125050 | -524353908cff8e79-0000000000   |        2 |        1 |    0 | 1432796506 |    1432796506 |   
     0 |                0 | 55f72103b7352 |          17 | NULL     |
| 4125458 |       1 | files/DesktopHome/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/fake_windows/users/brian/My Documents/.wine.old/dosdevices/e:/.wine/dosdevices/z:/net/pc/home/mythtv/.dvdcss/`25-bb985a43437746f8-0000000000     | 01bc5ca41f1131b3f143feedfd464e04 | 4125050 | 25-bb985a43437746f8-0000000000 |        2 |        1 |    0 | 1430647301 |    1430647301 |   
     0 |                0 | 55f72103cfa7a |          17 | NULL     |
+---------+---------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------+---------+--------------------------------+----------+----------+------+------------+---------------+-----------+------------------+---------------+-------------+----------+

Did you run the repair-tree command without dry-run already

I did not (yet).

kesselb commented 5 months ago

Thank you.

`-524353908cff8e79-0000000000

Another mystery, the path column contains the filename with a leading ` and the name column does not. I cannot replicate that locally. Either by creating a file structure like on your end and running files:scan or uploading a file with ` via web client. Are the path and name in oc_filecache are correct if you upload a new file with a leading `?

brianjmurrell commented 5 months ago

It could quite probably just be due to an old bug. This data has been around for a number of years. Given that they are entries for files not actually on the disk and should be pruned, I'm not terribly worried about them, or trying to replicate their oddness.

kesselb commented 4 months ago

Hi,

I learned about https://github.com/icewind1991/root_cache_cleaner today.

The app is looking for records in the oc_filecache with a leading $uid for the path column. We've seen that on older installation.

If you try the app, please make sure that have a recent and working database backup.

fauust commented 1 month ago

Hi, I came across this today while migrating local storage to s3. Apparently, my very old NC (which was OC at the beginning) has a ton of entry in the oc_filecache table. This is a problem since the tool I am trying to use to move to s3 complains about not finding all those files.

Finally, I found this solution and I am wondering if it could not be a temporary workaround until the cleanup command is fully working?

adsche commented 1 month ago

I'm having millions of nonsensical filecache entries because of #41563.

I removed the external storage that caused them (does not appear anymore in occ files_external:list --all). However, even then occ files:cleanup removed 0 orphans but they are still in the DB.

The removed storage is even still listed in oc_storages (with available = 1 no less!) and doing SELECT COUNT(*) FROM oc_filecache WHERE storage = 144; results in over 82 million entries.

As you see from the ID 144, there are a lot of entries still in that table, even though files_external:list --all only shows 2 (with IDs 1 and 4).

This all doesn't make much sense to me. There currently does not seem to be a way to remove external storages cleanly from either the admin interface nor occ, correct? I will have to go and manually modify my database as linked by the previous comment?

EDIT to add: The (later removed) external storage used the $user parameter as part of the path. That is what causes the 100s of entries in the oc_storages table. Maybe this is part of the problem, maybe it is only external storages with $user that do not get removed correctly?

kesselb commented 1 month ago

Disclaimer: Have a backup

The external storage configuration is within oc_external_applicacle, oc_external_config, oc_external_mounts and oc_external_options. Then connected to oc_filecache via oc_mounts and oc_storages.

If the configuration in oc_external is gone, but the storage or mount is still there, then it sounds okay to me to delete the row(s) in oc_storage, oc_mounts and run the cleanup command. As mentioned before, the cleanup command looks for records in filecache without a matching storage.

If you delete an external storage via web client or occ that should be done by: https://github.com/nextcloud/server/blob/4f88123d2ba649fb6fe2df3d069a297c101dd019/lib/private/Files/Cache/Storage.php#L200-L233

That might fail for 82 million records.

adsche commented 1 month ago

Thanks, @kesselb ! It is indeed completely gone from all 4 oc_external_ tables (and even oc_mounts), so I will attempt what you are suggesting after the next nightly backup.

adsche commented 1 month ago

Well, this turned out to be a bigger operation still... But it worked! Database size shrunk from 40G to 400M. Many thanks, @kesselb !

For future reference:

Any DELETE FROM oc_filecache WHERE storage IN (...); would run into ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction even with LIMITs as low as 1000. So yes, it is likely that the cleanup when removing the storage just failed.

Instead, I cloned the oc_filecache table and copied valid entries only:

CREATE TABLE oc_filecache_new LIKE oc_filecache;
INSERT INTO oc_filecache_new SELECT * FROM oc_filecache WHERE oc_filecache.storage NOT IN (...);

This has the added benefit that purging like here shouldn't be necessary.

Ideally, of course, maintenance mode would be on during all that. However, I left maintenance mode off because it would have been too long of a downtime. Instead, I am hoping that I got away cleanly with this:

*) this wouldn't run because of a lock that I couldn't get released in any way, so I had to restart mysql at this point

To be safe, I guess a occ files:scan --all would be an appropriate next step.

To get this issue back on track: Would it be desirable to have a step in e.g. occ maintenance:repair --include-expensive to check for these kinds of stale entries? They are seemingly not caught by files:scan --all (unless that also was a problem of just how many entries we had).