Empty directories building up in the `preview` folder.

petruchito commented 4 months ago

https://github.com/nextcloud/server/blob/1164c762eb2c120ca0d2d8da83dfe9f925ed652f/lib/private/Preview/BackgroundCleanupJob.php#L36

This should delete empty preview directories as well. After running a few weeks with automatic ZFS snapshots folder attached as a history (It was a bad idea), I've got a monstrous directory structure and the cron scan constantly utilized my disk for several days(SELECT-UPDATE on the oc_filecache table).

files:cleanup didn't do anything, 0 orphaned files.

files:scan-app-data took almost 6 hours to complete:

+---------+--------+--------------+
| Folders | Files  | Elapsed time |
+---------+--------+--------------+
| 5868385 | 151476 | 05:45:49     |
+---------+--------+--------------+

So I'm now deleting empty folders manually with find . -type d -empty -delete in the preview directory.

petruchito commented 4 months ago

occ files:scan-app-data after deleting empty directories:

+---------+--------+--------------+
| Folders | Files  | Elapsed time |
+---------+--------+--------------+
| 368149  | 151478 | 02:13:46     |
+---------+--------+--------------+

szaimen commented 3 months ago

Hi, which nc version?

petruchito commented 3 months ago

Hi, which nc version?

Nextcloud AIO v9.1.0 (NEXTCLOUD_VERSION | 29.0.3)

petruchito commented 3 months ago

Snapshots were ~400Gb each, 24 hourly, 7 daily, 4 weekly and 12 monthly, so every hour the server was introduced with 400Gb of data and 400Gb disappeared. IO utilization issues persisted after I unmounted the history.

Folders structure under the `history/` was like this:

``` initial zfs-auto-snap_daily-2024-06-07-00h07 zfs-auto-snap_daily-2024-06-08-00h07 zfs-auto-snap_daily-2024-07-04-00h07 zfs-auto-snap_daily-2024-07-05-00h07 zfs-auto-snap_daily-2024-07-08-00h07 zfs-auto-snap_daily-2024-07-11-00h07 zfs-auto-snap_daily-2024-07-12-00h07 zfs-auto-snap_hourly-2024-05-18-16h00 zfs-auto-snap_hourly-2024-05-18-17h00 zfs-auto-snap_hourly-2024-05-18-18h00 zfs-auto-snap_hourly-2024-05-18-19h00 zfs-auto-snap_hourly-2024-05-18-20h00 zfs-auto-snap_hourly-2024-05-18-21h00 zfs-auto-snap_hourly-2024-05-18-22h00 zfs-auto-snap_hourly-2024-05-18-23h00 zfs-auto-snap_hourly-2024-06-04-01h00 zfs-auto-snap_hourly-2024-06-04-04h00 zfs-auto-snap_hourly-2024-06-04-10h00 zfs-auto-snap_hourly-2024-06-04-12h00 zfs-auto-snap_hourly-2024-06-04-14h00 zfs-auto-snap_hourly-2024-06-05-14h00 zfs-auto-snap_hourly-2024-06-05-21h00 zfs-auto-snap_hourly-2024-06-05-23h00 zfs-auto-snap_hourly-2024-06-06-04h00 zfs-auto-snap_hourly-2024-06-07-04h00 zfs-auto-snap_hourly-2024-06-07-11h00 zfs-auto-snap_hourly-2024-07-04-19h00 zfs-auto-snap_hourly-2024-07-07-14h00 zfs-auto-snap_hourly-2024-07-07-19h00 zfs-auto-snap_hourly-2024-07-07-21h00 zfs-auto-snap_hourly-2024-07-12-00h00 zfs-auto-snap_monthly-2024-04-01-00h28 zfs-auto-snap_monthly-2024-05-01-00h28 zfs-auto-snap_monthly-2024-06-01-00h28 zfs-auto-snap_monthly-2024-07-01-00h28 zfs-auto-snap_weekly-2024-05-12-00h14 zfs-auto-snap_weekly-2024-05-19-00h14 zfs-auto-snap_weekly-2024-06-09-00h14 zfs-auto-snap_weekly-2024-07-07-00h14 # and /Documents/... /Pictures/... # under each directory of above ```

And this might also relate to the problem - I have 'filesystem_check_changes' => 1 in the config.php.

joshtrichards commented 3 months ago

I don't quite understand your bug report and you seem to have entirely bypassed our reporting template.

I understand your suggestion about the background job, but what was the originally problem that arose?

There are (or were) some bugs like #45654. Is this merely that?

petruchito commented 3 months ago

what was the originally problem that arose?

Constant disk utilization by nextcloud's postgres led me to finding 5M folders in the preview folder. The cleanup job seems not to remove all empty folders and they were building up there.

(#45654 does not seem to relate.)

petruchito commented 3 months ago

I suppose this deletes preview/b/5/7/7/2/e/7/39548/ but leaves preview/b/5/7/7/2/e/7/ orphaned even if it's an empty directory.

Thus you theoretically can end up with 7^16=3,32×10^13 empty directories.

oTTo-oTTo commented 2 weeks ago

I plead for 16⁷ directories 2.7E08 and yes, the empty ones should be deleted.

NC29.0.4: if have 1300 empty preview dirs in 3 months.

nextcloud / server

Empty directories building up in the `preview` folder. #46425