search

nextcloud / previewgenerator

Nextcloud app to do preview generation in the background.
https://apps.nextcloud.com/apps/previewgenerator
GNU Affero General Public License v3.0
460 stars 57 forks source link

How to config? How to reclaim space? #94

Open tavinus opened 6 years ago

tavinus commented 6 years ago

Hi!

Say I apply these settings

sudo -u www-data php /var/www/nextcloud/occ config:app:set --value="64 128" previewgenerator squareSizes
sudo -u www-data php /var/www/nextcloud/occ config:app:set --value="64 128" previewgenerator widthSizes
sudo -u www-data php /var/www/nextcloud/occ config:app:set --value="64 128" previewgenerator heightSizes

And my config.php has

  'preview_max_x' => 2048,
  'preview_max_y' => 2048,
  'preview_max_scale_factor' => 2,

OR

  'preview_max_x' => null,
  'preview_max_y' => null,
  'preview_max_scale_factor' => 2,

And then I run generate-all

Would the generator set the '_max' file to 128 pixels only and then just use that size as maximum even when I click to open an image? Or is just for pre-generating and nothing else and then the bigger image will be generated when clicked? I am but confused because I got some mixed results here, but I am guessing the browser cache was doing its thing.

What would be the recommended settings for a big install with lots of images? We want to be able to browse it fast and have the best full-screen experience, but we would like to avoid having 30+Gb (still processing) of preview images data (which also goes to the backup, btw).

And then there is also the question if it is safe to remove the preview files and/or folders to reclaim that space, which I already asked on core. https://github.com/nextcloud/server/issues/5263#issuecomment-355202378

I would probably prefer to make some simple bash scripting to remove just some bigger sized PNGs so I don't have to reprocess everything. That is easy if there is no problem with the database and other stuff.

In any case, because I was testing different configs as well, I can see many different preview images above 1024px for some files, which is probably the reason for such a big footprint.

I would prefer to have just the smaller sizes pre-generated (for menus and icons) and let the big stuff be done as needed. Can this be done with the config above?

Edit

Also, what does pre-generate really do? It runs in 1s and does not seem to create any files here. generate-all scans everything (for days) and sucks the processor and creates the pngs, but pre-generate does not seem to do anything.

tavinus commented 6 years ago

First of all, the most annoying thing that was happening here was that I had max_x/y set to 512 at some point and because of that many of my previews were still opening in 512 pixels even though I had it set to 2048 and null for a long time at my config. I really wanted to reset everything so I could have the bigger previews for those.

The other problem was that my data folder doubled in size, as mentioned (even though I was browsing in 512px).

To "reset" the preview cache

  1. Stop Webserver (apache / nginx)
  2. Moved the appdata_instaceName/preview folder outside the appdata folder. You can just remove it if you are confident.
  3. Run sudo -u www-data /usr/bin/php /var/www/nextcloud/occ files:scan-app-data to fix database
  4. Start Webserver and Test everything works
  5. Finish up (delete moved folder, etc).

A command to do this would still be nice and needed. Also something that checks for orphaned previews (from deleted files / removed external storage). Deletion seems to trigger preview deletion in recent versions but external storage removal does not (I tried).

I am still not sure on the optimal way to use the generator though. I was checking the image sizes generated when browsing and previewing some of my folders and it seems like the 32x32 and 256x256 are the most used for a desktop interface. So I tried using that for squared thumbs and null for the other two but I still got some extra thumbnail sizes generated. Maybe I will try to hack it to those 2 sizes only later.

At this point I gave up on pre-generating. I just browsed the main folders from gallery a little bit so it would pre-generate that and have a fast base tree.

I wonder if we could pre-generate exactly the same sizes needed for the browser on list / thumbs views. Maybe even use the same functions from there? Not sure...

Cheers! Gus

nachoparker commented 5 years ago

You might be interested in this

https://ownyourbits.com/2019/06/29/understanding-and-improving-nextcloud-previews/

ghost commented 5 years ago

Is there any work done to keep preview files from entering oc_filecache table in the SQL db? Ideally, if they indeed are needed, perhaps a second table or second db for previews would be preferred. It really slows sql performance down a lot.

ghost commented 5 years ago

I think this is an related issue: https://github.com/rullzer/previewgenerator/issues/162#issuecomment-507205408

dfischer02 commented 1 year ago

First of all, the most annoying thing that was happening here was that I had max_x/y set to 512 at some point and because of that many of my previews were still opening in 512 pixels even though I had it set to 2048 and null for a long time at my config. I really wanted to reset everything so I could have the bigger previews for those.

The other problem was that my data folder doubled in size, as mentioned (even though I was browsing in 512px).

To "reset" the preview cache

  1. Stop Webserver (apache / nginx)
  2. Moved the appdata_instaceName/preview folder outside the appdata folder. You can just remove it if you are confident.
  3. Run sudo -u www-data /usr/bin/php /var/www/nextcloud/occ files:scan-app-data to fix database
  4. Start Webserver and Test everything works
  5. Finish up (delete moved folder, etc).

A command to do this would still be nice and needed. Also something that checks for orphaned previews (from deleted files / removed external storage). Deletion seems to trigger preview deletion in recent versions but external storage removal does not (I tried).

I am still not sure on the optimal way to use the generator though. I was checking the image sizes generated when browsing and previewing some of my folders and it seems like the 32x32 and 256x256 are the most used for a desktop interface. So I tried using that for squared thumbs and null for the other two but I still got some extra thumbnail sizes generated. Maybe I will try to hack it to those 2 sizes only later.

At this point I gave up on pre-generating. I just browsed the main folders from gallery a little bit so it would pre-generate that and have a fast base tree.

I wonder if we could pre-generate exactly the same sizes needed for the browser on list / thumbs views. Maybe even use the same functions from there? Not sure...

Cheers! Gus

Do not delete the preview folder, only the contents of the preview folder. Otherwise:

  1. cd into the app data folder
  2. mkdir preview
  3. sudo chown www-data:www-data preview/
  4. Continue as described with the files scan-app-data
Jhyrachy commented 3 months ago

Hello, I'm using Hetzner as nextcloud provider, so I do not have direct access to the shell. Is there any other way to recreate the preview?