Preview generator does not generate any preview after mass data migrated images

[ovals12]

It seems there is a "bug" in Preview Generator app which prevents generation of previews after you performed a mass data migration (copy of images to data folder, perform occ files:scan --all).

I want to import very many image files into a new installation of Next Cloud and I would like to have the previews pre-generated. I do not want to upload several gigabytes via pc client or webdav (after that preview generation works, at least I tested webdav upload and previews for the uploaded files were generated). I think this is why there is this mass data migration available.

After mass data migration, the files are in NextCloud, you could access them via webdav or gui - all seems fine. But Preview Generator "does not see" them. It will not even generate previews for that 3 images which comes along default installation.

Below is the procedure I used for reproducing the issue. I used NextCloud as a snap installation.

1. Fresh install of nextcloud, no previews exists so far:

   # ls -al /var/snap/nextcloud/common/nextcloud/data/appdata_ocm286yevx6a/preview
   total 8
   drwxr-xr-x 2 root root 4096 Aug 22 14:23 .
   drwxr-xr-x 7 root root 4096 Aug 22 14:24 ..

2. Install Preview Generator app via GUI, no previews exists so far:

   # ls -al /var/snap/nextcloud/common/nextcloud/data/appdata_ocm286yevx6a/preview
   total 8
   drwxr-xr-x 2 root root 4096 Aug 22 14:23 .
   drwxr-xr-x 8 root root 4096 Aug 22 14:26 ..

3. nextcloud.occ preview:pre-generate, no previews exists so far:

   # ls -al /var/snap/nextcloud/common/nextcloud/data/appdata_ocm286yevx6a/preview
   total 8
   drwxr-xr-x 2 root root 4096 Aug 22 14:23 .
   drwxr-xr-x 8 root root 4096 Aug 22 14:26 ..

4. copy some jpg to data directory and perform files scan so the files are visible in nextcloud (2017-* were copied there, the rest 3 are coming with default installation)

   # ls -al /var/snap/nextcloud/common/nextcloud/data/user/files/Photos/
   total 28296
   drwxr-xr-x 2 root root    4096 Aug 22 14:30 .
   drwxr-xr-x 4 root root    4096 Aug 22 14:10 ..
   -rw-r----- 1 root root 1576687 Aug 22 14:30 2017-01-01_123325.jpg
   -rw-r----- 1 root root 1623375 Aug 22 14:30 2017-01-01_123328.jpg
   -rw-r----- 1 root root 3681799 Aug 22 14:30 2017-01-01_123335.jpg
   -rw-r----- 1 root root 3218276 Aug 22 14:30 2017-01-01_123725.jpg
   -rw-r----- 1 root root 3628021 Aug 22 14:30 2017-01-01_123742.jpg
   -rw-r----- 1 root root 3895594 Aug 22 14:30 2017-01-01_123853.jpg
   -rw-r----- 1 root root 3029542 Aug 22 14:30 2017-01-01_175627.jpg
   -rw-r----- 1 root root 3027073 Aug 22 14:30 2017-01-01_175717.jpg
   -rw-r----- 1 root root 2900706 Aug 22 14:30 2017-01-01_175730.jpg
   -rw-r--r-- 1 root root  819766 Aug 22 14:10 Coast.jpg
   -rw-r--r-- 1 root root  585219 Aug 22 14:10 Hummingbird.jpg
   -rw-r--r-- 1 root root  955026 Aug 22 14:10 Nut.jpg

   
   # nextcloud.occ files:scan --all
   Starting scan for user 1 out of 1 (user)

+---------+-------+--------------+ | Folders | Files | Elapsed time | +---------+-------+--------------+ | 5 | 16 | 00:00:00 | +---------+-------+--------------+

5. now we have 16 files in Next cloud - if I would go to GUI now I would see all the images there, but I will not do it and perform preview generation:

nextcloud.occ preview:pre-generate

#

6. No previews were generated:

ls -al /var/snap/nextcloud/common/nextcloud/data/appdata_ocm286yevx6a/preview

total 8 drwxr-xr-x 2 root root 4096 Aug 22 14:23 . drwxr-xr-x 8 root root 4096 Aug 22 14:26 ..

7. Copied 2 files via webdav and tried initiate preview generation, all previews for that 2 images were generated:

nextcloud.occ preview:pre-generate

ls -al /var/snap/nextcloud/common/nextcloud/data/appdata_ocm286yevx6a/preview

total 16 drwxr-xr-x 4 root root 4096 Aug 22 14:36 . drwxr-xr-x 8 root root 4096 Aug 22 14:26 .. drwxr-xr-x 2 root root 4096 Aug 22 14:36 124 drwxr-xr-x 2 root root 4096 Aug 22 14:36 125

find . -type f

./125/128-96.jpg ./125/256-256-crop.jpg ./125/683-512.jpg ./125/32-32-crop.jpg ./125/1024-1024-crop.jpg ./125/2048-1536-max.jpg ./125/1536-1536-crop.jpg ./125/85-64.jpg ./125/43-32.jpg ./125/512-384.jpg ./125/64-48.jpg ./125/1365-1024.jpg ./125/32-24.jpg ./125/64-64-crop.jpg ./125/512-512-crop.jpg ./125/341-256.jpg ./125/1024-768.jpg ./125/256-192.jpg ./125/171-128.jpg ./125/128-128-crop.jpg ./124/128-96.jpg ./124/256-256-crop.jpg ./124/683-512.jpg ./124/32-32-crop.jpg ./124/1024-1024-crop.jpg ./124/2048-1536-max.jpg ./124/1536-1536-crop.jpg ./124/85-64.jpg ./124/43-32.jpg ./124/512-384.jpg ./124/64-48.jpg ./124/1365-1024.jpg ./124/32-24.jpg ./124/64-64-crop.jpg ./124/512-512-crop.jpg ./124/341-256.jpg ./124/1024-768.jpg ./124/256-192.jpg ./124/171-128.jpg ./124/128-128-crop.jpg

[ovals12]

Maybe it will help - I roughly compared dump of mysql when I uploaded a jpg file via GUI with a dump when I did an import by copying the file in NextCloud's filesystem and nextcloud.occ files:scan --all, and also with a dump when uploaded a jpg via webdav.

GUI upload -> enters a row into oc_activity table Webdav upload -> enters a row into oc_activity table mass data migration + nextcloud.occ files:scan --all --> does not enter any row in oc_activity table

[oscarcarlsson]

I think that pre-generate doesn't do what you think it does, or I've misunderstood what it does. From my understanding, it tries to scan the changes since it's last run and generates new previews for the files that have changed. It sounds like you want to run the 'preview:generate-all' method.

I came here with a similar issue (just moved my jpg backup-export to my nextcloud library), my NC got stuck on a deadlock (!) which wasn't easily visible until I ran the generate-all method manually. The culprit on the web client was that previews had stopped been generated for my library.

To fix my issues, I simply restarted my php-fpm process (using nginx + php-fpm) and ran the generate-all method and it's chugging away now.

[ovals12]

methods of preview generator: preview:delete_old Delete old preview folder (pre NC11) preview:generate-all Generate previews preview:pre-generate Pre generate previews

Neither preview:generate-all nor preview:pre-generate does the job. Simply - imagine you have 1000 pictures (backup from phone's camera) and you install nextcloud, do mass data migration to this new nextcloud instance (migrate those 1000 pictures) by method occ files:scan --all, then you want to pre-generate previews for these 1000 images, so you are able to browse them fast for instances via gallery app. It does not work. You will end up with no previews pre generated and very slow displaying of the images via owncloud desktop client or mobile app. I did not find any way to make nextcloud (and also owncloud) usable for browsing old pictures.

Third party web gallery apps work pretty fine and fast for the same amount of pictures (piwigo for instance will pre-generate all the needed image sizes and then browsing is very very fast - you have the thumbnails displayed almost instantly). I think a lot of people use nextcloud for backing up their phone's cameras and then they have no direct way to browse the pictures at ease. This is sad.

[oscarcarlsson]

For my purposes, which I think is fairly close to yours (I've imported 14 Gbytes of high-resolution JPGs from another backup folder) I had to run the preview:generate-all script for them to have the previews generated. The images are then very fast to load on my Android client, and before this they were generated on the fly which took some time and eventually would hang.

However, it would be better to have this done as expected using the preview:pre-generate script in the first place. I'm considering running preview:generate-all in a cron job since it seems to only generate what's needed, which I guess is really what preview:pre-generate should have done in the first place?

Lets see if the author could help us out here, any ideas @rullzer?

[ovals12]

@oscarcarlsson how did you import those 14gbytes og JPEgs? Did you place them in the data directory and ran files:scan --all as me or did you use some other way?

[Benouare]

Hi, Same here, I ve loaded something like 100 gigs of pictures in nextcloud via the scan:files, and after that i have run the pre-generate cmd. It does nothing. Could be really good if the pre:generate (or something else but with the same result) works without any data in the oc_activity.
Or there is a workaournd that i didnt saw? Thx Ben

[robertalers]

Same problem! 20 gigs of pictures; scan:file following by pre-generate only shows the first 10 directory's and afterwards just stops. no error messages.

I did try the command: php occ preview:generate-all -vvv --path="/USerName/files/DIR"

Specicying the exact directory i want to scan does work on some directory's: /username/files/pictures -> doesn't work /username/files/pictures/subfolder -> DOES work

Is there progress on this issue?

(Running NC 21 on Docker on an unraid server)

[floeschie]

I use a Nextcloud Workflow to downscale new images uploaded by users using ImageMagick. The use case is to save disk space when no high resolution images are needed. Additionally, a watermark is added to those new images after the downscaling process. At the end, I run occ files:scan --all to notify Nextcloud about the changes ImageMagick did directly to the files.

It turns out that the previews are not re-generated, as the previews don't contain the watermark, while the original file does. I figure it's the same issue other users face: occ files:scan --all does not add an entry to the oc_activity table, and occ preview:pre-generate does not check changed files on the filesystem level. https://github.com/nextcloud/previewgenerator/issues/218 seems to be related to this topic as well.

:thought_balloon: Ideas:

1. occ files:scan --all should add an entry to the oc_activity table when a change on the filesystem level is found
2. occ preview:pre-generate should check timestamps the filesystem level as well
3. Give admins a way to trigger re-generate previews of a specific file, e. g. occ preview:re-generate <fileid> or occ preview:re-generate /path/to/file-on-disk

Option 3 would be my preferred solution, combined with a feature request for option 1 to Nextcloud core.