occ preview:pre-generate command not working?

dokeraj commented 3 years ago

Hi I have the nextcloud 21.0.0.18,

When I use the command occ preview:pre-generate, there is no log and it takes just a few seconds. But I can see that the images were not processed. In contrast - when I do the occ preview:generate-all command, then, I get the log and it takes longer, but it works as expected.

It says that I should use pre-generate in regular time intervals, but I can see that the generate-all also is just searching for those images that have not been processed before. So is it ok if I just use generate-all in a cronjob, instead of the other command?

Am I missing something?

Thanks.

jacotec commented 3 years ago

'generate-all' puts a pretty high load on the disk I/O and runs a reasonable time as it rescans all folders. I'd not recommend running it periodically via the standard cron, maybe once in a night.

However, I also have the feeling that 'pre-generate' has an issue and is not doing what it should as I often see missing previews.

freekvh commented 3 years ago

[I'm on version Nextcloud 22.1.0]

I also have issues, My picture overview used to load pretty quickly, but it stopped being fast recently, and indeed I noticed when I do:

docker exec -u www-data nextcloud_app php occ preview:pre-generate

I get:

Command "preview:pre-generate" is not defined.

  Did you mean one of these?
      notification:generate
      preview:repair
      preview:reset-rendered-texts

When I do:

docker exec -u www-data nextcloud_app php occ preview:generate-all -vvv

I get:

In Application.php line 676:

  [Symfony\Component\Console\Exception\CommandNotFoundException]
  Command "preview:generate-all" is not defined.

  Did you mean one of these?
      notification:generate
      preview:repair
      preview:reset-rendered-texts

Exception trace:
  at /var/www/html/3rdparty/symfony/console/Application.php:676
 Symfony\Component\Console\Application->find() at /var/www/html/3rdparty/symfony/console/Application.php:237
 Symfony\Component\Console\Application->doRun() at /var/www/html/3rdparty/symfony/console/Application.php:149
 Symfony\Component\Console\Application->run() at /var/www/html/lib/private/Console/Application.php:209
 OC\Console\Application->run() at /var/www/html/console.php:99
 require_once() at /var/www/html/occ:11

So how does one generate preview these days?

johnnyasantoss commented 3 years ago

@freekvh check if preview generator is still enabled on your instance. It might have been disabled during an update.

freekvh commented 3 years ago

@freekvh check if preview generator is still enabled on your instance. It might have been disabled during an update.

Ah yes, I have to "enable untested app", my apologies, I didn't realize. I works now, preview:pre-generate is finished in seconds, preview:generate-all -vvv is now running and taking a long time. Do you know if preview:pre-generate is still the best thing to run?

I guess preview:generate-all regenerates everything, not advisable on a regular basis I'd say.

johnnyasantoss commented 3 years ago

@freekvh check if preview generator is still enabled on your instance. It might have been disabled during an update.

Ah yes, I have to "enable untested app", my apologies, I didn't realize. I works now, preview:pre-generate is finished in seconds, preview:generate-all -vvv is now running and taking a long time. Do you know if preview:pre-generate is still the best thing to run?

I guess preview:generate-all regenerates everything, not advisable on a regular basis I'd say.

@freekvh yes, following the readme recommendations, it is not suggested to run the generate-all command regularly, only the pre-generate command should be scheduled.

jacotec commented 3 years ago

@freekvh yes, following the readme recommendations, it is not suggested to run the generate-all command regularly, only the pre-generate command should be scheduled.

That's the plan, but running the preview:pre-generate via cron every 30 minutes and I'm browsing into a folder with plenty new pictures a day after they're synced into, I can watch the previews being generated one-by-one at this time. So pre-generate does not appear to work.

freekvh commented 3 years ago

@freekvh yes, following the readme recommendations, it is not suggested to run the generate-all command regularly, only the pre-generate command should be scheduled.

That's the plan, but running the preview:pre-generate via cron every 30 minutes and I'm browsing into a folder with plenty new pictures a day after they're synced into, I can watch the previews being generated one-by-one at this time. So pre-generate does not appear to work.

Yes, I believe I am experiencing the same thing.

asm301 commented 2 years ago

"pre-generate" does absolutely nothing at all with no error output However "generate-all" creates previews perfectly, but a lot of resources are used depending on the directory/file size

jacotec commented 2 years ago

Strangely the command works when I issue it from the command line:

sudo -u www-data /usr/bin/php -f /var/www/nextcloud/occ preview:pre-generate

This is my crontab for the www-data user:

root@nextcloud:~# crontab -u www-data -l
*/5  *  *  *  * /usr/bin/php -f /var/www/nextcloud/cron.php > /dev/null 2>&1
0 1 * * * /usr/bin/php -f /var/www/nextcloud/occ files:scan --all > /dev/null 2>&1
*/15 * * * * /usr/bin/php -f /var/www/nextcloud/occ files:scan --unscanned --all > /dev/null 2>&1
*/15 * * * * /usr/bin/php -f /var/www/nextcloud/occ preview:pre-generate
*/30 * * * * /usr/bin/php -f /var/www/nextcloud/occ fulltextsearch:index -q > /dev/null 2>&1
*/15 * * * * /usr/bin/php -f /var/www/nextcloud/occ face:background_job -t 870 > /dev/null 2>&1
0 3 * * * /usr/bin/php -f  /var/www/nextcloud/occ maps:scan-photos

All commands from the are running fine except "occ preview:pre-generate".

I don't understand why it runs from the command line but not from the cron job.

sepo-one commented 1 year ago

Same for me. ./occ preview:pre-generate -vvv looks to do nothing. How to check if it's working ?

maherran commented 1 year ago

Same case:

nextcloud 25.0.4
php 8.1
every test I've made until now is telling that oc_preview_generation is empty

I wonder if listener is doing its work...

st3iny commented 1 year ago

Same for me. ./occ preview:pre-generate -vvv looks to do nothing. How to check if it's working ?

Upload some image files to your server and run the command. It should generate previews for new and changed files. It will also log all processed files when -vvv is enabled.

maherran commented 1 year ago

Upload some image files to your server and run the command. It should generate previews for new and changed files. It will also log all processed files when -vvv is enabled.

This is the way I did the test:

exclude files:scan and preview:pre-generate from cron jobs
upload 2 image files to the server (via syncthing)
execute manually preview:pre-generate (previously I've added some lines to check: see below), but I have the message _previewgeneration empty. So, for me, it sounds like listener is not adding info to database (?)

                    $qb->select('*')
                            ->from('preview_generation')
                            ->orderBy('id')
                            ->setMaxResults(1000);
                    $cursor = $qb->execute();
                    $rows = $cursor->fetchAll();
                    $cursor->closeCursor();

                        if ($this->output->getVerbosity() > OutputInterface::VERBOSITY_VERBOSE) {
                            $this->output->writeln('preview_generation running.');
                        }
                    if ($rows === []) {
                        if ($this->output->getVerbosity() > OutputInterface::VERBOSITY_VERBOSE) {
                            $this->output->writeln('preview_generation empty.');
                        }
                            break;
                    }

of course, previews are not generated

maherran commented 1 year ago

After a lot of test without getting the system to work as designed (probably because I do something wrong), I tried to find a solution that apparently works, although it is probably no more than a temporary fix. Let me explain:

preview:pre-generate should work as a cron job, but no way to have this solution working.
I've done some minor changes on preview:generate-all, and now I have this included as a cron job instead of preview:pre-generate (every 10 min, for instance), and it seems it's working. The solution is putting a limit (I've tried 500 seconds for a cron of 600 seconds, but this could be tunned depending on your systems), so this is working every 10 minutes for a maximum of 500 seconds, in order not to block the server, at least when the system needs to generate new previews). Aparently it works and generate previews. You can test and revert.
the proposed changes are as follows (you need to use the command nano /var/www/nextcloud/apps/previewgenerator/lib/Command/Generate.php) and add the lines between // maher 20230308 init and // maher 20230308 end

    protected function execute(InputInterface $input, OutputInterface $output): int {
            // maher 20230308 init
            $this->sec_ini=time();
            // maher 20230308 end
            if ($this->encryptionManager->isEnabled()) {

                    foreach ($nodes as $node) {
                            // maher 20230308 init
                            $sec_cur=time();
                            $sec_delta=$sec_cur-$this->sec_ini;
                            if ($sec_delta > 500) {
                                return;
                            }
                            // maher 20230308 end
                            if ($node instanceof Folder) {

My advice if you want to try:

_sudo tree -a your nextcloud data folder/appdataxxxxx/preview |tail -1: note the number of preview files obtained.
change /var/www/nextcloud/apps/previewgenerator/lib/Command/Generate.php as described above.
add a cronjob as: /10 * php /var/www/nextcloud/occ preview:generate-all
upload some image files to your server
wait 15-20 minutes to obtain the first completion of the cronjob, and repeat step 1. Compare new number of preview files obtained.

I have to insist. This has to be just a temporary solution for those, like me, who can't get the system to work as designed (just in case this is working for you).

st3iny commented 1 year ago

upload 2 image files to the server (via syncthing)

Externally added files are not picked up by pre-generate. It only handles file changes from within Nextcloud (Client, Web, Android, etc.).

If you often do external changes, you should run files:scan regularly so that Nextcloud properly picks up the changes.

maherran commented 1 year ago

upload 2 image files to the server (via syncthing)

Externally added files are not picked up by pre-generate. It only handles file changes from within Nextcloud (Client, Web, Android, etc.).

If you often do external changes, you should run files:scan regularly so that Nextcloud properly picks up the changes.

This is my cron. Do you think it's enough? /5 php -f /var/www/nextcloud/cron.php > /dev/null 2>&1 0 10 /usr/bin/php -f /var/www/nextcloud/occ files:scan --all > /dev/null 2>&1 /19 /usr/bin/php -f /var/www/nextcloud/occ files:scan myuser --unscanned > /dev/null 2>&1 /37 /usr/bin/php -f /var/www/nextcloud/occ preview:pre-generate

maherran commented 1 year ago

New test:

File uploaded to the server (folder 1) manually from the web: oc_preview_generation filled (OK)
File uploaded to the server (folder 1) using nexctcloud client on my PC: oc_preview_generation filled (OK)
File uploaded to the server (folder 1) using nextcloud client on my android phone: oc_preview_generation filled (OK)
File uploaded to the server (folder 1) using syncthing: oc_preview_generation not filled (NOK)

My conclusion: everything works as expected when using nextcloud, but this is not the case when using syncthing, even sending the same file to the same folder in Nextcloud server. Why? I don't know, and I would like to know. files:scan is supposed to help to solve this, as @st3iny mentioned? If so, what should be the procedure?

mike8206 commented 1 year ago

I tried this command in docker's container and it starts generating previews sudo -u www-data php occ preview:generate-all -vvv

joshtrichards commented 4 months ago

The listener, which must get triggered for pre-generate to have anything to process, is only triggered for write events within Nextcloud Server. A scan isn't sufficient.

This just sounds like #218 (unless this is occurring for non-externally added files).

nextcloud / previewgenerator

occ preview:pre-generate command not working? #246