background job run via cron claims pdlib isn't present

qudiqudi commented 1 year ago

Hello and a happy new year to everyone! I've come across to an interesting issue concerning the background job run via cron.

Expected behaviour

Using cron to call the background_job periodically with: */60 * * * * php -f /var/www/html/occ face:background_job -t 1800

and the job should run every hour.

Actual behaviour

The admin face recognition GUI tells me there are photos to be scanned and it should finish in a few minutes - BUT it never actually runs... I than piped this cronjob into a .log-file by appending 2>&1 >> cron.log to the job. Turns out the job will never run in the background as the log gives following message:

04/01/2023 09:00:00 1/8 - Executing task CheckRequirementsTask (Check all requirements)
04/01/2023 09:00:00 The PDlib PHP extension is not loaded. Cannot continue without it.Please read the documentation again about how to install the application: https://github.com/matiasdelellis/facerecognition/wiki/Installation
04/01/2023 09:00:00 Task CheckRequirementsTask signalled we should not continue, bailing out

Strangely, this doesn't appear in my admin logs.

Ok, so there is a problem with pdlib one might think and I should fix this. NO! Running face:background_job manually finishes perfectly every time. Strange.❔ ❔ ❔ _See the logs section for -vvv backgroundjob log

Steps to reproduce

See above.

Server configuration

Operating system: Docker installation of NC 25 with custom Docker Image - pdlib backed into it as instructed in the readme.

Pdlib version:

www-data@9f9a30d9342d:~/html$ php -i | grep pdlib
/usr/local/etc/php/conf.d/pdlib.ini,
pdlib
pdlib support => enabled
pdlib extension version => 1.0.2

How is DLib installed: Make sure it is working correctly with this tool It's working.
How is PDlib installed: Make sure it is working correctly with this tool Also working.
PHP version: 8.1.13
Web server: apache2 latest
Database: psql 14
Nextcloud version: 25.0.2

Logs

Background task log with debug.

sudo -u apache php occ -vvv face:background_job

``` www-data@9f9a30d9342d:~/html$ php occ -vvv face:background_job 1/8 - Executing task CheckRequirementsTask (Check all requirements) System: Linux System memory: 67182137344 PHP Memory Limit: 4294967296 2/8 - Executing task CheckCronTask (Check that service is started from either cron or from command) 3/8 - Executing task DisabledUserRemovalTask (Purge all the information of a user when disable the analysis.) yielding yielding yielding yielding yielding yielding 4/8 - Executing task StaleImagesRemovalTask (Crawl for stale images (either missing in filesystem or under .nomedia) and remove them from DB) Skipping stale images removal for user REDACTED as there is no need for it Skipping stale images removal for user REDACTED as there is no need for it Skipping stale images removal for user face_rec_user as there is no need for it Skipping stale images removal for user admin as there is no need for it Skipping stale images removal for user REDACTED as there is no need for it Skipping stale images removal for user REDACTED as there is no need for it 5/8 - Executing task CreateClustersTask (Create new persons or update existing persons) Skipping cluster creation, not enough data (yet) collected. For cluster creation, you need either one of the following: * have 1000 faces already processed * or you need to have 95% of you images processed Use stats command to track progress yielding Skipping cluster creation, not enough data (yet) collected. For cluster creation, you need either one of the following: * have 1000 faces already processed * or you need to have 95% of you images processed Use stats command to track progress yielding Found 19 faces without associated persons for user face_rec_user and model 1 Oldest face without persons for user face_rec_user and model 1 is from 2023-01-04 08:35:35 Clusters already exist, estimated there is no need to recreate them yielding Skipping cluster creation, not enough data (yet) collected. For cluster creation, you need either one of the following: * have 1000 faces already processed * or you need to have 95% of you images processed Use stats command to track progress yielding Skipping cluster creation, not enough data (yet) collected. For cluster creation, you need either one of the following: * have 1000 faces already processed * or you need to have 95% of you images processed Use stats command to track progress yielding Skipping cluster creation, not enough data (yet) collected. For cluster creation, you need either one of the following: * have 1000 faces already processed * or you need to have 95% of you images processed Use stats command to track progress yielding 6/8 - Executing task AddMissingImagesTask (Crawl for missing images for each user and insert them in DB) Skipping image scan for user REDACTED that has disabled the analysis Skipping image scan for user REDACTED that has disabled the analysis Skipping full image scan for user face_rec_user Skipping image scan for user admin that has disabled the analysis Skipping image scan for user REDACTED that has disabled the analysis Skipping image scan for user REDACTED that has disabled the analysis 7/8 - Executing task EnumerateImagesMissingFacesTask (Find all images which don't have faces generated for them) yielding 8/8 - Executing task ImageProcessingTask (Process all images to extract faces) NOTE: Starting face recognition. If you experience random crashes after this point, please look FAQ at https://github.com/matiasdelellis/facerecognition/wiki/FAQ ```

Also just for the prove that face recognition works on my system:

+----------------+--------+-------+----------+---------+
| User                    | Images | Faces | Clusters | Persons |
+----------------+--------+-------+----------+---------+
| admin                  | 0      | 0     | 0        | 0       |
| REDACTED          | 0      | 0     | 0        | 0       |
| REDACTED          | 0      | 0     | 0        | 0       |
| REDACTED          | 0      | 0     | 0        | 0       |
| REDACTED          | 0      | 0     | 0        | 0       |
| face_rec_user     | 15013  | 12771 | 6638     | 62      |
+----------------+--------+-------+----------+---------+

Web server error log

``` Insert your webserver log here ```

Nextcloud log (data/nextcloud.log)

Nextcloud log

``` Insert your Nextcloud log here ```

Browser log

``` Insert your browser log here, this could for example include: a) The javascript console log b) The network log c) ... ```

rhatguy commented 1 year ago

How are you call cron inside the Nextcloud docker container? I don't believe cron is normally available inside the Nextcloud container. I suspect there is something environmental causing this as I'm using the same Nextcloud container inside unraid to do exactly what you are. Below is the command I use to execute the background job, but critically, this is run from cron in unraid (my host os) and execs into the docker container to run the background job.

docker exec Nextcloud php -f /var/www/html/occ face:background_job -u pictures -t 840

qudiqudi commented 1 year ago

Cron is handled by it's own container via docker compose - and except for this background job is running fine. The nc own cron.php is running as well as the preview generator plugin. Also, cron is called, it's just that face recognition doesn't seem to see pdlib when called through the cron container and only works when called through the nextcloud app container manually with occ, which is weird.

rhatguy commented 1 year ago

Can you have cron execute "php -i | grep pdlib" as you tested manually above to see whether pdlib shows up inside the environment that cron commands are being executed in (the same environment that the background scan is running in)? Maybe cron is using a different php environment than your shell is when you test manually?

Its obviously something about the environment its being called from since it works for other people inside the Nextcloud docker container and even manually in your own environment.

qudiqudi commented 1 year ago

I'm unsure what you mean with "environment"? You mean from inside the nextcloud-container? Or from the docker host? Or from the cron container?

rhatguy commented 1 year ago

Basically create a new cron job that executes "php -i | grep pdlib" exactly the same way you have the cron job today that runs the background scanning job. Maybe log the output of the test cron job to a file somewhere so you can see it later. It looks like previously you tested by running the php command from a shell. I'm guessing there is some difference in the environment that the background job is running from than the environment that your shell is running the test from.

qudiqudi commented 1 year ago

Alright got ya. Wasn't sure as to what you refer to as env... It seems that running php -i | grep pdlib through cron like php -i | grep pdlib >> /var/spool/cron/crontabs/cron.log 2>&1 and pipe it into a file does not return anything. double checked that it is run by docker logs cron container and, yes it there:

crond: file www-data:
crond:  line php -i | grep pdlib >> /var/spool/cron/crontabs/cron.log 2>&1
crond:  job: 0 php -i | grep pdlib >> /var/spool/cron/crontabs/cron.log 2>&1
crond: child running /bin/bash
crond: USER www-data pid  48 cmd php -i | grep pdlib >> /var/spool/cron/crontabs/cron.log 2>&1

That is a bad sign, as it means pdlib is not present when cron executes php -i

rhatguy commented 1 year ago

As expected, the environment is different. Have you followed all of the instructions at the following link and the augment them with the instructions in the second "issue" linked below?

https://github.com/matiasdelellis/facerecognition/wiki/Docker https://github.com/matiasdelellis/facerecognition/issues/624

qudiqudi commented 1 year ago

You linked to this issue? Well, the instructions have nothing to do with cron. Building the image with dlib via Dockerfile is not related to my issue, remember: pdlib is present when running the background_job manually.

qudiqudi commented 1 year ago

I may have found the cause for this: in docker compose I use a different image for the cron container than for the app container, the latter is the one I've build with pdlib. So no wonder why cron can't find pdlib.

qudiqudi commented 1 year ago

Yep that's it. https://help.nextcloud.com/t/clarification-regarding-cron-jobs-setup-config/134450/6 Containers cron and app need to have the same set of resources. When you use docker compose and build the nextcloud image with Dockerfile as per the instructions, you need to use the same image for cron and for the app. Like this:

services:
  app:
    image: nextcloud-with-face-reco
  ...
  cron:
  image: nextcloud-with-face-reco

@matiasdelellis maybe add that to the readme, as this concerns everyone who's using docker compose and builds the nc image with Dockerfile.

matiasdelellis / facerecognition