Split up scan job in smaller tasks

[EricInBj]

i have 70000 photos( and some videos） in a local folder with two or three level sub folders ( mount from a samba server)

i ran librephotos by docker in a vm with 8C16G, gunniWorks set to 8 in the .env file.

in the past three days, only 13000 photos were added, and i saw 9 gunicorn process in the docker container, but only 1 eat 100% cpu, others are idle.

because there was only one photo scan job ?

if so, i think the scan job may break into some sub jobs:

1. generate the full file list to be scan
2. one or some sub-job to do the basic works ( generate thumbnails, read meta info, save to the db)
3. one or some sub-job to do the advanced works ( face detection, object detection, etc.)

[mprivoro]

single worker on import? 13 photos in 7 min on x16 cores CPU??? is it possible to run it multithreaded ?

[kaushikray1]

I have a similar setup as mentioned by OP, i am running LIbrePhotos latest in docker on a Ubuntu VM, I have assigned 8C and 16GB ram to this VM, gunniWorks set to 8 in the .env file. All my data is on a NAS so i mounted my folder with all the pictures and videos in the /data folder with a SMB share. I have a uptime of 3 days now and only 996 photos have been scanned. I have multiple sub directories with at least 100k photos. I notice when i start a scan after a few files have been added the scan stops. If i cancel the scan and start a new scan then a few pictures will be added then it stops again. I have also noticed when the scan is running, it's pretty slow as well. I would have been happy if i even got 5k pics scanned a day but 996 pics in 3 days i am definitely doing something wrong. Please suggest.

Thank you

[alexislefebvre]

Is it possible to launch as many scanners as workers? It looks like starting scan only use one worker.

[derneuere]

Multithreading is not yet implemented see #17 I like the idea of splitting the tasks.

[bushibot]

same issue, multi core box, fast uraid build but 500 photos in HOURS of scan work. Going to take weeks to hand the 6500 from the flickr roll and possible years if I point it at the raw archive. Is it using the graphics card for hardware acceleration? Could it? What can speed this up?

[scepterus]

There is a gpu variant, you need to change things in the docker compose, but you must have an nvidia gpu.

[bushibot]

There is a gpu variant, you need to change things in the docker compose, but you must have an nvidia gpu.

Oh interesting, can you point me at specifics? I have a nvidia 370 for Plex transcoding in my unraid box. I’m shit with docker compose, but this is likely worth sorting out and Unraid makes basic edits pretty easy.

[scepterus]

https://docs.librephotos.com/docs/installation/environment-variables/ you mean a 3070? Because you need cuda cores for this kind of stuff.

[derneuere]

Done!