search

nextcloud / previewgenerator

Nextcloud app to do preview generation in the background.
https://apps.nextcloud.com/apps/previewgenerator
GNU Affero General Public License v3.0
455 stars 57 forks source link

Multithreading Support #434

Open cmdrkarl opened 10 months ago

cmdrkarl commented 10 months ago

Hello,

can multithreading be added to use modern CPUs?

Thanks, Kurt

MrRinkana commented 8 months ago

This would be really nice, first time generating previews on an instance with a decent amount of media is pain, it took on my instance the about 24h on a i7 3770k..

All that is needed is to start multiple preview generations simultaneously set by a command line option such that some cores can be left for serving nextcloud.. I'm not sure how it works when executed trough php but ffmpeg supports multithreading per default, so one could keep a single queue but generate each image a lot faster instead..

This would also help when a lot of images are being uploaded at the same time, as currently its possible on my instance to upload a lot faster than the preview generation keeps up..

MrRinkana commented 7 months ago

I suspect this will be left as h2non/imaginary is being recommended instead of this app - and in that setup multithreading will be used!

iamdoubz commented 5 months ago

What would happen if you ran this command:

bash -c 'for i in {1..4}; do (occ preview:pre-generate -vvv &); done'

Is this a hack? Or, would it destroy my system? From what I gather, it would spawn 4 different instances of pre-generate at the same time... is this okay/allowed?

Update

So I just went ahead and tested with one folder. It spawned 4 instances and they all scanned the same file... not what we want... But, my library is organized by Year and Month, so I changed it to:

sudo htop
bash -c 'for i in {1..4}; do (occ preview:pre-generate -vvv --path="/username/files/Photos/2024/0$i" &); done'

Because of how the second command is run, you cannot input the sudo password, so you have to sudo something first before running the command... I choose sudo htop. What the second command did is that it spawned four processes looking in my Photos directory, year 2024, then the $i corresponds to what month I want... Not "multithreading" but does make scanning four times faster in this example...