Allow multiprocessing for generate-all

[vwbusguy]

Currently, generate-all serializes all pictures for all users and processes them one at a time. This can take an excruciating amount of time for Nextcloud instances with a large amount of photos. It could potentially happen much more quickly if thumbnails were allowed to be processes in parallel (on multi-core instances). While I can certainly understand that certain instances wouldn't want this, it would be great to have an option to enable a certain number of workers (1 to max cpu threads) to significantly speed up this operation.

[josejsarmento]

I concur, this should be the norm. I haven't tested this yet, but I guess you could always try to run several generate-all runs on specific folders at the same time?

[Tirpitz93]

I concur, this should be the norm. I haven't tested this yet, but I guess you could always try to run several generate-all runs on specific folders at the same time?

I am trying this now (one instance per user), appears to be working but there is a bottleneck elsewhere, possibly in my DB or server config.

[st3iny]

Unfortunately, this is not possible because PHP still doesn't support multi threading. You could split the initial generation to multiple disjunct folders (e. g. run one process for /user1/files and a separate process for /user2/files).

[vwbusguy]

To be clear, it is still possible to do multi-threading in PHP, but it has to be coded for it: https://www.php.net/manual/en/book.pthreads.php

[st3iny]

Yeah it is in theory. However, the language and its standard lib is not entirely thread safe. One possibility is using multiple processes (forks) to parallelize the execution.