Allow users to pass max_threads to the avif encoder via Image.save

[yit-b]

It's not always desirable to set max_threads equal to the number of cpus on the host machine - especially in shared or containerized environments. Performance is often significantly worse if you have too many CPUs (see benchmarks below).

This PR gives users control over how many threads will be used for encoding by adding a max_threads argument accepted by Image.save(). Example:

save_test.py

import time
import pillow_avif
from PIL import Image

if __name__ == "__main__":
    img = Image.open("tests/images/flower.jpg")

    n_warmups = 1
    n_iters = 10

    for n_threads in range(0, 17):
        for _ in range(n_warmups):
            img.save("flower.avif", quality=80, max_threads=n_threads)
        start = time.time()
        for _ in range(n_iters):
            img.save("flower.avif", quality=80, max_threads=n_threads)
        print(
            f"N threads: {n_threads}. Avg time: {((time.time() - start) / n_iters * 1000):.2f} ms/img"
        )

Output:

N threads: 0. Avg time: 71.45 ms/img
N threads: 1. Avg time: 128.72 ms/img
N threads: 2. Avg time: 76.32 ms/img
N threads: 3. Avg time: 61.18 ms/img
N threads: 4. Avg time: 62.81 ms/img
N threads: 5. Avg time: 61.81 ms/img
N threads: 6. Avg time: 61.33 ms/img
N threads: 7. Avg time: 65.15 ms/img
N threads: 8. Avg time: 62.29 ms/img
N threads: 9. Avg time: 63.38 ms/img
N threads: 10. Avg time: 62.56 ms/img
N threads: 11. Avg time: 61.89 ms/img
N threads: 12. Avg time: 64.02 ms/img
N threads: 13. Avg time: 64.10 ms/img
N threads: 14. Avg time: 66.18 ms/img
N threads: 15. Avg time: 64.52 ms/img
N threads: 16. Avg time: 64.76 ms/img

The default behavior remains unchanged. It is 0 if not specified which is set to the cpu count.

Maybe this should be changed to a more reasonable default since performance seems the same with max_threads=0 (all CPUs) as it does with just 2 - probably because of contention? Maybe changing the default is outside the scope of this PR but I'd like to be able to tailor the parallelism to my compute environment.

Reproduce my tests:

conda activate foobar # This is some new empty conda environment
conda install -c conda-forge pillow 'libavif>=1.0.2' aom 'python=3.10.*' pillow
cd pillow-avif-plugin
pip install --no-deps .
python save_test.py

[yit-b]

CI failures seem unrelated to my changes.

[fdintino]

I think the rav1e and SVT-AV1 codecs probably make better use of max_threads. Could you run your script and post the results when you pass codec="rav1e" and codec="svt" to save()?

[yit-b]

@fdintino

Codec: aom. N threads: 0. Avg time: 67.91 ms/img
Codec: aom. N threads: 1. Avg time: 130.41 ms/img
Codec: aom. N threads: 2. Avg time: 77.05 ms/img
Codec: aom. N threads: 3. Avg time: 61.73 ms/img
Codec: aom. N threads: 4. Avg time: 63.33 ms/img
Codec: aom. N threads: 5. Avg time: 61.71 ms/img
Codec: aom. N threads: 6. Avg time: 63.15 ms/img
Codec: aom. N threads: 7. Avg time: 64.46 ms/img
Codec: aom. N threads: 8. Avg time: 63.20 ms/img
Codec: aom. N threads: 9. Avg time: 61.67 ms/img
Codec: aom. N threads: 10. Avg time: 63.49 ms/img
Codec: aom. N threads: 11. Avg time: 62.05 ms/img
Codec: aom. N threads: 12. Avg time: 63.82 ms/img
Codec: aom. N threads: 13. Avg time: 66.58 ms/img
Codec: aom. N threads: 14. Avg time: 63.10 ms/img
Codec: aom. N threads: 15. Avg time: 65.07 ms/img
Codec: aom. N threads: 16. Avg time: 71.24 ms/img
Codec: rav1e. N threads: 0. Avg time: 742.48 ms/img
Codec: rav1e. N threads: 1. Avg time: 734.62 ms/img
Codec: rav1e. N threads: 2. Avg time: 730.26 ms/img
Codec: rav1e. N threads: 3. Avg time: 731.19 ms/img
Codec: rav1e. N threads: 4. Avg time: 736.11 ms/img
Codec: rav1e. N threads: 5. Avg time: 742.12 ms/img
Codec: rav1e. N threads: 6. Avg time: 736.13 ms/img
Codec: rav1e. N threads: 7. Avg time: 729.60 ms/img
Codec: rav1e. N threads: 8. Avg time: 729.46 ms/img
Codec: rav1e. N threads: 9. Avg time: 726.91 ms/img
Codec: rav1e. N threads: 10. Avg time: 722.95 ms/img
Codec: rav1e. N threads: 11. Avg time: 740.03 ms/img
Codec: rav1e. N threads: 12. Avg time: 724.34 ms/img
Codec: rav1e. N threads: 13. Avg time: 726.96 ms/img
Codec: rav1e. N threads: 14. Avg time: 730.37 ms/img
Codec: rav1e. N threads: 15. Avg time: 731.43 ms/img
Codec: rav1e. N threads: 16. Avg time: 728.80 ms/img
Codec: svt. N threads: 0. Avg time: 578.26 ms/img
Codec: svt. N threads: 1. Avg time: 394.54 ms/img
Codec: svt. N threads: 2. Avg time: 193.15 ms/img
Codec: svt. N threads: 3. Avg time: 208.09 ms/img
Codec: svt. N threads: 4. Avg time: 205.94 ms/img
Codec: svt. N threads: 5. Avg time: 198.58 ms/img
Codec: svt. N threads: 6. Avg time: 283.46 ms/img
Codec: svt. N threads: 7. Avg time: 278.86 ms/img
Codec: svt. N threads: 8. Avg time: 279.64 ms/img
Codec: svt. N threads: 9. Avg time: 278.56 ms/img
Codec: svt. N threads: 10. Avg time: 287.89 ms/img
Codec: svt. N threads: 11. Avg time: 285.51 ms/img
Codec: svt. N threads: 12. Avg time: 284.84 ms/img
Codec: svt. N threads: 13. Avg time: 287.96 ms/img
Codec: svt. N threads: 14. Avg time: 294.32 ms/img
Codec: svt. N threads: 15. Avg time: 282.62 ms/img
Codec: svt. N threads: 16. Avg time: 294.30 ms/img

[una-dinosauria]

@fdintino ping.

This PR makes the plugin much more usable for large-scale image compression and does not alter the default behaviour. Please let us know if there's anything else you'd like to see.

[yit-b]

@fdintino gentle bump

[una-dinosauria]

Looks like the issue was closed by accident and moved to https://github.com/fdintino/pillow-avif-plugin/pull/54

[fdintino]

Yes, I cross-referenced it in the PR description there, but should have commented here as well. Thanks.