TID2013 scores

[kornelski]

FYI, scores for 290e39552e213ed352761114561f92b93c902f55 measured against TID2013 database:

TID2013 Category	Spearman correlation
Noise	-0.822
Actual	-0.853
Simple	-0.915
Exotic	-0.543
New	-0.818
Color	-0.843
Full	-0.634

[Ser-Gen]

@kornelski do you still have the results txt?

[kornelski]

No, but that's the script I use to run the test on macOS via Wine. It uses gnu parallel. Expects reference BMP images converted to PNG.

set -e
set -o pipefail
{
    for i in reference_images/*png; do
        for dis in distorted_images/`basename -s .png "$i"`_*.png; do
            echo 'printf "%s\t%s\n" $(../butteraugli/bazel-bin/butteraugli "'"$i"'" "'"$dis"'") "'"$dis"'"'
        done
    done
} | time parallel | sort -k2 > testtmp

 /Applications/Wine.app/Contents/Resources/bin/wine metrics_values/spearman.exe mos.txt testtmp

[Ser-Gen]

Latest version from current repo: https://github.com/google/butteraugli/commit/71b18b636b9c7d1ae0c1d3730b85b3c127eb4511

butteraugli_71b18b6.txt

TID2013 Category	Spearman correlation
Noise	-0.871
Actual	-0.872
Simple	-0.849
Exotic	-0.711
New	-0.852
Color	-0.858
Full	-0.760

---

Latest version from pik: https://github.com/google/pik/commit/b4866ff9332fe13b7f7f70e55de02459f5fbb3b3 As suggested in issue https://github.com/google/butteraugli/issues/46

butteraugli_main_pik_b4866ff.txt

TID2013 Category	Spearman correlation
Noise	-0.798
Actual	-0.814
Simple	-0.825
Exotic	-0.584
New	-0.782
Color	-0.831
Full	-0.643

[jyrkialakuijala]

It would be interesting to take a p-norm over the butteraugli field and use that for the spearman correlation. In JPEG XL development we take an average of 3, 6 and 12-norms to support decision making and mostly use the maximum based butteraugli score for reference only.

[Ser-Gen]

@jyrkialakuijala how can I force butteraugli to use p-norm?

[jan-wassenberg]

@Ser-Gen the Butteraugli inside JPEG XL supports this - would it be an option to use https://gitlab.com/wg1/jpeg-xl/-/blob/master/tools/butteraugli_main.cc ? To build it, we recommend Docker (documentation).

[Ser-Gen]

@jan-wassenberg can't run butteraugli_main from host system.

./butteraugli_main: error while loading shared libraries: libIlmImf-2_2.so.22: cannot open shared object file: No such file or directory

I will try to use it from docker, if fail to find workaround.

[jan-wassenberg]

Hi @Ser-Gen, did you successfully install the libopenexr-dev and other packages mentioned in doc/developing_in_debian.md?

[Ser-Gen]

@jan-wassenberg initially no, I used docker to build tools as you suggested. https://gitlab.com/wg1/jpeg-xl/-/blob/master/doc/developing_in_docker.md

After installing packages from doc/developing_in_debian.md same error is appeared.

[Ser-Gen]

Current master of JPEG-XL https://gitlab.com/wg1/jpeg-xl/-/tree/0709f3a494c901c3a9f93159312b1177e3fd4fe4

butteraugli-jpegxl-0709f3a4.txt

TID2013 Category	Spearman correlation
Noise	-0.809
Actual	-0.835
Simple	-0.858
Exotic	-0.607
New	-0.777
Color	-0.843
Full	-0.658

butteraugli-jpegxl-3-norm-0709f3a4.txt

TID2013 Category	Spearman correlation
Noise	-0.865
Actual	-0.884
Simple	-0.899
Exotic	-0.517
New	-0.813
Color	-0.862
Full	-0.655

[jan-wassenberg]

Looks like you got it working, nice! Are any changes needed to our documentation? I just verified that the Docker image has /usr/lib/x86_64-linux-gnu/libIlmImf-2_2.so.22, which is referenced from butteraugli_main.