The encoder outputs incorrect qoi with some inputs

wx257osn2 commented 2 years ago

According to below script, the encoder of rapid-qoi outputs incorrect data with some inputs:

#!/bin/bash

QOICONV_REF=./qoi/qoiconv
QOICONV_TARGET=./rapid-qoi/target/release/qoiconv
IMAGES=./qoi/images

rm -f /tmp/{ref,target}.{qoi,png}

result=0
find ${IMAGES} | grep -E '.png$' | while read x; do
  ${QOICONV_REF} "${x}" /tmp/ref.qoi
  ${QOICONV_TARGET} "${x}" /tmp/target.qoi 2>/dev/null
  ret=$?
  if [ ${ret} -ne 0 ]; then
    echo ${x} : encode failed by target encoder. skip...
    rm /tmp/ref.qoi
    continue
  fi
  ${QOICONV_REF} /tmp/ref.qoi /tmp/ref.png
  ${QOICONV_REF} /tmp/target.qoi /tmp/target.png >/dev/null
  ret=$?
  if [ ${ret} -ne 0 ]; then
    echo ${x} : decode failed by reference decoder
    rm /tmp/ref.{qoi,png} /tmp/target.qoi
    result=1
    continue
  fi
  diff /tmp/ref.png /tmp/target.png > /dev/null
  ret=$?
  if [ ${ret} -ne 0 ]; then
    echo ${x} : encode by target encoder is not correct
    result=1
  fi
  rm /tmp/{ref,target}.{qoi,png}
done

exit ${result}

./qoi/images/photo_wikipedia/034.png : encode by target encoder is not correct
./qoi/images/screenshot_game/Severed_screenshot_05.png : encode by target encoder is not correct
./qoi/images/screenshot_game/TowerFall_Flight.png : encode by target encoder is not correct
./qoi/images/screenshot_game/Mapperkmud.png : encode by target encoder is not correct
./qoi/images/screenshot_game/Severed_screenshot_04.png : encode by target encoder is not correct
./qoi/images/screenshot_game/Towdie_Screenshot.png : encode by target encoder is not correct
./qoi/images/screenshot_game/Htdamg_affinity.png : encode failed by target encoder. skip...
./qoi/images/screenshot_game/3dbrick2.png : encode by target encoder is not correct
./qoi/images/screenshot_game/Nidhogg_video_game_screenshot_2.png : encode by target encoder is not correct
./qoi/images/screenshot_game/Triplane_Turmoil_screenshot.png : encode by target encoder is not correct
./qoi/images/screenshot_game/Cibele_video_game_screenshot_2.png : encode by target encoder is not correct
./qoi/images/screenshot_game/Noctis_Screenshot.png : encode by target encoder is not correct
./qoi/images/screenshot_game/TowerFall_Thornwood.png : encode by target encoder is not correct
./qoi/images/screenshot_game/Job_Simulator_screenshot_-_Auto_01.png : encode by target encoder is not correct
./qoi/images/screenshot_game/Frogatto.png : encode by target encoder is not correct
./qoi/images/screenshot_game/Blek_screenshot_collection_2.png : encode by target encoder is not correct
./qoi/images/screenshot_game/Kye.png : encode by target encoder is not correct
./qoi/images/screenshot_game/Bankai.png : encode by target encoder is not correct
./qoi/images/screenshot_game/Ballerburg.png : encode by target encoder is not correct
./qoi/images/screenshot_game/Heartlight.png : encode by target encoder is not correct
./qoi/images/screenshot_game/Sega-Mega-Drive-regional-lockout-error.png : encode by target encoder is not correct
./qoi/images/screenshot_game/Core_War_PMars_Screenshot.png : encode by target encoder is not correct
./qoi/images/screenshot_game/Flinthook_screenshot_-_Pax_square_5.png : encode by target encoder is not correct
./qoi/images/screenshot_game/TowerFall_Backfire.png : encode by target encoder is not correct
./qoi/images/screenshot_game/Blek_screenshot_collection_3.png : encode by target encoder is not correct
./qoi/images/screenshot_game/Ai_dungeon_with_a_custom_setting.png : encode by target encoder is not correct
./qoi/images/screenshot_game/Openciv-0.95.png : encode by target encoder is not correct
./qoi/images/screenshot_game/Gang_Garrison_2.png : encode by target encoder is not correct
./qoi/images/screenshot_game/This_Is_the_Police_-_Screenshot_09.png : encode by target encoder is not correct
./qoi/images/screenshot_game/Neverball_Kursauswahl.png : encode by target encoder is not correct
./qoi/images/screenshot_game/The_Bub%27s_Brothers.png : encode by target encoder is not correct
./qoi/images/screenshot_game/Smiler_in_Arrowe_Land.png : encode by target encoder is not correct
./qoi/images/screenshot_game/Wc4source.png : encode by target encoder is not correct
./qoi/images/screenshot_game/Blek_screenshot_collection_4.png : encode by target encoder is not correct
./qoi/images/screenshot_game/Flinthook_screenshot_-_Pax_square_3.png : encode by target encoder is not correct
./qoi/images/screenshot_game/Maelstrom_screenshot.png : encode by target encoder is not correct
./qoi/images/screenshot_game/Volcano.png : encode by target encoder is not correct
./qoi/images/screenshot_game/Hedgeweap.png : encode by target encoder is not correct
./qoi/images/screenshot_game/StationSnapshot2.png : encode by target encoder is not correct
./qoi/images/screenshot_game/TowerFall_Sacred_Ground.png : encode by target encoder is not correct
./qoi/images/screenshot_game/OXO_emulated_portion.png : encode by target encoder is not correct
./qoi/images/screenshot_game/Dr._Langeskov,_The_Tiger,_and_The_Terribly_Cursed_Emerald_Screenshot_6.png : encode by target encoder is not correct
./qoi/images/screenshot_game/A7xpg-continue.png : encode by target encoder is not correct
./qoi/images/screenshot_game/Threes_development_process_art_mockup_21.png : encode by target encoder is not correct
./qoi/images/screenshot_game/Temporal_game_screenshot.png : encode by target encoder is not correct
./qoi/images/screenshot_game/Netrek-client-cow.png : encode by target encoder is not correct
./qoi/images/textures_pk/pkw_wall02bk.png : encode by target encoder is not correct
./qoi/images/textures_pk/pk_conbuild01c.png : encode by target encoder is not correct
./qoi/images/textures_pk/+0mod_button1.png : encode by target encoder is not correct
./qoi/images/textures_pk/pkf_tilewall1e2.png : encode by target encoder is not correct
./qoi/images/textures_pk/pkf_wall4k.png : encode by target encoder is not correct
./qoi/images/textures_pk/pkw_wall01g.png : encode by target encoder is not correct
./qoi/images/textures_pk/mod_screen1g.png : encode by target encoder is not correct
./qoi/images/textures_pk/SW_Window01a.png : encode by target encoder is not correct
./qoi/images/textures_pk/pk_chinwall2l.png : encode by target encoder is not correct
./qoi/images/textures_pk/mod_screene.png : encode by target encoder is not correct
./qoi/images/textures_pk/pkw_wall02ag.png : encode by target encoder is not correct
./qoi/images/textures_pk/pkw_wall01c.png : encode by target encoder is not correct
./qoi/images/textures_pk/mis_wall2b.png : encode by target encoder is not correct
./qoi/images/textures_pk/pkf_containr2a.png : encode by target encoder is not correct
./qoi/images/textures_pk/pkf_containr1a.png : encode by target encoder is not correct
./qoi/images/textures_pk/mod_screen1h.png : encode by target encoder is not correct
./qoi/images/textures_pk/{pkf_auttrees1a.png : encode by target encoder is not correct
./qoi/images/textures_pk/pkw_wall02ac.png : encode by target encoder is not correct
./qoi/images/textures_pk/SW_Gate01.png : encode by target encoder is not correct
./qoi/images/textures_pk/pkf_wall4h.png : encode by target encoder is not correct
./qoi/images/textures_pk/pk_chinwall2g.png : encode by target encoder is not correct
./qoi/images/textures_pk/pkw_gwall02a.png : encode by target encoder is not correct
./qoi/images/textures_pk/pkw_wall04m.png : encode by target encoder is not correct
./qoi/images/textures_pk/mod_labvent1b.png : encode by target encoder is not correct
./qoi/images/textures_pk/scroll_modwater.png : encode by target encoder is not correct
./qoi/images/textures_pk/pkf_wall4g.png : encode by target encoder is not correct
./qoi/images/textures_pk/pkf_wall4o.png : encode by target encoder is not correct
./qoi/images/textures_pk/pkw_wall01e.png : encode by target encoder is not correct
./qoi/images/textures_pk/pkw_wall02bc.png : encode by target encoder is not correct
./qoi/images/textures_pk/pkf_wall5c.png : encode by target encoder is not correct
./qoi/images/textures_pk/pk_chmud1.png : encode by target encoder is not correct
./qoi/images/textures_pk/pk_chinflw1b.png : encode by target encoder is not correct
./qoi/images/textures_pk/pkf_tilewall1g.png : encode by target encoder is not correct
./qoi/images/textures_pk/pkw_wall07k.png : encode by target encoder is not correct
./qoi/images/textures_pk/pkf_tilewall1c.png : encode by target encoder is not correct
./qoi/images/textures_pk/pkw_wall02ae.png : encode by target encoder is not correct
./qoi/images/textures_pk/+amod_button1.png : encode by target encoder is not correct
./qoi/images/textures_pk/pkf_containr2c.png : encode by target encoder is not correct
./qoi/images/textures_pk/pkw_wall01j.png : encode by target encoder is not correct
./qoi/images/textures_pk/pkf_containr3c.png : encode by target encoder is not correct
./qoi/images/textures_pk/pk_chrooft1c.png : encode by target encoder is not correct
./qoi/images/textures_pk/pkw_wall01b.png : encode by target encoder is not correct
./qoi/images/textures_pk/pkf_tilewall1f2.png : encode by target encoder is not correct
./qoi/images/textures_pk/SW_Gate03.png : encode by target encoder is not correct
./qoi/images/textures_pk/gow_wallceiling1.png : encode by target encoder is not correct
./qoi/images/textures_pk/pkf_shopwall1e.png : encode by target encoder is not correct
./qoi/images/textures_pk/pkw_food1c.png : encode by target encoder is not correct
./qoi/images/textures_pk/pk_chinwall2h.png : encode by target encoder is not correct
./qoi/images/textures_pk/pkw_wall02bf.png : encode by target encoder is not correct
./qoi/images/textures_pk/pkw_wall07i.png : encode by target encoder is not correct
./qoi/images/textures_pk/pkw_wall07j.png : encode by target encoder is not correct
./qoi/images/textures_pk/pkf_containr1c.png : encode by target encoder is not correct
./qoi/images/textures_pk/mod_vtrim01a.png : encode by target encoder is not correct
./qoi/images/textures_pk/pkw_wall02bd.png : encode by target encoder is not correct
./qoi/images/textures_pk/pkf_tilewall1b.png : encode by target encoder is not correct
./qoi/images/textures_pk/pkw_gwall03a.png : encode by target encoder is not correct
./qoi/images/textures_pk/mod_vtrim01b.png : encode by target encoder is not correct
./qoi/images/textures_pk/pkf_light2b.png : encode by target encoder is not correct
./qoi/images/textures_pk/mod_metfloor2c.png : encode by target encoder is not correct
./qoi/images/textures_pk/pkw_wall01d.png : encode by target encoder is not correct
./qoi/images/textures_pk/pkw_wall02ad.png : encode by target encoder is not correct
./qoi/images/textures_pk/pk_chrock1.png : encode by target encoder is not correct
./qoi/images/textures_pk/pkf_vent1a.png : encode by target encoder is not correct
./qoi/images/textures_pk/{SW_Tree02.png : encode by target encoder is not correct
./qoi/images/textures_pk/pkw_wall01i.png : encode by target encoder is not correct
./qoi/images/textures_pk/mod_labvent1a.png : encode by target encoder is not correct
./qoi/images/textures_pk/pkf_tilewall1e.png : encode by target encoder is not correct
./qoi/images/textures_pk/pk_chinwall2c.png : encode by target encoder is not correct
./qoi/images/textures_pk/pkw_wall03f.png : encode by target encoder is not correct
./qoi/images/textures_pk/pkf_vent1b.png : encode by target encoder is not correct
./qoi/images/textures_pk/pkw_wall07b.png : encode by target encoder is not correct
./qoi/images/textures_pk/pk_chinflw1a.png : encode by target encoder is not correct
./qoi/images/textures_pk/mod_screenh.png : encode by target encoder is not correct
./qoi/images/textures_pk/mis_wall2a.png : encode by target encoder is not correct
./qoi/images/textures_pk/pkf_tilewall1g2.png : encode by target encoder is not correct
./qoi/images/textures_pk/SW_Wall01b.png : encode by target encoder is not correct
./qoi/images/textures_pk/SW_Wall02b.png : encode by target encoder is not correct
./qoi/images/textures_pk/pkw_gwall03b.png : encode by target encoder is not correct
./qoi/images/textures_pk/pkw_trim06.png : encode by target encoder is not correct
./qoi/images/textures_pk/pkf_wall5b.png : encode by target encoder is not correct
./qoi/images/textures_pk/pkf_creds.png : encode by target encoder is not correct
./qoi/images/textures_pk/mod_labcab1b.png : encode by target encoder is not correct
./qoi/images/textures_pk/mod_door3.png : encode by target encoder is not correct
./qoi/images/textures_pk/pkf_containr3a.png : encode by target encoder is not correct
./qoi/images/textures_pk/pkw_wall01f.png : encode by target encoder is not correct
./qoi/images/textures_pk/pkw_wall07d.png : encode by target encoder is not correct
./qoi/images/textures_pk/pkw_gwall02b.png : encode by target encoder is not correct
./qoi/images/textures_pk/pkf_door1b.png : encode by target encoder is not correct
./qoi/images/photo_tecnick/RGB_OR_1200x1200_046.png : encode by target encoder is not correct
./qoi/images/photo_tecnick/RGB_OR_1200x1200_003.png : encode by target encoder is not correct
./qoi/images/photo_tecnick/RGB_OR_1200x1200_001.png : encode by target encoder is not correct
./qoi/images/photo_tecnick/RGB_OR_1200x1200_034.png : encode by target encoder is not correct

zakarumych commented 2 years ago

Interesting. Thank you for the report! I will check and see if encoder or decoder step is to blame.

zakarumych commented 2 years ago

So here what I found. Encoders indeed produced a bit different output. And reference qoiconv converted them to a bit different PNGs. But they all were encoding the exactly same pixels.

I've added

${QOICONV_TARGET} /tmp/ref.png /tmp/ref.raw
${QOICONV_TARGET} /tmp/target.png /tmp/target.raw

which saves raw pixel data into files.

Turned out, the output was exactly the same for all listed images except for 3 occurrences.

./qoi_benchmark_suite/images/screenshot_game/Ballerburg.png
./qoi_benchmark_suite/images/screenshot_game/Sega-Mega-Drive-regional-lockout-error.png
./qoi_benchmark_suite/images/screenshot_game/OXO_emulated_portion.png

Those three have Luma8 format and reference qoiconv decoder converts them to RGBA QOIs. While rapid-qoi conversion tool produces RGB QOIs. I've added temporary change to convert Luma to RGBA and script prints no differences, only two failures to parse source PNGs by image crate.

If you don't mind, I'll add modified version of your script to the repository to test that all four possible roundtrips produce exactly the same pixel values.

wx257osn2 commented 2 years ago

If you don't mind, I'll add modified version of your script to the repository to test that all four possible roundtrips produce exactly the same pixel values.

Sure.

I want to check the result. Could you like to push the raw-data output feature or put the patch for qoiconv/src/main.rs here?

zakarumych commented 2 years ago

Yea, I will push changes to allow conversion to raw pixels data and the script.

zakarumych commented 2 years ago

I wasn't active lately, but I've pushed fix and validation script. The script contains paths to binaries and images directory at the top.

wx257osn2 commented 2 years ago

@zakarumych I've confirmed the new commit, but the validate.sh outputs like below for entire qoi_benchmark_suite :

../qoi/images/screenshot_game/Htdamg_affinity.png : encode failed by target encoder. skip...
../qoi/images/screenshot_game/Kikinanobot.png : encode failed by target encoder. skip...
../qoi/images/pngimg/viber_PNG2.png : encode by target encoder is not correct
../qoi/images/pngimg/viber_PNG2.png : roundtrip by target encoder-decoder is not correct

Let's ignore the two in screenshot_game, the result for pngimg/viber_PNG2.png seems to be not correct.

wx257osn2 commented 2 years ago

@zakarumych In my persornal project qoi-benchmark, rapid-qoi passes below 3 validations:

input png --stb_image--> original raw
original raw --reference encoder--> qoi encoded by reference implementation
original raw --rapid_qoi encoder --> qoi encoded by rapid-qoi
1. qoi encoded by reference implementation --rapid-qoi decoder--> raw decoded by rapid-qoi == original raw
2. qoi encoded by rapid_qoi --reference decoder--> raw decoded by reference implementation == original raw
3. qoi encoded by rapid_qoi --rapid-qoi decoder--> raw decoded by rapid-qoi == original raw

So, it seems that the core logic of rapid-qoi had been fixed at last week. Like the difference of behaviors between image crate and stb_image library you had told me, the problem of viber_PNG2.png might be in the image reader of rapid-qoi/qoiconv IMO.

In summary, rapid-qoi/qoiconv may encode differently than the reference implementation, but the rapid-qoi as a library had been fixed. It'd be better if this problem will be fixed, but I don't mind to close this issue since it now works correctly as a library. (If you will close this issue without fixing above problem, it may be better to write about qoiconv behavior in the README. )

zakarumych / rapid-qoi

The encoder outputs incorrect qoi with some inputs #6