RGGB compression somtimes worse than RawSpeedCompress+LZMA2

[StephanBusch]

In my ongoing tests with RGGB created by dcraw compression of FLIF is sometimes worse than RawSpeedCompress+LZMA2. RawSpeedCompress uses different Delta versions for uncompressed pixel data and applies ZigZag-encoding which helps all kinds of general purpose compressors.

https://drive.google.com/file/d/0ByLIAFlgldSoNFR6RmcwdTgxQTQ/view?usp=sharing leica_m82_05.rggb 11.359.612 (FLIF 0.1) leica_m82_05.rggb 10.928.429 (FLIF 0.1.3_slower_but_stronger -n) leica_m82_05.rggb 5.560.365 (RawSpeedCompress3 + LZMA2)

https://drive.google.com/file/d/0ByLIAFlgldSoalhkSkdsNmJNaDA/view?usp=sharing nikon_1_v2_17.rggb 9.931.720 (FLIF 0.1) nikon_1_v2_17.rggb 9.636.696 (FLIF 0.1.3_slower_but_stronger -n) nikon_1_v2_17.rggb 8.810.004 (RawSpeedCompress3 + LZMA2)

https://drive.google.com/file/d/0ByLIAFlgldSoZmJDUXl2YWVPS2M/view?usp=sharing nikon_d5200_14.rggb 24.334.053 (FLIF 0.1) nikon_d5200_14.rggb 23.684.359 (FLIF 0.1.3_slower_but_stronger -n) nikon_d5200_14.rggb 21.527.675 (RawSpeedCompress3 + LZMA2)

Can you please check why FLIF is worse here? Will future versions also get something like Delta+ZigZag+LZ?

In almost every case the -n switch provided better compression on RGGB

[psykauze]

Indeed, because it's just a minor correction. This evening I will work on more improvements, I will push this correction after or maybe @jonsneyers will do it.

RGGB method seems (to me) not implemented in the best way. For now, it's just an hack and users should manipulate image for matching the RGGB CFA (-rotate -flip -flop...) and FLIF can't handle (yet) the image orientation that is an issue.

Could you send me your results please ?

[StephanBusch]

here are my results. None of the images were rotated.

camera-raw.xlsx

[StephanBusch]

it would be cool if FLIF would rotate the RGGB automatically or it tries all 4 orientations and picks the best - may rotation also help on RGB images?

[psykauze]

@StephanBusch "may rotation also help on RGB images?" -> No, rotation "should" not change anything also for RGGB method.

FLIF has a bug that it can't detect the CFA Pattern "yet" so for now it only work with a [R G1][G2 B] RGGB file that's why we need to rotate the RGGB file for matching this CFA Pattern.

I've made raw2rggb script that detect "some" CFA patterns and rotate the rggb to the right way. (But this is not working with Windows, sorry)

[psykauze]

I've tested raw2flif with the RAW files of the usual set. Here the result. raw2flif.csv.txt (Edited)

//raw2flif fail with the Mayima-Leaf and the Phase One raws because exiftool didn't detect the orientation. Also, raw2flif fail with the fujifilm x e1 and the sigmas raw because of their attypic sensors.

Edit.: I've corrected the Mayima-Leaf and Phase One issues but I didn't make a pull request yet. You can get the new raw2rggb in my repository.

[psykauze]

There is an issue with raw2rggb and Phase_One_20130624-CF043185.iiq. I don't know why but the CFA pattern is not correctly decoded by dcraw(???). So, my assumption and the raw2rggb assumption is wrong. I've manualy converted it and I got ~1% compression improvement (~41,0MiB unless 41,4MiB).

[StephanBusch]

I cannot use that scripts raw2flif and raw2rggb because I don't have a linux system. Here are the results of FLIF 0.1.7 -n without scripts and without rotation: http://www.squeezechart.com/camera-raw4.html

[psykauze]

@StephanBusch Here is a *.bat that do the same as raw2rggb (It's buggy but it works) You need exiftool.exe, dcraw.exe and sed.exe (and its dependencies, see GnuWin website) in the same directory. raw2rggb.bat.txt

@jonsneyers Please add this script in the root directory please. Also add this correction. raw2rggb.sh.txt

[StephanBusch]

@psykauze Thank you for the .bat file. Since my benchmarks compare more compressors on exctly the same input files, rotating would feel like cheating so I can test that script but I cannot publish that results because then the other compressors also need to be tested with rotated files. This would be different, of course, if Flif itself would detect CFA pattern and if it would rotate automatically. My benchmark was created to see how far we can push lossless compression of RGGB and I really do appreciate the excellent work all of you do here. And I am very curious how your proposed MDL transform would work like and what further compression gains it will unlock. Thank you very much for all your efforts

[psykauze]

I'm so sorry to read that :'( The shell and *.bat files is for dev purpose, we know there is an issue about metadatas in the encoder. I consider this thing like a tweak, not a cheat

So there is some solutions I consider: 1) Working on rggb CFA detection internally but until we can't store the CFA Pattern in the flif file, it would be impossible to restore the original rggb file but we will have a great RGB file. 2) We can also rotate the file internally so the rggb output file can be decoded but rotated. 3) Developing the "channel sorting method" for YIQ (or MDL) transform will make the "CFA Pattern detection" useless (at least for "2x2 square" CFA Pattern)

[jonsneyers]

Storing metadata is trivial: just put it in a file and include it in the ar, see raw2flif which extracts the exiv metadata and puts it in the .flif file. It would be cleaner if we use libraw and avoid shell scripts, and perhaps let the decoder take rotation/CFA metadata into account directly. On Nov 25, 2015 15:25, "psykauze" notifications@github.com wrote:

I'm so sorry to read that :'( The shell and *.bat files is for dev purpose, we know there is an issue about metadatas in the encoder. I consider this thing like a tweak, not a cheat

So there is some solutions I consider: 1) Working on rggb CFA detection internally but until we can't store the CFA Pattern in the flif file, it would be impossible to restore the original rggb file but we will have a great RGB file. 2) We can also rotate the file internally so the rggb output file can be decoded but rotated. 3) Developing the "channel sorting method" for YIQ (or MDL) transform will make the "CFA Pattern detection" useless (at least for "square" CFA Pattern)

— Reply to this email directly or view it on GitHub https://github.com/FLIF-hub/FLIF/issues/151#issuecomment-159621197.

[StephanBusch]

@psykauze I decided to run those tests with your batch because I want to help you developing FLIF by testing. I would also want to have a batch file that produces a .flif. Instead of the linux-based ar you can use zlib or zstd.

Would it be better to design .flif with metadata as a compressed archive such as LibreOffice/MS Office or Mmiya-Leaf .eip files and store all metadata in a XML inside that archive? Or would it be better to attach metadata directly to .flif as it is with all original raw files?

I have downloaded raw2rggb.bat and put exiftool.exe, dcraw.exe sed.exe (with libintl3.dll, regex2.dll and libiconv2.dll). Then tried to run raw2rggb.bat canon_eos_5d_mark_iii_05.cr2 canon.rggb but I got an error message: "(" cannot be syntactically processed at this position.

[psykauze]

OK, I've found the issue(s). I'm sorry, I've made some corrections but I haven't tested the bat file :( (Reason, I have not Windows at home). I'm working on it.

[jonsneyers]

Well, currently FLIF supports metadata as arbitrary files, so it could be compressed or plaintext XML, or whatever. The FLIF decoder simply ignores all metadata at the moment, but perhaps we should standardize some metadata in order to allow viewflif to do something sensible with raw files. Information about rotation, CFA layout and channel multipliers would be needed (and sufficient?) to produce a reasonable non-interpolated, half-resolution image.

It would be nice if we can eventually losslessly convert DNG to FLIF and back without any shell/bat scripts, perhaps using libraw.

[StephanBusch]

@psykauze thank you very much

@jonsneyers yes, that would be my biggest wish - convert DNG to FLIF and then back to DNG again :))

@psykauze @jonsneyers Some men see things as they are and say "why?". We dream of things that never were and say "Why not?"

[psykauze]

Sorry @StephanBusch, I've tried to debug the bat script but for some reason I've got random results (same input file, output file size change each try).

You could use the previous version of bat file. It was tested and it works.

[psykauze]

@StephanBusch @jonsneyers The bat script is now working. I've got some issues due to Batch scripting limitation.

raw2rggb.txt raw2rggb.bat.txt

[StephanBusch]

thank you for the update I tried it with: raw2rggb.bat canon_eos_5d_mark_iii_05.cr2 c1.rggb

and no .rggb was created. The message was: D:\TESTSETS\TEST_CAMERA_flif_rotate>IF "RGGBRGGBRGGBRGGB" == "RGGB" (SET Rotate=-t 0 ) ELSE IF "RGGBRGGBRGGBRGGB" == "GRBG" (SET Rotate=-t 270 ) ELSE IF "RGGBRGGBRGGBRGGB" == "BGGR" (SET Rotate=-t 180 ) ELSE IF "RGGBRGGBRGGBRGGB" == "GBRG" (SET Rotate=-t 90 ) ELSE ( ECHO This pattern is unknown: RGGBRGGBRGGBRGGB EXIT /B 1 ) This pattern is unknown: RGGBRGGBRGGBRGGB

D:\TESTSETS\TEST_CAMERA_flif_rotate>

on next image canon_eos_6d_14 the error message is: SET PNMTYPE=P6 GOTO GT_DECOMP ) ELSE ( ECHO This raw color is unknown: EXIT /B 1 ) ) This raw color is unknown:

[psykauze]

Please use this dcraw instead: http://www.centrostudiprogressofotografico.it/en/dcraw/ (rename the exe as dcraw.exe)

Also, could you edit the bat and delete the first "REM" ? (Just the word, not the line)

[StephanBusch]

thank you.. using the provided DCRAW helped but not on the fujifilm_x_e1_20.raf.. the message here is:

D:\TESTSETS\TEST_CAMERA_flif_rotate>IF "GGRGGBGGBGGRBRGRBGGGBGGRGGRGGBRBGBRG" == "RGGB" (SET Rotate=-t 0 ) ELSE IF "GGRGGBGGBGGRBRGRBGGGBGGRGGRGGBRBGBRG" == "GRBG" (SET Rotate=-t 270 ) ELSE IF "GGRGGBGGBGGRBRGRBGGGBGGRGGRGGBRBGBRG" == "BGGR" (SET Rotate=-t 180 ) ELSE IF "GGRGGBGGBGGRBRGRBGGGBGGRGGRGGBRBGBRG" == "GBRG" (SET Rotate=-t 90 ) ELSE ( ECHO This pattern is unknown: GGRGGBGGBGGRBRGRBGGGBGGRGGRGGBRBGBRG EXIT /B 1 ) This pattern is unknown: GGRGGBGGBGGRBRGRBGGGBGGRGGRGGBRBGBRG

[psykauze]

Yeah. This is not an issue. This file can't be performed as a RGGB file in FLIF. The result will be worse than converting it in PGM mode.

[StephanBusch]

I get 11.098.920 bytes when input is .pgm and 11.064.698 bytes if input is .rggb

[StephanBusch]

after trying to compress the first canon image, I get an error with FLIF:

=== flif -v -v -v -v -n c1.rggb c1.flif === [] [_ | | | **] FLIF 0.1.9 [23 November 2015] | | || | Free Lossless Image Format ]|**|_[ (c) 2010-2015 J.Sneyers & P.Wuille, GNU GPL v3+

Loading input file: c1.rggb RGGB file should be a PGM, like the output of "dcraw -E -4". Cannot read other types. Could not read input file: c1.rggb

Execution time: 2.985 s

[psykauze]

I'm sorry but the script work as I expected. This file has a "weird" CFAPattern. FLIF can encode and decode it as a RGGB file without issues but if we try to decode it as a pam (or PNG) the result will be weird. I prefer the script fails for this kind of file than not be sure of the result.

[StephanBusch]

you mean the fuji file in the above message?

[psykauze]

Yeah.

I will check the canon issue maybe it's just because your flif is not from the last commit.

[jonsneyers]

It isn't, as you can tell from the date. Renaming it to .pnm should do the trick, @StephanBusch

@psykauze: I agree with not pretending that weird CFA patterns are actually RGGB. What do you think is the best approach to handling such files?

[StephanBusch]

if I use .pnm extension, only the PLC transform will be applied and the canon file is treated as 1 channel 16 bit: compression is also worse than it would be with .rggb extension

[jonsneyers]

Yes but if the file is not a PGM (P5) then it doesn't contain Bayered data anyway, but RGB pixels.

[StephanBusch]

I thought that raw2rggb.bat script creates RGGB rather than RGB pixels

[psykauze]

I found out why flif fails to decode the canon rggb file. It's because there is a 'space' caracter after P5 ("P5 \n" instead of "P5\n").

The new bat will not produce this 'space' raw2rggb.bat.txt

[jonsneyers]

hm, I'm not sure about the PGM spec but if it allows that space to be there, then this is a bug in the rggb/pgm reader...

[StephanBusch]

The problem is still there - FLIF cannot read the input file.

[psykauze]

@jonsneyers The fuji x e1 CFA pattern contain 8 red pixels, 8 blue pixels and 20 green pixels. We can't debayerize this file in 4 planes with the same dimensions. 1) Easiest way: Encoding it only in greyscale mode 2) Not so easy but bitstream changing: Encoding each planes alone (like RGB mode) but with size different plane 3) Harder: Allowing flif to use more than 4 planes. For fuji x e1, 2 blue + 2 red + 5 green = 9 planes. YIQ and MDL transform can be applied.

@StephanBusch It seems the batch file add an extra non-printable character (0x0d) before the end of line. I think this is a flif issue not really a raw2rggb.bat issue.

raw2rggb.bat.txt

[conradgaunt]

hello , im not a c coder, im a pascal coder ..so i won't understand flif code easily .. but ill share my limited knowledge re: raw / how ive been dealing with it in my own adventures .. although i only came on here to ask a question ..

first , a bit about me .. last march (2017) i spent a week throwing together a huffman coder , unhappy with 48bit cinema/ real world image compression options .. not sure what to expect...more as a learning exercise, mixed with frustration at being trapped inside 1980's image formats ... after 2 days i had imagemagick pumping out RGB files , and im compressing (easy)...and more importantly ..5 days later ..decompressing! (..got stoned after the success of compressing ....) ..compression results weren't great .. but reasonable .. so then i tried pre-filtering the image with a dumb filter (no logic like paeth etc). at that point a 300tb image archive test showed that change made files 56% smaller than the openj2k encoder managed (on average / overall etc) ..flif was a bit better of course, at about 59% smaller ..but i didn't know that for another 3weeks ..literally (*). i have j2k files that come out smaller than flif btw, just a few. my system never beats flif , but stays quite close. annoyingly , the dumb filter / first attempt/ placeholder filter seems to be the best ive come up with .. which should be good , but it means since ive picked up the code after a year long break (got stoned again) ..ive now spend several weeks wasting my time testing alternatives without results .. but i do have 7 different reversible color transforms now , and they are ALL useful / get used (hint)

during my initial experiments i also made a modified version to handle bayer patterns . i downloaded the blackmagic cinema camera sample files (all 11gig) , .. DCRAW can output a text file with meta data from the file read , i parse that , and get the RGGB order .. but ive only implemented the one BMCC uses out of laziness . anyway, thats enough about my unfeasibly sussessful attempts at compressing images losslessly

as for raw ..my advice for handing odd bayer patterns is to just decode them to full RGB . DCRAW can output 16bit linear bayer patterns , but odd patterns i just do a full decode , then throw away 2/3 of the values in any bayer configuration i want (or more precisely .. the same one BMCC uses..as ive implemented it already!)..reverse engineering the raw.. HOWEVER, what i intend to do in the future is count the number of original values in the red green and blue channels , and pick the bayer battern that maximizes the number of original values , OR has the widest dynamic range , with the latter trumping the former. as im compressing sequences of large images , ill do this analysis once / or a few times per sequence of images (same when choosing the correct RCT),.. then assume that pattern for all images in the sequence (or test images from the start/ middle / end of sequences and try using any filter already chosen as optimal by one of those etc) , but its pretty quick analysis. you can't cater of every proprietory pattern , unless perhaps you store metadata that can describe colors other than pure Red, Green, or Blue ( ie, by specify each bayer color component as a seperate set of R/G/B values in the metadata, yellow and purple filters can be used and colour adjusted .. but because im lazy , and manufacturers making sensors with purple and yellow filters can just f**k off / are perverts ..). the reality is , the physical colour filters used on the sensor will not be pure red green or blue anyway , and if you throw away the same amount of information that was never captured by the sensor taking the image in the first place , especially after its been fully decoded .. my tests show that reconstruction using simple bi-linear interolation never produces artifacts

ive been focusing on debayering recent;y..wondering if adaptive debayering techniques can cause jitter in moving images. no artifacts appears in decoded redcode footage when simple bi-linear interpolation is used to debayer a "pseudo-reverse engineered raw file",..even after the image is downsampled to quarter resolution .. so im starting to think thats how red debayer too (no doubt because the sensor has a strong OLPF dropping optical resolution/ edge contrast making demosaicing trivial) . ive been calling these pseudo raw images "documentary format" , as they save a LOT of space compared to the original 48bit TIFFs i generally get to play with . optionally , the pixels can be converted to 16bit half floats (a lossy transformation resulting in even smaller files , but identical dynamic range) .the main reason im trying to re-invent the wheel is because im not prepared to store more pixels than the cameras i use captured at the time, long -term .. not until storage is free

as for compressing DNG files (or as i call them .. dung files ) ., i reencoded the blackmagic sample files , and compressed them using the correct bayer pattern , think i knocked off between 10-20%. i used the bayer reversible colour transform described on the microsoft research site as the RCT option , and i prefiltered the raw files before compressing by simply sweeping right to left across each image row , and subtracting the value 2 places to the left (filtering right to left during encoding, filtering left to right while decoding etc ) .. but then i also filter the filtered results vertically (from bottom to top, again, subtracting the value 2 rows above from the current value) . the results of pre comrpession filtering are signed values btw, bit depth arbitrary

not sure if any of that is useful , but feel free to reach out to discuss my ramblings etc/

so ..i do have a couple of questions myself .. which is why i came on here : 1) is flif ready for prime time yet , or still work in progress ? .. btw, id like you to add more RCT's ;) 2) i couldn't get the flif encoder on the flif site to compress anything 48bit , so im using imagemagick to deal / make FLIF files.. are there any paramters for image magick that anyone knows of , such that i can do a brute force optimization using all available options ..and is it possible to encode bayer patterns with flif (i know it is ... but using imagemagick?)

thanks in advance

k.r

[conradgaunt]

ps, final thought,

many of the RCT's ive used in my compression system were designed by other people, and based around their understanding of how the human eye/ brain works etc .. however , lossless compression has nothing to do with how the eye works, or the brain recieves colour .. its just about reducing bytes ..but a lot of research papers i read start with lossy transforms yuv etc .. then elaborate them into lossless versions .. why!! i think zig zag patterns largely fall into that category . i believe the zig zag pattern is more useful in a lossy context, and is used for perceptual reasons , less useful in a lossless context