RGGB compression somtimes worse than RawSpeedCompress+LZMA2

[StephanBusch]

In my ongoing tests with RGGB created by dcraw compression of FLIF is sometimes worse than RawSpeedCompress+LZMA2. RawSpeedCompress uses different Delta versions for uncompressed pixel data and applies ZigZag-encoding which helps all kinds of general purpose compressors.

https://drive.google.com/file/d/0ByLIAFlgldSoNFR6RmcwdTgxQTQ/view?usp=sharing leica_m82_05.rggb 11.359.612 (FLIF 0.1) leica_m82_05.rggb 10.928.429 (FLIF 0.1.3_slower_but_stronger -n) leica_m82_05.rggb 5.560.365 (RawSpeedCompress3 + LZMA2)

https://drive.google.com/file/d/0ByLIAFlgldSoalhkSkdsNmJNaDA/view?usp=sharing nikon_1_v2_17.rggb 9.931.720 (FLIF 0.1) nikon_1_v2_17.rggb 9.636.696 (FLIF 0.1.3_slower_but_stronger -n) nikon_1_v2_17.rggb 8.810.004 (RawSpeedCompress3 + LZMA2)

https://drive.google.com/file/d/0ByLIAFlgldSoZmJDUXl2YWVPS2M/view?usp=sharing nikon_d5200_14.rggb 24.334.053 (FLIF 0.1) nikon_d5200_14.rggb 23.684.359 (FLIF 0.1.3_slower_but_stronger -n) nikon_d5200_14.rggb 21.527.675 (RawSpeedCompress3 + LZMA2)

Can you please check why FLIF is worse here? Will future versions also get something like Delta+ZigZag+LZ?

In almost every case the -n switch provided better compression on RGGB

[psykauze]

Could you try renaming the RGGB file in PPM and compression it in FLIF interlaced ?

[matthiaskrgr]

I wrote a simple crusher script for flif (matthiaskrgr/flifcrush), you can try running it on one of the images:

export FLIF=/path/to/flif/binary
main.py nikon_d5200_14.rggb

It will take a lot of time but it might get the size down a bit further.

edit: BE CAREFUL, it might need a lot of ram

[jonsneyers]

I found out what is happening. I know how to solve this. I just need to implement it. Soon FLIF will beat that other method.

[jonsneyers]

Here you go:

11,359,333  leica_m82_05.rggb.flif0.1.3
5,117,762  leica_m82_05.rggb.flif0.1.4
8,450,275  nikon_1_v2_17.rggb.flif0.1.4
20,384,559 nikon_d5200_14.rggb.flif0.1.4

[StephanBusch]

There are still RGGB that compress better with Delta + ZigZag + LZMA2 than they do with FLIF:

https://drive.google.com/file/d/0ByLIAFlgldSoNlR4ZURaTUE2QTg/view?usp=sharing pentax_q10_19.rggb 10.692.568 (FLIF 0.1.4) pentax_q10_19.rggb 10.167.315 (Delta + ZigZag + LZMA2)

https://drive.google.com/file/d/0ByLIAFlgldSoSkZyQzJvWFNpQTQ/view?usp=sharing panasonic_lumix_dmc_gh3_10.rggb 14.186.367 (FLIF 0.1.4) panasonic_lumix_dmc_gh3_10.rggb 13.496.039 (Delta + ZigZag + LZMA2)

PLC helped a lot - on most files FLIF is stronger than without it, and sometimes it also outperforms RawSpeedCompress3 + Delta + ZigZag + MCM 0.83 -x9 but I think it has the capability of beating it on every RGGB image. can you check that again, please? See the almost complete test here: http://www.squeezechart.com/camera-raw.html

[psykauze]

As I see, Flif outperform the other method by >1% on average, that is really good. Because each compressor has its own method, sometimes one compressor that usually be worse than others can be more efficient for some case.

You can check some plots at issue #144 you will see that FLIF can sometimes outperform JPEG in its own sandbox, and sometimes webp outperform FLIF.

[jonsneyers]

The Bayer CFA layout is different for different camera models; FLIF currently assumes some specific subpixel layout for its YIQ transform, and if this assumption is wrong, then compression suffers a bit. You can use flif -R to disable YIQ. Also, disabling channel compactification with -C can help.

14,005,394 panasonic_lumix_dmc_gh3_10.rggb.R.flif 13,416,647 panasonic_lumix_dmc_gh3_10.rggb.RC.flif

10,481,645 pentax_q10_19.rggb.R.flif 10,481,603 pentax_q10_19.rggb.RC.flif

Does anyone have a good source of information on these subpixel layouts? I tried reading the dcraw source code, but that code is not very readable...

[psykauze]

Dcraw's exif output can show the CFA pattern as a matrix {Color1, Color2}{Color3, Color4}. DNGs and many (all?) RAWs formats has an Exif data for it.

Edit: I've found the raw files of panasonic_lumix_dmcgh3*.rw2 here http://www.photographyblog.com/reviews/panasonic_lumix_dmc_gh3_review/sample_images/ The CFA patern is GB/RG

[StephanBusch]

maybe RawSpeed can help? https://github.com/klauspost/rawspeed/tree/Rawspeed they have a camera.xml database for supporting new cameras: https://github.com/klauspost/rawspeed/blob/develop/cameras-xml.md

[psykauze]

I found that Exiftools does not mention the CFA pattern of the RW2, libopenraw documentation say that the pattern should be BGGR and DCRAW say the CFA pattern of the file is GBRG.

Which one to trust ?

From "ISO TC 42 N 5737" TIFF & DNGs allows patterns matrix bigger than 2x2 and can use Red, Green, Blue, Cyan, Magenta, Yellow or White colors. TIFFtag name is CFAPattern (0x828e), maybe you could check this value before converting it.

[jonsneyers]

Also: how does DCRAW output the extracted data? When FLIF reads a .rggb file, it is using the following layout, which I think is what DCRAW produces:

GG RB

Another complication is that these camera raw formats do not seem to have the same range for R, G and B. I'm not even sure if G1 and G2 are always in the same range. So that also messes up the YIQ.

Probably the way to get best compression (if you're not interested in progressive decoding) is to use flif -Rbn, with or without -C. You can usually improve further by using higher -r and -S (simultaneously), e.g. flif -RCbnS120 -r8.

[psykauze]

I'm sorry but DCRAW outputs with the original pattern, the most common case is: RG GB

I'm working at image-rggb.cpp there is also an issue for maxval < 0xff at blue plane

[jonsneyers]

Thanks, @psykauze . Now we should probably preprocess raw input to be in RG GB format.

[psykauze]

You should also change the decoder part ;)

Edit.: I've made some tests with "only" the planes correction here the results: filename,before,after TEST/canon_eos_5d_mark_iii_05.rggb,20098313,20175045 TEST/canon_eos_6d_14.rggb,19328964,19460476 TEST/canon_eos_m_04.rggb,18844941,18949992 TEST/fujifilm_finepix_x100_11.rggb,9652032,9640802 TEST/fujifilm_x_e1_20.rggb,11347023,11308835 TEST/fujifilm_xf1_08.rggb,6197773,6216062 TEST/leica_m82_05.rggb,5117725,5132863 TEST/leica_x1_10.rggb,9790809,9735075 TEST/nikon_1_v2_17.rggb,8449538,8416517 TEST/nikon_d4_10.rggb,15701302,15851157 TEST/nikon_d5200_14.rggb,20384199,20444366 TEST/olympus_epm2_16.rggb,13435899,13375579 TEST/olympus_om_d_e_m5_24.rggb,13260184,13345509 TEST/olympus_xz2_10.rggb,10397033,10394995 TEST/panasonic_lumix_dmc_gh3_10.rggb,14186056,14233905 TEST/panasonic_lumix_g5_15.rggb,14095164,14130656 TEST/pentax_k5_ii_12.rggb,18068348,18073325 TEST/pentax_q10_19.rggb,10692439,10398867 TEST/samsung_nx1000_19.rggb,14594902,14513184 TEST/samsung_nx20_01.rggb,15892754,15799060 TEST/sony_a55.rggb,13066285,13096163 TEST/sony_a77_08.rggb,17466963,17564793 TEST/sony_a99_04.rggb,15125553,15079356

[psykauze]

So... I've made some analysis on the RAW files and found there is also rotation to take account. The results after rotating images for getting the right CFA pattern is worse than before.

Here the new results: log-newRGGB.txt

[jonsneyers]

Maybe one image per camera model is not enough to really draw conclusions, but I wonder what is going on here. The differences in compression when changing the CFA pattern are not huge anyway, which suggests to me that doing the YIQ transform on R G1 B while storing 1+G2 in the 'alpha' channel is not the best approach -- it was just a quick & dirty hack anyway, to match RGGB with the RGBA functionality that was already there. Maybe a custom color transform for RGGB would be better.

[StephanBusch]

I would also vote for a custom color transformfor RGGB. At least for Sony there is an explanation: http://diglloyd.com/blog/2014/20140212_2-SonyA7-RawDigger-posterization.html

The Fuji CFA pattern is different from others - see wikipedia: https://en.wikipedia.org/wiki/Bayer_filter

[psykauze]

That's why I disqualified the fuji x e1.

Also, I would like to try the R, (G1+G2)/2, B, (G1-G2) algo.

You're right too concerning the R, G and B ranges, there's is correction factor for each color.

[jonsneyers]

Converting to R, (G1+G2)/2, B, (G1-G2) is not lossless: one bit is lost in the division by two and it cannot be recovered. But we could try to directly apply a modified variant of YIQ, maybe something like this: Y = R+B+G1+G2 I = R - B Q = R+B -G1-G2 Gdiff = G1-G2

I'm not sure if encoding G1-G2 is really more efficient than just encoding G1 or G2. The difference is expected to be closer to zero, but it needs an extra bit to be represented.

[psykauze]

There's no lost on R, (G1+G2)/2, B, (G1-G2) conversion if you check if numbers is odd or even:

X = (G1+G2)/2 (Always round to lower int like N.5 => N); W = G1-G2; Case W odd: G1 = X + W/2 ; G2 = X - W/2 ; Case W even: G1 = X + (W+1)/2 ; G2 = X - (W-1)/2;

You're right, difference need to be represented with an extra bit but I think this W extra bit can be moved to X in some cases (like if Xdec+Wdec/2 > 0xFFFF then that means Xdec = Xenc << 1 + extra_bit). I need some calculations to check if it is possible.

[jonsneyers]

What is Y in that case distinction?

[psykauze]

Y was W I've edited the post sorry.

[jonsneyers]

Ah, right, the parity of the sum is equal to the parity of the difference so there would indeed be enough information to restore the exact sum. I'll try that transformation.

Also I'm assuming that raw files have no more than 14 bpc, is that a valid assumption? It helps if we want to represent G1-G2 in an uint16_t...

[psykauze]

I haven't see yet sensors up to 14bits so I think this is a valid assumption.

Be carreful, (G1 - G2) is a signed value but maybe you want store the data as '(Diff << 1) + sign' ?

[jonsneyers]

I'm storing it as 0x4000 + G1 - G2, which is always > 0 if the numbers are at most 0x3fff.

But unfortunately, my preliminary results (just testing on a single image) seem to indicate that any manipulation on the R G1 G2 B values (sum, avg, diff) just makes compression worse.

[psykauze]

Could you send me your new image-rggb.cpp, please ? I would like to try some things with the Stephan set.

[jonsneyers]

I pushed it as commented-out code (only in the rggb loading part, not in the decode/save part),

[psykauze]

I've tried with R,mean(G),B,diff(G) with many hacks but it is still very worse (+20-50%) than before.

Also, I think I understand why the results was worse after we corrected the RGGB pattern. The red plane is about half the luminosity than it should (usualy corrected by RAW decoders) and blue plane is ~70%. So maybe the YIQ transform is not efficient in this case.

I will try this hack: R => Alpha plane, G2 => Red Plane, G1 => Green plane, B => Blue Plane and compare results.

[StephanBusch]

There are also Bayer sensors that capture in 16-Bit. For example Mamiya-Leaf and Phase One raw images and also Hasselblad raw images. I will upload some samples (converted with DCRAW) later.

[psykauze]

I made some (long) tests in interlaced mode with some hacks: 1) I've rotated and mirrored the greyscales for having the same (RGGB) pattern on every images. You should see the transformation in the filename. 2) I've disqualified the sigma and the finepix x-e1 pictures due to their attypic CFA pattern 3) Test named "before" is before modifications of image-rggb.cpp 3) Test "RGGB corrected" is just after finding there is an issue to the demosaicing algo in image-rggb.cpp 3) Test "RGGB+bpp": I've made a bit-depth detection before initialising the planes 4) Test "RGGB+invertG2": I've only inverted the alpha plane (representing G2) with "maxval - pixel" instead of "1 +pixel" 5) Test "bpp+invertG2": All corrections above 6) Tests "G2G1BR", "G2G1RB", "G2RBG1" and"G1RBG2": I've tried to change planes order to see what happening (G2RBG1 is like before the RGGB correction) 7) Test "all invert": All planes was inverted ("maxval - pixel")

image-rggb-patch.txt results-psykauze.txt

[jonsneyers]

I added these results to the spreadsheet over here: https://docs.google.com/spreadsheets/d/1LxY78fbm47VmrYGTXkBXXitGjhGl32NsuHPH2QXufgA/edit?usp=sharing I made the results a bit more visual using color.

The order of the RGGB planes makes surprisingly little difference (at most 3%, and typically less than 1%).

Is maxval-pixel safe? If the alpha plane value is zero, the other planes don't get encoded...

[psykauze]

maxval-pixel is as safe than 1+pixel, if "pixel=maxval" you will get a fully-transparent pixel.

Also, my bit-depth detection can fail if the maxval measured = 2^N

I've edited the bit-depth detection, so in case it measure maxval = 2^N it make the bit-depth 2^(N+1)

Edit.: Preliminary results show that my bit-depth detection make in some case, loss of data due to alpha plane clean-up. Also, after correction, this bit-depth detection does not help anything (loss of ~3bytes by files). (For now, image-rggb.cpp can lost datas due to alpha plane cleanup with 16bpp pictures maybe we should remove the alpha-cleanup when image-rggb is used.)

Inverting the alpha plane can be better in some cases but can be worse in others. With this set, "1+pixel" method is better of 1393bytes (over 304718028 bytes total). bitdepth-detection2.csv.txt

So, 1) Detecting the bit-depth don't help 2) Inverting the alpha plane don't help 3) Correcting the RGGB pattern as it should make worse compression than G2RBG1 previous method 4) There is a potential issue if "pixel" = maxval (I've set it to "2^16 -1") due to alpha-cleanup

[psykauze]

I think I understand why the plane ordering change the compression ratio.

R, Gs and B planes have not the same luminosity factor. Also G1 and G2 are just slighty different so YIQ transform with this 2 planes should be more efficient.

In natural environment, Red plane is more different of green planes than blue of green.

(For info: Red coefficient on RAW is 0,5, Green = 1, Blue = 0,7)

My results with ordering plane: plane-order.txt

My patch with some comments: image-rggb.txt

[StephanBusch]

16-bit RGGB

Phase One: https://drive.google.com/file/d/0ByLIAFlgldSoU2dyOXptOUxxdjg/view?usp=sharing Mamiya-Leaf: https://drive.google.com/file/d/0ByLIAFlgldSoTDB6b0NNOWxjWjQ/view?usp=sharing Hasselblad: https://drive.google.com/file/d/0ByLIAFlgldSoTk96NU4xTmlrTVU/view?usp=sharing

[jonsneyers]

The Phase One image seems to use 15 bits on R and B and 16 bits on G1 and G2. The Mamiya-Leaf image seems to use 13 bits on R and B and 14 bits on the greens. The Hasselblad image seems to use 15 bits on all colors. (I'm just looking at the number of distinct values per channel; the next power of two is what I guess the bit depth is)

So for now, putting 1+R at alpha is safe ;) but of course it would be cleaner to actually distinguish RGGB from RGBA. Probably best to add an extra template argument to the encoder and decoder to disable the A=0 branches, that would also improve the speed of encoding/decoding flat RGB images.

[psykauze]

I've tried to (hardly) deactivate the alpha-cleanup but "some" files fails in decoding. When the filesize is not the same than "before" that means the flif file is broken.

[jonsneyers]

Deactivating alpha=0 interpolation alone is not enough. There are also changes inside the inner loops. I'll try take a look at it tomorrow. Added a new issue for this, since it is also useful for RGB images (some small speedup to be expected) and RGBA images where you for some reason want to store the original RGB values at A=0.

[psykauze]

@StephanBusch The "Fujifilm x e1" CFA is [GRBGBR BGGRGG RGGBGG GBRGRB RGGBGG BGGRGG] I've tried to make a new preprocessor for this kind of CFA pattern but it could be difficult. Also because there is 20 Green pixels, 8 Blue pixels and 8 Red Pixels. So, for now you should compress it in greyscale mode (.pgm) instead of (.rggb).

Also, there is not yet an "Orientation" detection. FLIF assume that the top-left corner pixel is a red pixel, so you have to rotate the file manualy.

[jonsneyers]

Here are some results for those 3 huge test images @StephanBusch provided (thanks for the test images, btw!)

-rw-r----- 1 jon jon  80300323 Nov 15 21:44 20130624-CF043185.pgm
-rw-r--r-- 1 jon jon  45620822 Nov 15 23:35 20130624-CF043185.pgm.flif
-rw-r--r-- 1 jon jon  65960722 Nov 15 23:42 20130624-CF043185.pgm.png
lrwxrwxrwx 1 jon jon        21 Nov 15 23:56 20130624-CF043185.pgm.rggb -> 20130624-CF043185.pgm
-rw-r--r-- 1 jon jon  43346608 Nov 15 22:28 20130624-CF043185.pgm.rggb.flif
-rw-r--r-- 1 jon jon  43248365 Nov 16 00:14 20130624-CF043185.pgm.rggb.new.flif
-rw-r----- 1 jon jon 121768435 Nov 15 21:50 Hasselblad_H4D-60.pgm
-rw-r--r-- 1 jon jon  67266732 Nov 15 23:48 Hasselblad_H4D-60.pgm.flif
-rw-r--r-- 1 jon jon  98908643 Nov 15 23:49 Hasselblad_H4D-60.pgm.png
lrwxrwxrwx 1 jon jon        21 Nov 15 23:56 Hasselblad_H4D-60.pgm.rggb -> Hasselblad_H4D-60.pgm
-rw-r--r-- 1 jon jon  66591477 Nov 15 22:36 Hasselblad_H4D-60.pgm.rggb.flif
-rw-r--r-- 1 jon jon  66895047 Nov 16 00:21 Hasselblad_H4D-60.pgm.rggb.new.flif
-rw-r----- 1 jon jon 112008019 Nov 15 21:50 Wales_4_10_91.pgm
-rw-r--r-- 1 jon jon  60465878 Nov 15 23:53 Wales_4_10_91.pgm.flif
-rw-r--r-- 1 jon jon  86270819 Nov 15 23:50 Wales_4_10_91.pgm.png
lrwxrwxrwx 1 jon jon        17 Nov 15 23:56 Wales_4_10_91.pgm.rggb -> Wales_4_10_91.pgm
-rw-r--r-- 1 jon jon  58670752 Nov 15 22:35 Wales_4_10_91.pgm.rggb.flif
-rw-r--r-- 1 jon jon  58609200 Nov 16 00:18 Wales_4_10_91.pgm.rggb.new.flif

[psykauze]

It seems the RGGB preprocessor is not very efficient with the Hasselblad image. Is the CFA pattern "RGGB" ? Is the top-left pixel Red ?

[jonsneyers]

No idea. @StephanBusch do you have the metadata from the original raw files? Also, I would be interested to know what format those raw files used originally, and how big those files were.

[StephanBusch]

here are the original raw files:

Hasselblad: https://drive.google.com/file/d/0ByLIAFlgldSoUlUxWGtFVGJXTVE/view?usp=sharing Mamiya-Leaf: https://drive.google.com/file/d/0ByLIAFlgldSoaEJxeUE0WlVXQTA/view?usp=sharing Phase One: https://drive.google.com/file/d/0ByLIAFlgldSoUGxJaDJzU1BtRmc/view?usp=sharing

I can upload more Raw if you want Thank you for the info @JonSneyers and @psykauze

Can the compression of Sony raw also be improved - given the info about 11bit +7 bit architecture? http://diglloyd.com/blog/2014/20140212_2-SonyA7-RawDigger-posterization.html This compression scheme is used on all Sony raw by default

[StephanBusch]

the Sigma Raw do not use a color filter array but the Foveon X3 sensor. Should they also be compressed as pgm?

[psykauze]

For this kind of sensor (Foveon X3), no need to preprocess the data because each pixel is directly encoded as a RGB value. You should encode it as a ppm (P6). Encode it with other methods will not improve compression.

[psykauze]

So, after raw analysis, the Hasselblad and the Phaseone files are rotated by "270 CW" so they should rotated by 90° (convert -rotate 90 $1 $2) more for matching the right CFA Pattern.

The Mamiya file is horizontaly oriented so no need to change orientation.

Edit.: Results with the latest commit. 20130624-CF043185-90.rggb,43358378 (G1G2B-R is worse than BRG2-G1 method) Hasselblad_H4D-60-90.rggb,66848703 (G1G2B-R is worse than G1BG2-R method) Wales_4_10_91.rggb,58609200 1) Why reordering planes in RGB part (not alpha) can make a difference in Hasselblad? Maybe I should read more about YIQ transform. Also indoor and outdoor have different kind of luminosity. 2) Maybe implementing an automatic ordering plane that check histogram similarities could improve this method but it needs extra datas to store the plane order.

[psykauze]

After learning how the YIQ transform is made, I've made some new test with G1BG2-R ordering planes and as I expected compression improve. G1BG2-R method beat all the time the RG1B-G2 method (with this set)

Using R instead of B could be better in some case (peoples blue wears/jeans, blue signs,..?) Switching G1 with G2 should not significantly improve the compression.

Edit.: And Yes!, I have not forget the "--keep-alpha-zero" switch

[jonsneyers]

You can forget the --keep-alpha-zero switch, if the input file is .rggb it is set automatically :)

[psykauze]

Thank you for the info :)

Also, I made tests for @StephanBusch with both interlaced and non-interlaced coding

FLIF_0.1.6_G1BG2-R.csv.txt

[StephanBusch]

thank you @psykauze

[StephanBusch]

I get different results with FLIF 0.1.6 so I guess that G1BG2-R ordering planes you did is not yet implemented