JPEG compression support in DNGs?

[TomArrow]

When I try to compress a dng I'm writing with JPEG, the resulting file is 1 KB big and thus obviously not working (the normal file is around 10 MB). Any ideas what I'm doing wrong? With disabled compression all works fine. ADOBE_DEFLATE also works but isn't very efficient sadly. I'm trying to get results close to what Adobe DNG Converter can do, which uses JPEG according to RawDigger. Lossless JPEG is what I'm after. Oh and if I did get it to work, how would I make the JPEG lossless? Or is it automatically?

Adobe DNG specification also says that lossless JPEG is one of the supported compression modes for DNG.

            using (Tiff output = Tiff.Open(fileName, "w"))
            {
                // Basic TIFF functionality
                output.SetField(TiffTag.IMAGEWIDTH, width);
                output.SetField(TiffTag.IMAGELENGTH, height);
                output.SetField(TiffTag.SAMPLESPERPIXEL, 1);

                    output.SetField(TiffTag.BITSPERSAMPLE, 16);

                output.SetField(TiffTag.ORIENTATION, Orientation.TOPLEFT);
                output.SetField(TiffTag.ROWSPERSTRIP, height);
                output.SetField(TiffTag.PHOTOMETRIC, Photometric.MINISBLACK);
                output.SetField(TiffTag.FILLORDER, FillOrder.MSB2LSB);

                    output.SetField(TiffTag.COMPRESSION, Compression.JPEG);

                output.SetField(TiffTag.PLANARCONFIG, PlanarConfig.CONTIG);

                float[] cam_xyz =
                {
                3.2404542f, -1.5371385f, -0.4985314f, -0.9692660f, 1.8760108f, 0.0415560f, 0.0556434f,
                -0.2040259f, 1.0572252f
            }; 
                float[] neutral = { 1f, 1f, 1f }; 
                short[] bayerpatterndimensions = { 2, 2 }; 

                //DNG 
                output.SetField(TiffTag.SUBFILETYPE, 0);
                output.SetField(TiffTag.MAKE, "blah");
                output.SetField(TiffTag.MODEL, "blah");
                output.SetField(TiffTag.SOFTWARE, "blah");
                output.SetField(TiffTag.DNGVERSION, "\x1\x4\x0\x0");
                output.SetField(TiffTag.DNGBACKWARDVERSION, "\x1\x4\x0\x0");
                output.SetField(TiffTag.UNIQUECAMERAMODEL, "blah");
                output.SetField(TiffTag.COLORMATRIX1, 9, cam_xyz);
                output.SetField(TiffTag.ASSHOTNEUTRAL, 3, neutral);
                output.SetField(TiffTag.CALIBRATIONILLUMINANT1, 21);
                output.SetField(TiffTag.ORIGINALRAWFILENAME, sourceFileNameForTIFFTag);
                output.SetField(TiffTag.PHOTOMETRIC, 32803);
                output.SetField(TiffTag.SAMPLESPERPIXEL, 1);
                output.SetField(TiffTag.EXIF_CFAPATTERN, 4, bayerPatternTag);
                output.SetField(TIFFTAG_CFAREPEATPATTERNDIM, bayerpatterndimensions);
                output.SetField(TIFFTAG_CFAPATTERN,
                    bayerPatternTag); //0=Red, 1=Green,   2=Blue,   3=Cyan,   4=Magenta,   5=Yellow,   and   6=White

                output.WriteEncodedStrip(0, rawImageData, rawImageData.Length);

Thanks in advance.

[TomArrow]

I have now tried to read a dng file created with Adobe DNG Converter too, and it fails at the ReadEncodedStrip stage in the subIFD that actually contains the raw data. The function returns -1.

[shibaev]

Thank you for the report! Could you please send us DNG files to reproduce both issues? You can send them to support@bitmiracle.com

[TomArrow]

Yes, gladly, thanks for taking the time. I've sent an email containing a MEGA link to two uncompressed and two Adobe DNG Converter created dng files along with a quick console app to test all the things I talked about.

Now that I've written it in a console app, I can already see a few errors pop up that I didn't notice in the WPF app. It says that 16 bits of depth are not supported for the JPEG compression. I have looked a bit more into the TIFF/DNG specifications and it says that the JPEG algorithm to be used for >8 bits must be lossless:

https://www.adobe.com/content/dam/acom/en/products/photoshop/pdfs/dng_spec_1.4.0.0.pdf (page 19)

This document says it should be lossless sequential DPCM using Huffman: https://web.archive.org/web/20060212163025/http://www.map.tu.chiba-u.ac.jp/IEC/100/TA2/recdoc/N4378.pdf (page 15)

Other than that, I am getting an error that I can't read strips from a tiled image. However I still decided to send the example project because I think the lossless JPEG might not yet be implemented at all even if I used tiles. However I'd be glad to be proven wrong.

[TomArrow]

I've now tried to do some tile-reading (not sure if entirely correctly) and I get this error message in the output: JPEGLib: Unsupported JPEG process: SOF type 0xC3

I do recall reading that the normal libjpeg hasn't implemented some of the JPEG variants due to some patent issues or similar, though I might have misunderstood that. I read that this variation of libjpeg is a complete implementation with all the variants: https://github.com/thorfdbg/libjpeg

But I can't really confirm it.

[TomArrow]

According to this place: https://de.wikipedia.org/wiki/JPEG_File_Interchange_Format SOF C3 might simply be SOF3

This library here says it can decode SOF3: https://github.com/yigolden/JpegLibrary

Maybe LibTiff.NET could use JpegLibrary for this task? However it doesn't seem to be able to do any encoding in SOF3 either.

[shibaev]

You are correct in your observations. Unfortunately, we are limited by restrictions of LibJpeg.Net, which are based on libjpeg 6. As you already found, LibJpeg.Net (and libjpeg 6) does not support lossless JPEG:

• encoding with 16 bits per sample
• decoding of SOF3 data

Currently, we are not interested in migrating of LibTiff.Net to libjpeg 7 or other JPEG libraries.

You can implement your version of TiffCodec using an alternative JPEG encoder/decoder. Look at JpegCodec for example.