Support for JPEG-compressed DNG files

[Beep6581]

Request from: https://discuss.pixls.us/t/compressed-dng-support/6370

Raw file: https://raw.pixls.us/getfile.php/2204/nice/Canon%20-%20EOS%205D%20Mark%20III%20-%2014bit%20(2.3471882640587).dng

The DNG file seems to come from Magic Lantern and uses JPEG compression.

-TIFF:ExifByteOrder=Little-endian (Intel, II)
-TIFF-IFD0:BitsPerSample=14
-TIFF-IFD0:Compression=JPEG
-TIFF-IFD0:Make=Canon
-TIFF-IFD0:Model=Canon EOS 5D Mark III
-TIFF-IFD0:Software=MLV App
-TIFF-IFD0:DNGVersion=1.4.0.0

[jedypod]

I would love to see support for LJ92 compressed "Cinema DNG" files in RawTherapee.

I have some more example files and some information which may help push this forward:

Compressed Cinema DNG are created by a few different cameras:

• Digital Bolex D16
• Canon DSLR Magic Lantern raw (converted to CinemaDNG/DNG)
• BlackMagic Cinema Cameras

When opening these DNG files in RawTherapee, it has problems decoding the file. See this screenshot:

There is some useful information on LJ92 DNG compression here: https://thndl.com/how-dng-compresses-raw-data-with-lossless-jpeg92.html

That post mentions a lj92 compression library that is open sourced as part of the MLVRawViewer project: https://bitbucket.org/baldand/mlrawviewer/src/master/liblj92/?at=master

I have also uploaded a few more example compressed cinema dng files, included below, as well as uploaded to https://raw.pixls.us/

cinemadng.tar.gz

I hope this helps! Thank you for your consideration!

[nvki]

I would like to +1 this request as I'm having the same issue opening .dng files from my Blackmagic Pocket Cinema Camera. In addition to jeypod's example files I can add a example file with a color checker: bmpcc_cinemadng

[Waveluke]

I am also seeking a resolution to this request, primarily for my Blackmagic Pocket Cinema Camera, and can volunteer to hunt for someone who can do it if I can get the info of: How much of a time commitment will this project be? What sort of specific expertises within the vast world of Computer Science/ Coding are needed?

Take note that I do not have any computer coding knowledge aside from basic and shaky web html, but an advanced understanding of digital photography from the photography side, and multiple social networks that I could potentially tap for talent to solve this issue.

[Waveluke]

Update: someone at Pixls gave me the skillset info needed to recruit someone to work on this, but I would like to know, besides Pixls and here at GitHub, would there be any other options for me to get the my recruit in contact with the developers?

[paperdigits]

Yes, #pixlsus or #rawtherapee on Freenode, which is IRC, a chat application.

[heckflosse]

I take this issue. I test with Blackmagic files atm.

For reference the version using adobe dng converter:

The original dng file has two tiles (onle left, one right). Here's a screenshot of my current mockup where the left tile already is decoded correctly. Don't care about the colours (matrix, white level...) atm. Decoding has to be done correctly first:

[Waveluke]

Thanks so much.

So excited for native support!

Thanks, Waveluke.

On Wed, Nov 7, 2018 at 11:49 AM Ingo Weyrich notifications@github.com wrote:

I take this issue. I test with Blackmagic files atm.

For reference the version using adobe dng converter:

[image: grafik] https://user-images.githubusercontent.com/13733487/48156519-dab0e800-e2cd-11e8-917c-c0b658429754.png

The original dng file has two tiles (onle left, one right). Here's a screenshot of my current mockup where the left tile already is decoded correctly. Don't care about the colours (matrix, white level...) atm. Decoding has to be done correctly first:

[image: grafik] https://user-images.githubusercontent.com/13733487/48156773-5b6fe400-e2ce-11e8-8a83-8917a1e11d82.png

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Beep6581/RawTherapee/issues/4285#issuecomment-436754379, or mute the thread https://github.com/notifications/unsubscribe-auth/AqvRAs1IJVcFnvqgtKzWv2cCV3ihu1o8ks5uszk2gaJpZM4RWoph .

[Waveluke]

I have the Blackmagic Pocket Cinema Camera on hand if anything else is needed, in terms of files, camera tests or other things.

[heckflosse]

Now the right tile is also decoded:

[heckflosse]

@Waveluke Can you provide files as described here? If you have access to a ColorChecker, please also provide shots as described here.

[heckflosse]

With last commit I get 1.5x speed for decoding of dng files with 2 tiles (blackmagic pocket...), and triple speed for decoding of dng files with 8 tiles (blackmagic 4k) measured on my 8-core (files loaded from SSD)

[heckflosse]

I added support for the Magic Lantern files. Screenshot shows the raw from first post:

[heckflosse]

@jedypod Can you provide files from Digital Bolex D16 ?

[heckflosse]

Using linearization table gives much better colors for blackmagic files:

[Thanatomanic]

@heckflosse I can try to get you a colour chart file from the black magic compact cinema. Would it matter that it wasn't shot under any controlled lighting or perfect exposure?

[heckflosse]

@Thanatomanic Don't know. @Beep6581 knows

[heckflosse]

With last commit the camera model is shown in info-dialog instead of Unknown for blackmagic cameras

[heckflosse]

Short status report:

I tested the latest code with all Canon DNG files I have (from Magic Lantern, Adobe DNG converter, Adobe Lightroom) as well as with all Blackmagic DNG files I have (straight out of camera and also converted using Adobe DNG converter). All decode correctly. Imho this issue can be set to 5.5 Milestone.

[Beep6581]

@Thanatomanic

I can try to get you a colour chart file from the black magic compact cinema. Would it matter that it wasn't shot under any controlled lighting or perfect exposure?

That would be good. There's a good chance that it would improve the situation. I can judge whether they're usable once I see the file.

[Thanatomanic]

How about these two? https://filebin.net/qgfqlh62gjsbls8j

[iliasg]

Ingo !!!

Nice .. only one problem .. when we convert the in_camera DNG to "AdobeDNG" with adobe dng converter .. RT does not recognize the colormatrix2 of the new file. The colors are as if no color matrix is used. BTW I am uploading in the above filebin the adobe converted files and the dcps I extracted from both the original and converted files. Looks like both the dng converted and dcp tool somehow change the color matrices .. and adobe's matrix gives more reasonable WB values than BlackMagic's ..

[heckflosse]

@iliasg

only one problem .. when we convert the in_camera DNG to "AdobeDNG" with adobe dng converter .. RT does not recognize the colormatrix2 of the new file

I didn't change that. Should be no difference for this case before/after changes

[iliasg]

Yes it's same (wrong) as before :)

[heckflosse]

@iliasg Decoding of native Blackmagic DNG files (without using Adone DNG converter) gives good results now imho. Do you agree that it's ready for 5.5 release?

[iliasg]

@heckflosse Ingo yes .. it's almost ready for release .. waiting for a proper dcp profile (I cannot build one from the avaible shots :( )

I was wrong for the "more reasonable WB values" coming from "adobe's matrix" I mentioned before .. the "more reasonable" WB values is the result of RT not detecting the colormatrix2 in the converted file ..

BTW I think the extracted dcp gives better colors when used (it's a mystery for me because it olny consists the matrices !!) .. should we bundle the above dcp also ?.

[heckflosse]

@iliasg Ilias, let me explain a bit more:

Blackmagic DNG files need to be processed fast (as there are a lot of them for one video). For that reason, native processing (without need to use Adobe DNG converter) is preferable to reduce processing time and storage space.

@Waveluke may correct me if my assumption is wrong

[Thanatomanic]

(I cannot build one from the avaible shots :( )

@iliasg Unfortunately, I don't have access to the camera anymore, and cannot help (producing better shots if needed).

[heckflosse]

@iliasg ping @Beep6581 for the dcp and the matrices. Let's wait for @Waveluke . He offered to provide images.

[Beep6581]

Since one of the shots was taken using fluorescent light, a good dual-illuminant D50/StdA profile is not possible. However, I made one just for testing. I uploaded both to the bin: https://filebin.net/qgfqlh62gjsbls8j

[Waveluke]

"Blackmagic DNG files need to be processed fast (as there are a lot of them for one video). For that reason, native processing (without need to use Adobe DNG converter) is preferable to reduce processing time and storage space."

You are correct with that, and I am just seeking native support for proper decoding of CDNG files from the Blackmagic Pocket Camera, in time for v. 5.5 release, and if the colors are inaccurate, then I can correct them myself, using RTs powerful color correcting tools until someone does a proper dual illuminant DCP.

I do not have a color checker, but I was considering buying one anyway. In the short term all I have is the camera, but no way of producing a color checker image, though I do have the knowhow to create color checker samples if I were to have a color checker.

[Waveluke]

Any other things besides color checker shots I can provide, however.

[iliasg]

@heckflosse Ingo, another feature which the cinemaDNG users will ask is to automatically crop at the default size .. here 1920x1080 .. RT does no use of the default origin/crop DNG tags .. we can either use camconst.json for this or improve RT's tag recognition and use ... what do you think ?.

[heckflosse]

@Waveluke Which OS are you using? In case you use Windows, I can ask for a build of the Blackmagic branch...

[heckflosse]

@iliasg Just set the raw border to 16 will be fine :) No need to use camconst.json for this

[heckflosse]

@iliasg will be fine for the blackmagic file I meant

[Waveluke]

Windows is correct. As to the concern about not autocropping to 1920 by 1080, I like having access to the extra pixels for a slight recompose. If a user is bothered by it, they can set a custom crop and save as a Blackmagic BMPCC processing profile, and load it whenever setting parameters for a BMPCC chain of stills.

[Waveluke]

Apparently there is also this raw border crop feature in the new Rawtherapee as well, I do not think I saw it in 5.4 stable release.

[Beep6581]

Another solution would be to use dynamic profiles. Once RT ships with some built-in dynamic profiles (#4488), handling the crop origin/size for the Blackmagic Pocket Cinema Camera would be trivial.

[heckflosse]

@Waveluke

Apparently there is also this raw border crop feature in the new Rawtherapee as well, I do not think I saw it in 5.4 stable release.

Yes, that's a new feature in > 5.4 builds. That's what I refered to mentioning the 16 pix raw border above.

[iliasg]

@iliasg Just set the raw border to 16 will be fine :) No need to use camconst.json for this

OK looks fine for the BM samples we have .. but ..

• I am sure there are cases where the border is not symmetric or maybe even the 16pixels is not enough ..
• It's a little faster if you start with the correct crop ;)

[Thanatomanic]

@Beep6581 See https://filebin.net/qgfqlh62gjsbls8j I've added jpegs of amsterdam.pef with your DCP applied in various ways, and also jpegs of the daylight color chart with camera default and DCP applied. Was that what you wanted?

[Waveluke]

Oh and the proper border for 1080 output for the BMPCC is 12 pixels not 16, at least the according to the misdecoded native BMPCC files and the properly decoded Adobe DNG converter files. For both instances I have seen a resolution of 1944 by 1104, and subtract 1920 by 1080 from both dimensions and you get 24 extra pixels of width, and 24 extra pixels of height. since there are two borders per dimension, that means each border should be 12 pixels.

However, if Adobe and the magenta artifact misinterpreted RT decode are BOTH throwing away the same number of pixels, then I stand corrected.

[iliasg]

Since one of the shots was taken using fluorescent light, a good dual-illuminant D50/StdA profile is not possible. However, I made one just for testing. I uploaded both to the bin: https://filebin.net/qgfqlh62gjsbls8j

@Beep6581 Fine !! .. do you have the spyder24 reference values available .. just to check a bit ..

[Waveluke]

And full pixel readout should always be an option, so that footage can be stabilized in the NLE of the users choice and require a less intense crop. Although I will say, center crop 540 by 960 upscaled to 1080p BMPCC footage raw developed in RT can look as sharp or sharper than native DSLR h264 captured full 1080p!

[Waveluke]

RT developed BMPCC 1080p has certainly put me in the 4k is overrated camp!

[iliasg]

Oh and the proper border for 1080 output for the BMPCC is 12 pixels not 16, at least the according to the misdecoded native BMPCC files and the properly decoded Adobe DNG converter files. For both instances I have seen a resolution of 1944 by 1104,

It's 1944 x 1104 after cropping of the default 4pix border .. it's 1952 x 1112 originally ;)

[heckflosse]

@Waveluke

[iliasg]

@heckflosse Ingo, where can I download the samples you use for your screenshots ?.

[heckflosse]

@iliasg raw.pixls.us

[Waveluke]

As I figured I might, I stand corrected. But that is good news, for video stabilization and extreme low light where using a marginally bigger sensor area trumps sharpness loss of resampling.