Cinema DNG export

[bouncyball-git]

I remember a while ago this topic been discussed on the forum and conclusion was that: first of all MLV App is the processing software (I fully agree with this statement) and let's leave exporting DNGs to other dedicated software. But...

In reality there IS NO cross platform GUI based cute proggie which will do this task for many who can't fiddle with cli and mlv_dump (MLVFS is another story). There is Danne's very cool script based (no full blown gui) project but it also not fully cross platform yet.

So... I thought that because of my quite good experience in this particular cDNG field why not implement this to our lovely MLV App, besides half of the job is already done (llrawproc) ;). I think I can do this quite easily.

What do you think?

[masc4ii]

Yes, you are right. MLVFS is the only good working tool (I love it), when using something GUI based. There are other small crossplatform GUI tools like MLV-Mystic I often used in the past on WIn&OSX. So for me this is still now not a must have feature. But I don't want to say we never should do that. Today it makes already a bit more sense, since we have the RAW corrections implemented. This is not so far away from the functions MLVFS offers. And good to know, that you have good experience in the cdng field! Nice! On the other hand I would like to finish our planned features for MLV App as processing tool first - I mean caching, wbPicker, working audio, spanning mlv support, ffmpeg pipe, AVFoundation on OSX,... I think, one day we will do it! :-) Good, that you wrote this enhancement issue, so the idea don't gets lost. What thinks our boss @ilia3101 ? ;-)

[bouncyball-git]

Let's wait for his opinion :)

You are right, there are so many things to do that the priority of this enhancement is surely low. I think mlv mystic is long time abandoned project. Actually there are 3 projects deserving attention which offer GUI and exporting DNG/cDNG:

1. mlrawviewer (abandoned needs update)
2. raw2cdng (windows only, don't know about compressed mlv support)
3. mlvp (windows only) others are old and abandoned ones.

My solution (mlv_dump on steroids) offers exporting of real cinema dngs of any bit depth (2-16) including cDNGs compressed with lossless compression. None of other solutions do this at the moment :)

No problem with time I will do it slowly bit by bit. May be I'll wrap all into cdngexport class on c++.

[ilia3101]

What thinks our boss Makes me feel very important 😃

Let's wait for his opinion :) Of course I'm all for it :) If you can easily make a CDNG export module, that is great

In reality there IS NO cross platform GUI based cute proggie which will do this task for many who can't fiddle with cli and mlv_dump Totally agree - Me and many people would appreciate if you did such a thing. (there is no hurry of course)

Speaking of GUIs for exporting DNGs, I also did make one over recent weeks, a wrapper for mlv_dump, (simply to learn how to use Apple's tableview widget before I implement sessions in Cocoa GUI) infact it looks a little bit similar to that app that violated the GPL :) (maybe I'll release it, just to save a couple people from using that outdated thing)

[masc4ii]

I prepared CDNG export from GUI side. Commit

I did nothing for audio export yet. Do we also need the option from MLVFS: file name: normal/davinci resolve? From the GUI you get the framenumber and a filename "folder.cdng". Delete the ".cdng" and put the cdngs inside (if folder does not exist yet, build it)! ;-)

Folder selection, CDNG Export selection, progressbar and abort works already.

[bouncyball-git]

Cool man! I hope I'll have more time to do it in a good/pretty way soon and this enhancement not gonna share the destiny of the WB picker ;)

[bouncyball-git]

As for naming scheme, think it would be good to have both.

[masc4ii]

So I would use the option combobox in the preferences dialog to switch between these two options. You would get another bool variable as parameter.

Yes, it is a bit sad, that wbPicker is not ready yet... but it is hobby and no commercial app... ;-)

Edit: Done. But it is a integer 0/1.

[bouncyball-git]

OK :) thx

[bouncyball-git]

@masc4ii

Hey man do me a favor, I don't wanna mess with 2 (Davinci, Normal) filename schemes in the dng lib itself can you get it sorted out in 'mainwindow.cpp' and supply me with the constructed (path/name) names for both according to 'm_codecOption'? I would so appreciate this :)

[masc4ii]

Haha. And I thought I don't have to understand how the Davinci Naming Scheme is working :-P No problem, I can do that. But could you please tell me how the name is built? There is a ...C0000... and other "nice" things in it. Some of them (especially the C0000) I don't understand... are there rules?

[bouncyball-git]

Yeah... I never used this scheme myself in MLVFS. Here is the definition made by dfort.

Maybe we could just propose our own scheme? For example:

CAMNAME_year_month_day_minutes_secondsNNNNN.dng CAMID yearmonthdayminutesseconds_NNNNN.dng

or just stick to the MLVNAME_NNNNN.dng ;)

[bouncyball-git]

If we stick to the MLVNAME_NNNNN.dng then we could just made

Profile: Cinema DNG

Oprions:

1. Uncompressed
2. Losless
3. Pass original

[bouncyball-git]

And I think the later one it the way to go ;) sorry about making you redo things lot of times. I just need your opinion. Are there lot of peaople who's gonna miss the blackmagic naming scheme? Just for showing the dirs of the DNG as a clips in the davinci resolve?

[masc4ii]

Yes, I would like it too. That is the easiest. I always used Resolve Naming Scheme in MLVFS. So I can't tell what may not work when not doing so... ;-)

[masc4ii]

But we can also leave it as it is and I implement this naming scheme is I always saw it. MLVNAME_1_YEAR-MONTH-DAY_0001_C0000_FRAME.dng

[bouncyball-git]

I've done 75% of dng export already. Adapting it to MLV App turned out to be more work than I thought :P

[bouncyball-git]

OK then! Let's do as you suggest :)

[bouncyball-git]

Default: MLVNAME_NNNNN.dng DR: MLVNAME_01_YEAR_MONTH_DAY_0001_C0000_NNNNN.dng

MLVNAME without MLV et the end, it is neeed by MLVFS code to do the job in it's code.

what about dir names and path to where they will be created?

[masc4ii]

But inside the dngs... is there a metadata "real name"? Because Davinci knows always how my MLV name was before! For the example, it tells "M21-2109". Folder for an example: M21-2109MLV_1_2017-07-21_0001_C0000 DNGs (number 11): M21-2109MLV_1_2017-07-21_0001_C0000_000011.dng Wave: M21-2109MLV_1_2017-07-21_0001_C0000.wav

The folder will be created where the user told in the export folder dialog.

[bouncyball-git]

Yes reel name is the MLV name derived from m_pMlvObject->path. Let's ommit 'MLV' at the end of mlvname:

Folder for an example: M21-2109MLV_1_2017-07-21_0001_C0000

M21-2109_1_2017-07-21_0001_C0000

DNGs (number 11): M21-2109MLV_1_2017-07-21_0001_C0000_000011.dng

M21-2109_1_2017-07-21_0001_C0000_000011.dng

Wave: M21-2109MLV_1_2017-07-21_0001_C0000.wav

M21-2109_1_2017-07-21_0001_C0000.wav

[masc4ii]

Is done.

[bouncyball-git]

Thank you man! Stay tuned :)

[bouncyball-git]

First version for testing is here.

I was coding it blindly two days without compilation/check so bugs are there ;). Yo can check console for some info output.

Uncompressed DNG export is working compressed and pass through have issues.:

1. Compressed case: compresses something and calculates compression ratios to the console, I guess some minor bug/typo will correct this.
2. Pass through case: crashes wildly :P, some illegal memory access I think.

I'm quite tired today, will trace the bugs maybe tomorrow.

[masc4ii]

For me that works. I had a little bug in my export action - maybe you got wrong file names? For me all three options create dngs without crash. But 1. and 3. is very very dark, and 2. is completely black. The output from MLVFS looks different - is a little pinkish and OSX Finder preview can show the images already. The only proggi on my small mac was LynApp which could open a dng. Also Adobe programs could not open... is this normal?

[bouncyball-git]

I have to trace all of this with exiftool and debug.

[masc4ii]

Tested a bit more. MLRV and Lightroom can handle the uncompressed files! I don't find a app which can handle the lossless files.

[bouncyball-git]

Losless files produced by my mlv_dump on steroids (uses the same lib modded by me) works in any software I used. This is some bugs I missed here. It's just that I rewrote a big portion of the code to adapt it to the MLV App.

Honestly this newer code looks lot more prettier than used in steroid version because implementing it to mlv_dump needed all kind of hoop jumping and more :P which is unnecessary for MLV App, here all is straightforward.

[bouncyball-git]

Hey guys I think the Cinema DNG export now on production level! Test it please.

I eliminated mistakes I did not noticed yesterday. All three dng export modes are working as expected for me (nailed pass through mode bug causing crash on Linux and fixed it too).

[masc4ii]

Works brilliant on Mac! If I have to find something... something really unimportant: in the settings dialog, write the options all small, or with a beginning big letter... ("Fast Pass"). But the rest: it works really good, no crash. Lossless works best for me, because the dngs can be viewed using OSX previewer (using space). This is like it is with MLVFS. Great job!

[bouncyball-git]

Thank you :-D

You are allowed to change the option names in the settings dialog according your desire ;)

I dig how fast pass mode works on SSD :P. BTW those dngs also should show up in finder without problem. IDK why uncompressed ones are not.

DNG spec defines little endian byte order for 8, 16, 32 bit dngs and big endian byte order for all other bit depths. MLVFS writes pseudo 16bit DNGs. I'll explain: they are 16bit, little endian but actually the data values are no more then 14bit. e.g. the 10 bit data (0-1023) represented as 16bit words and 6 bits are wasted for nothing. But Mac os shows those in finder without of problem.

I'll do some 16bit dng creation fixes for dual iso because 20bit dual isos are converted to real 16bit values and it is required to writhe to the real 16bit DNG.

You can try exiftool on all produced dngs and see that all is ok in the header metadata. i's just finder does not like lo bit big endian dngs without thumbnail in the dng file, which is our case.

[bouncyball-git]

Thank you for your great help from the QT side :)

[masc4ii]

I found one small bug, but it is not cdng only... it is for all exports: If I uncheck focus pixels, switch clip in session and switch back to first clip: the focus dots are there. Also when Opening a session with unchecked focus dots. Okay, fine. Then, I export the clip. The option is still unchecked, but the dots are away - in the exported clip and in the viewer, until I check and again uncheck the option. Do you have an idea why? Maybe that is for all RAW corrections like that?

[bouncyball-git]

Hehe I'll try to reproduce this tomorrow I'm far from my devel PC right now :)

[masc4ii]

Found it! ;-) My fault. But now I saw the bug when exporting... hehe. Will fix it. EDIT: Done.

[masc4ii]

One more issue on CDNG export: there is some metadata missing in the WAV. So Davinci interprets the folder as 2 clips, one video and one audio. In MLVFS it is video+audio at once. I analysed the metadata with MediaInfo: Our WAV:

General
Complete name                            : /Users/masc/Desktop/M19-1000_1_2017-11-19_0001_C0000a.wav
Format                                   : Wave
File size                                : 4.41 MiB
Duration                                 : 24 s 80 ms
Overall bit rate mode                    : Constant
Overall bit rate                         : 1 536 kb/s
IsTruncated                              : Yes

Audio
Format                                   : PCM
Format settings                          : Little / Signed
Codec ID                                 : 1
Duration                                 : 24 s 80 ms
Bit rate mode                            : Constant
Bit rate                                 : 1 536 kb/s
Channel(s)                               : 2 channels
Sampling rate                            : 48.0 kHz
Bit depth                                : 16 bits
Stream size                              : 4.41 MiB (100%)

MLVFS WAV:

General
Complete name                            : /Users/masc/Desktop/M19-1000_1_2017-11-19_0001_C0000b.wav
Format                                   : Wave
File size                                : 4.41 MiB
Duration                                 : 24 s 80 ms
Overall bit rate mode                    : Constant
Overall bit rate                         : 1 537 kb/s
Producer                                 : Canon EOS 5D Mark II
Encoded date                             : 2017:10:19 10:00:15
IsTruncated                              : Yes
Producer_Reference                       : JPCAN-21474831122591E26410001547

Audio
Format                                   : PCM
Format settings                          : Little / Signed
Codec ID                                 : 1
Duration                                 : 24 s 80 ms
Bit rate mode                            : Constant
Bit rate                                 : 1 536 kb/s
Channel(s)                               : 2 channels
Sampling rate                            : 48.0 kHz
Bit depth                                : 16 bits
Stream size                              : 4.41 MiB (100%)

Will try to find out, how to add this metadata.

[masc4ii]

What we need is a BWF chunk including the missing information. ATM I have no idea how to get that :-( EDIT: mlv_dump does it the right way. @bouncyball-git : can we copy here?! ^^

[masc4ii]

Found it out watching in the code of mlv_dump. Will test a bit and commit then! Seems to work ;-)

[masc4ii]

I commited a version. The meta data is now identical (only the "IsTruncated = yes" is missing). Filesize of the WAV is identical to the MLVFS version. But it does still not work in Davinci :-( If I just copy a mlv_dump wav or the MLVFS wav in the folder it works. Can it be the IsTruncated? What t ** is that?

General
Complete name                            : /Users/masc/Desktop/M19-1000_1_2017-11-19_0001_C0000.wav
Format                                   : Wave
File size                                : 4.41 MiB
Duration                                 : 24 s 80 ms
Overall bit rate mode                    : Constant
Overall bit rate                         : 1 537 kb/s
Producer                                 : Canon EOS 5D Mark II
Encoded date                             : 2017:11:19 10:00:15
Producer_Reference                       : JPCAN-21474831122591E26410001540

Audio
Format                                   : PCM
Format settings                          : Little / Signed
Codec ID                                 : 1
Duration                                 : 24 s 80 ms
Bit rate mode                            : Constant
Bit rate                                 : 1 536 kb/s
Channel(s)                               : 2 channels
Sampling rate                            : 48.0 kHz
Bit depth                                : 16 bits
Stream size                              : 4.41 MiB (100%)

[bouncyball-git]

I know I had this on my list it is not related to DNG itself only wav header if you need I can implement it today.

[bouncyball-git]

I already did this in mlv_dump on steroids :) long time ago.

[masc4ii]

Maybe you could have a look... somewhere must be an error in my implementation. I copied over the structs from mlv_dump and filled it in the (more or less) same way...

[bouncyball-git]

Or you can look yourself

https://bitbucket.org/bouncyball/magic-lantern/commits/f66204c9aea1d6405644c9987debae9154e0f911?at=mlv_dump_cdng

[bouncyball-git]

Here is my wav.h

[bouncyball-git]

Ok I'll take a look

[masc4ii]

OH MY GOD! I think now it works. This XML thing was missing. But the "IsTruncated = yes" is still missing. :-D

[bouncyball-git]

IsTruncated = yes ?

[masc4ii]

Look some posts above... MediaInfo tells that in the meta data file information. But I have no clue what it is. https://mediaarea.net/en/MediaInfo MLVFS WAV:

General
Complete name                            : /Users/masc/Desktop/M19-1000_1_2017-11-19_0001_C0000b.wav
Format                                   : Wave
File size                                : 4.41 MiB
Duration                                 : 24 s 80 ms
Overall bit rate mode                    : Constant
Overall bit rate                         : 1 537 kb/s
Producer                                 : Canon EOS 5D Mark II
Encoded date                             : 2017:10:19 10:00:15
IsTruncated                              : Yes
Producer_Reference                       : JPCAN-21474831122591E26410001547

Audio
Format                                   : PCM
Format settings                          : Little / Signed
Codec ID                                 : 1
Duration                                 : 24 s 80 ms
Bit rate mode                            : Constant
Bit rate                                 : 1 536 kb/s
Channel(s)                               : 2 channels
Sampling rate                            : 48.0 kHz
Bit depth                                : 16 bits
Stream size                              : 4.41 MiB (100%)

Ours is now identical, only the IsTruncated is missing.

[bouncyball-git]

I found this on mediainfo forum:

i can see an information : IsTruncated mentionned as "Yes" Can you tell me what this is about ?

It means the the file has not the expected size i.e. the copy was not complete. But this is not really a reason for unsync (it is expected that both video and audio stop before the end, >at the same moment).

I guess It is good for us that IsTruncated is not shown by mediainfo ;)

[masc4ii]

Haha... so this is the first magic lantern tool, which makes it the right way? Okay... I'll stop searching for (non existing) errors :-D

[bouncyball-git]

Dual iso DNG export is broken. Only preview mode is working. It's is because of the 20bit processing produces real 16bit values and I did not took this in account. DNGs are pitch black, I have to fix it :)

Edit: only pass mode is working as expected.

I never implemented the dual iso stuff from MLVFS to anywhere except of MLV App ;) This was my 1st dual iso experience.