Have you considered writing a darktable module?

[pedrocr]

Hi,

I'm one of the darktable[1] developers and came across this project when a user asked us to look into some x3f_extract written DNG files. I've had a look at the code and was wondering if you've ever considered writing a darktable module to process the data in these files? rawspeed[2] already supports these files and if we load them in darktable as RGB raw they already produce visible output. It's just not very good because the colors are muted/off. I have on my TODO list to port this dcraw code into a darktable module:

https://github.com/pedrocr/dcraw/blob/master/dcraw.c#L3033-L3735

which I assumed would fix the color issue and produce good output. I haven't looked extensively into the x3f code but you probably have something similar. darktable has a series of modules that work sequentially to process an image so adding a x3f_color one would fit right in:

https://github.com/darktable-org/darktable/tree/master/src/iop

That would make these files work like any others in the darktable workflow. We've done similar things in the past. We already have "scalepixels" to handle the non-square pixels in the D1X sensor and "rotatepixels" to handle the old rotated pixel sensors from fuji.

If you'd like to discuss this further we're very active in #darktable on freenode.

Cheers,

Pedro

[1] https://github.com/darktable-org/darktable [2] https://github.com/klauspost/rawspeed

[mmroden]

A very intriguing idea-- we wanted to make a lightroom plugin, so I think we'd be very amenable to this. A few questions:

• How straightforward is writing a plugin for new image formats for darktable? Is there documentation?
• The converter requires denoising to produce anything useful. Erik submitted the noise reduction he implemented as part of this project to OpenCV, and that's a large dependency. Would using OpenCV be an option, or will the noise reduction need to be redone? The noise reduction algorithm is a 16bit color version of Antonio Buades' NLMeans algorithm; I can send you a paper if you're interested.
• This converter is light on metadata; I assume any plugin we write would need to put metadata into some appropriate structure. I'm also hoping it'd be covered in whatever documentation, but it's a significant enough hole that I think it's worth calling out.
• Can the converter just go to XYZ, and then have the rest of the program (ie, darktable) handle other colorspace issues?

Thanks, Mark

On Saturday, February 6, 2016, Pedro Côrte-Real notifications@github.com wrote:

Hi,

I'm one of the darktable[1] developers and came across this project when a user asked us to look into some x3f_extract written DNG files. I've had a look at the code and was wondering if you've ever considered writing a darktable module to process the data in these files? rawspeed[2] already supports these files and if we load them in darktable as RGB raw they already produce visible output. It's just not very good because the colors are muted/off. I have on my TODO list to port this dcraw code into a darktable module:

https://github.com/pedrocr/dcraw/blob/master/dcraw.c#L3033-L3735

which I assumed would fix the color issue and produce good output. I haven't looked extensively into the x3f code but you probably have something similar. darktable has a series of modules that work sequentially to process an image so adding a x3f_color one would fit right in:

https://github.com/darktable-org/darktable/tree/master/src/iop

That would make these files work like any others in the darktable workflow. We've done similar things in the past. We already have "scalepixels" to handle the non-square pixels in the D1X sensor and "rotatepixels" to handle the old rotated pixel sensors from fuji.

If you'd like to discuss this further we're very active in #darktable on freenode.

Cheers,

Pedro

[1] https://github.com/darktable-org/darktable [2] https://github.com/klauspost/rawspeed

— Reply to this email directly or view it on GitHub https://github.com/Kalpanika/x3f/issues/78.

[pedrocr]

Hey @mmroden, all the issues you point out sound straightforward:

• New iops are pretty straightforward to write. They're a single C file that implements a set of functions that get called by the pipe. There's a dummy iop that explains how that works (https://github.com/darktable-org/darktable/blob/master/src/iop/useless.c)
• We already have a NLM module so maybe that can be used instead of OpenCV. Adding an OpenCV dependency would need to be discussed with the rest of the team but it should be easy enough to make it an optional dependency and not build the module if it's not available.
• I'm not exactly sure what you mean by the metadata. On image load we get whatever metadata we need from the file with rawspeed and exiv2 and that becomes available throughout the pipeline and gets put into the output image on export. This probably requires exiv2 work as it doesn't currently read the files. exiftool works though so the format is already understood.
• The pipe does color conversions already so it's easy to make the module output some form of RGB (XYZ would be fine) and then everything gets handled.

Thanks for considering this,

Pedro

PS: We've already made the x3f_extract DNGs work in darktable. The git versions (master and darktable-2.0.x) now load them without issues but they start as very red. The reason for that seems to be that you're tagging the WB values in the metadata but they're already applied to the image. Disabling the darktable WB module makes the image look good.

[rolkar]

An alternative route would be to update the Darktable DNG plug in so that it can handle Linear RAW. That seems like a much easier task. It is also a generally good idea for Darktable.

Then, our converter could be used in the work flow, in a similar way it is planned for Lightroom.

[pedrocr]

@rolkar, darktable since yesterday works fine with your DNGs. It was a minor issue with the way rawspeed reads SubIFDs (you use type 13 for the offset and not all the code was updated to support that). By default the images show up very red but that's because you've encoded a WhiteLevel while also applying the values to the data. Once you disable the whitebalance module in darktable the image loads just fine.

Using DNGs is still a much clumsier workflow though as it requires creating these huge files to then load them into the program. We already have all that's needed to load the RGB data from the .X3F files so it should be a simple integration to have an additional iop just to make the colors look nice. That way X3F files would load just like any other CR2/NEF/ARW/ORF/RW2/RAF/etc.

[mmroden]

Hi Pedro,

Is the noise reduction a full 16 bit implementation, or is it 8bit? The increased precision is necessary for the conversion, and takes a couple of seconds to run, at least in opencv on the cpu.

My C isn't as strong as it once was (and as Roland and Erik can tell you, never really up to snuff), so if I were to do this, it would be pretty clumsy and need a lot of review.

Thanks, Mark

On Sunday, February 7, 2016, Pedro Côrte-Real notifications@github.com wrote:

@rolkar https://github.com/rolkar, darktable since yesterday works fine with your DNGs. It was a minor issue with the way rawspeed reads SubIFDs (you use type 13 for the offset and not all the code was updated to support that). By default the images show up very red but that's because you've encoded a WhiteLevel while also applying the values to the data. Once you disable the whitebalance module in darktable the image loads just fine.

Using DNGs is still a much clumsier workflow though as it requires creating these huge files to then load them into the program. We already have all that's needed to load the RGB data from the .X3F files so it should be a simple integration to have an additional iop just to make the colors look nice. That way X3F files would load just like any other CR2/NEF/ARW/ORF/RW2/RAF/etc.

— Reply to this email directly or view it on GitHub https://github.com/Kalpanika/x3f/issues/78#issuecomment-181048053.

[pedrocr]

@mmroden the darktable pipe is all 32bit float from start to end. It only gets downconverted to 16 or 8bit (or left at 32) for final output. A lot of the operations also have SSE and/or OpenCL support to make them faster.

[erikrk]

As far as I understand, the DNG files are in compliance with version 1.4.0.0 of the DNG specification. They are at least rendering correctly with Adobe software and they pass the DNG validation utility supplied with Adobe's SDK (dng_valdate.exe of dng_sdk_1_4.zip).

It is true that white balance correction has been applied both to the data and to WhileLevel. It has to be done that way. White balance has to be applied to the data for chroma denoising to work and WhileLevel has to be where the actual saturation occurs for correct handling of overexposed channels (WhileLevel is not intended for while balance correction but rather for indicating where clipping occurs). Applying WhiteLevel thus regenerates the uncorrected data. White balace correction then has to be applied using CameraCalibration1 and AsShotNeutral.

On dom, 2016-02-07 at 08:52 -0800, Pedro Côrte-Real wrote:

@rolkar, darktable since yesterday works fine with your DNGs. It was a minor issue with the way rawspeed reads SubIFDs (you use type 13 for the offset and not all the code was updated to support that). By default the images show up very red but that's because you've encoded a WhiteLevel while also applying the values to the data. Once you disable the whitebalance module in darktable the image loads just fine.

Using DNGs is still a much clumsier workflow though as it requires creating these huge files to then load them into the program. We already have all that's needed to load the RGB data from the .X3F files so it should be a simple integration to have an additional iop just to make the colors look nice. That way X3F files would load just like any other CR2/NEF/ARW/ORF/RW2/RAF/etc.

— Reply to this email directly or view it on GitHub.

[pedrocr]

Hi @erikrk. I was misremembering the tags. AsShotNeutral is indeed what we use, we don't use CameraCalibration1 though for WB, only for color conversion. Basically the AsShotNeutral coeffs are multiplied for WB and then whichever matrix is the D65 one (or the first one that is available if none are D65) gets used for color conversion. If the WB coeffs are already applied to the data just disabling the WB module naturally works fine. The DNG spec has a more complex WB application method but from what I understand it requires two matrices with two illuminants and interpolating between then, and that you don't provide. I'm far from an expert in this stuff though so there may be stuff I'm missing.

[erikrk]

Providing two matrices to interpolate between is possible, but it's not a requirement. We only provide one matrix and that should work perfectly fine.

The WB coefficients have technically been applied to the data, but division by WhiteLevel should undo that. Is that also disabled when the WB module is disabled (WhiteLevel is is principle not rated to WB)? Is it true that the colors are good with the WB module disabled and bad with the WB module enabled?

On dom, 2016-02-07 at 15:41 -0800, Pedro Côrte-Real wrote:

Hi @erikrk. I was misremembering the tags. AsShotNeutral is indeed what we use, we don't use CameraCalibration1 though for WB, only for color conversion. Basically the AsShotNeutral coeffs are multiplied for WB and then whichever matrix is the D65 one (or the first one that is available if none are D65) gets used for color conversion. If the WB coeffs are already applied to the data just disabling the WB module naturally works fine. The DNG spec has a more complex WB application method but from what I understand it requires two matrices with two illuminants and interpolating between then, and that you don't provide. I'm far from an expert in this stuff though so there may be stuff I'm missing.

— Reply to this email directly or view it on GitHub.

[pedrocr]

@erikrk the colors seem fine with WB disabled, although I don't have a good reference for what the file should look like.

[mmroden]

Hi Pedro-- I've been thinking about a darktable module for a while now :) We just released a wrapper that copies over exif information from the x3f into a dng, but still, nothing's working as we would expect.

Does darktable support linear DNGs? Because that would be a quick stopgap to rewriting this as a plugin. Right now, I'm getting errors with respect to the white balance not being importable-- should we be setting the white balance to a particular setting on export (ie, AsShotNeutral, as suggested above)?

[pedrocr]

Not sure what you mean by linear DNG. Practically all raw formats, including DNG encode linear data. But there are some DNG encodings that rawspeed doesn't support which is probably what you're hitting upon. If you want to interoperate without integration writing a TIFF is probably your best bet. A full module is a much better long term bet. If you want to go that route I suggest showing up on IRC and speaking to the people there. I've stopped working on darktable a while ago so can't be of much help these days.

[rolkar]

http://wwwimages.adobe.com/content/dam/Adobe/en/products/photoshop/pdfs/dng_spec_1.4.0.0.pdf

There is a tag, called PhotometricInterpretation that can have the values CFA or LinerRaw. The latter means that you have no CFA filter, but instead three complete channels. That it is called LinearRaw is just a misnomer IMHO, as CFA (e.g. Bayer) also is Raw and Linear. This name has caused lots of misunderstandings.

Many softwares lack the support for LinearRaw. If Darketable had that support (which is quite reasonable IMHO) then no plug in would be needed.

[pedrocr]

Adding support for that in rawspeed (not in darktable) is probably easy but having it be integrated would make for a better workflow without having to generate intermediate files that are just processed versions of the originals.

[rolkar]

I fully agree on that.

But ... it is a matter of work and return. As the X3F format is secret and also quite confused, changing unpredictable between cameras, it is quite difficult to support it. And, Sigma cameras have very limited use base. It might be too much work to make the plug in.

Not saying it is a bad idea if someone finds the energy and time.

One question. Do a Darktable plugin have access to color space conversion, that can handle clipping nicely? Because, we have not implemented that. Thus our TIFF output might look strange.

[pedrocr]

The format itself is already supported, rawspeed already reads it and darktable can already process it fine. The problem is that the colors are way off because foveon sensors require special processing just to get acceptable color.

[rolkar]

The DNG we produce has Foveon "colors" and they need to be converted. We provide conversion matrices in the DNG meta data and also other meta data, such as field flattening data and bad pixels. Your DNG support should handle those to be DNG compatible. Then that would be solved.

This is true also for normal CFA data files, but to a much lesser extent.

[pedrocr]

I'm not working on darktable anymore and don't own these cameras so I'm not really an interested party anymore. But surely you're not saying that all this code does is read the X3F and add some metadata? Because if that's the case then that would be trivial to reimplement in darktable.

[rolkar]

Two comments.

1. Ah ... trivial :) No - it is not trivial. There is lots involved in generating this meta data. But, in principle this is what we do. We also do - severe noise reduction, finding black level, etc, etc.

2. I repeat - if Darktable do not use the color matrices and other meta data for DNG, then it is not a full DNG implementation. There might be som camera calibration data there or bad pixels or whatever. OK - you are no longer involved, but just so you do know. There is more to support than just being able to read the RAW data.

[pedrocr]

I find that odd because dcraw actually processes these colors in special ways but I don't really care anymore, feel free to close the issue if this isn't something you care about yourselves.

As for supporting all the quirks in DNG, you should realize that DNG is mostly a failed format that tries to do way too much. No one really cares about it beyond getting the data out of it from cameras that produce DNG by themselves, and those are all simple bayer pattern stuff. I'm sure you're following the spec just fine but nobody cares to support it besides Adobe. And in this case having the user have to use an external tool to generate extra files he has to save, doubling the space usage, just so he can load his raws isn't much of a workflow either.

[Vanek]

So their own attempt, reversed - https://github.com/darktable-org/rawspeed/commit/1cda622444fb13abe35ff42d1b3a7d940f728100 smdy can fix it?