Preprocession option needed for inverting image colors

[Beep6581]

Originally reported on Google Code with ID 2193

When capturing a negative image in a RAW file and attempting to process with rawtherapee
using the inverted tone curves to process a negative image and it is very problematic.
 All the functions operate in reverse and I could not get the image to process without
loosing data in the highlights or shadows... (the program is designed to work on positives
in subtle and not so subtle ways, it is biased towards highlight preservation which
is shadow on a negative image) I did find a workaround though.  What I do is process
the image as neutral and only invert the colors nothing else then I save it as a 16
bit TIFF.  Next I open the image from the TIFF file and process it the way I want...seems
to work, but very cumbersome.  What I suggest is adding a very simple conversion in
the "preprocessing tab", a check box which inverts the color values in the virgin RAW
image data before any other functions so the program is working on an image as a positive
from the start.  This would make it possible to process the entire image in one operation,
even automated if you have lots of negatives to scan.  

saw a similar request in issue 1275 which makes me believe this feature would benefit
lots of people which copy old negatives.

Describe the problem and how to reproduce it.
I experienced this when I copied a fully color balanced negative image on a 5dmkii.
 The histogram showed all colors in the middle of the histogram with no clipping. 
I even tried changing exposure to reposition the curves, but when inverting the data
in RT and doing an auto exposure correction or even manually, usually one of the colors
would fall off the left of the histogram and you wind up clipping some color.  

Branch: default
Version: 4.0.11.1
Changeset: b741cc937a4d
Compiler: gcc 4.7.1
Processor: 1
System: Windows 7
Bit depth: 64 bits
Gtkmm: V2.22.0
Build type: RELEASE
Build flags:   -fopenmp -O3 -DNDEBUG -mthreads -funroll-loops
Link flags:   -mwindows -s
OpenMP support: ON
MMAP support: ON

Reported by soccerdadphotos@yahoo.ca on 2014-01-10 01:20:53

[Beep6581]

For the moment, for raw files, the best is to use an inverted camera dcp profile ..

http://rawtherapee.com/forum/viewtopic.php?f=1&t=5210&start=30

It is easy with adobe's DNG_profile_editor (just invert the tone curve)

When a developer will take this issue it should be easy to add a negative option check
box in color management menu ..

Reported by iliasgiarimis on 2014-01-10 02:53:29

[Beep6581]

I have created code to provide this feature.  I just have to figure out the technical
details on how to provide a patch to let other people test it out.  The code I wrote
adds a button to the Color Management panel as a alternative to the DCP input profiles.
 Clicking on the button inverts the colors in the image.  The user would then use the
other RT options to improve the flat appearance of the raw image such as a concave
tone curve, white balance and set the exposure to obtain a good result.

Reported by smartyrawtherapee on 2014-02-06 18:59:55

[Beep6581]

Great! Tell us about what software/system you use, then we can help.

Reported by entertheyoni on 2014-02-06 20:36:16

• Status changed: Started
• Labels added: Type-Enhancement, Component-Raw
• Labels removed: Type-Defect

[Beep6581]

Okay... I figured out what I needed to do.  I'm on a windows 7 machine and ran in to
some compatibility issues with my editor (unix vs dos), but I've worked around them.

I created a patch off the latest RT and I have attached it to this issue.  I'm not
sure how the process goes fro here, but I figure if it is here people can try it for
themselves.  I've tested it and it works pretty good for my purposes.   Basically,
once the image is loaded, you go to color management, select 'invert colors' and the
image gets inverted.  Then you use normal processing to do the rest.  I had very good
results by using a concave the tone curve, auto exposure, auto white balance to start
and playing with exposure a bit.  To get really good results you need to white balance
in the camera (get rid of the orange mask to avoid clipping certain colors).   

Reported by smartyrawtherapee on 2014-02-07 03:10:09

---

• _Attachment: 2590_for_issue2193.patch_

[Beep6581]

Thanks for the patch. Compiles fine. Please upload a color negative file somewhere (www.filebin.net
then give us the URL) to test. Because your change only works with raw files, wouldn't
it be better to move it to the raw tab?

Ingo

Reported by heckflosse@i-weyrich.de on 2014-02-07 13:03:39

• Status changed: PatchSubmitted

[Beep6581]

Thanks smartyra.. :)

I think color management is a good place for this feature.

I didn't try yet but I suppose that it should work with line rgb files also (not only
raws).

What remains is to expand it's utilization on all files ..

Reported by iliasgiarimis on 2014-02-07 14:04:21

[Beep6581]

Thanks for the patch! 
I observe some (unnatural) tonal flattening after the inversion.
Does the patch ensure floating point precision is used?
I expected that the inversion result would be identical to use of a linear inversion
RGB tone curve. Consequently, the inversion would be 100% revertable by the linear
inverse RGB curve. However, this is not the case:)

Reported by michaelezra000 on 2014-02-07 14:38:09

[Beep6581]

The difference might be from filtering that happens later on in the code.  The invert
I did is done very early to avoid having to invert filtered data which you would have
to double filter in reverse to get back to desired point.  I loaded up an image from
Adobe DNG and using their tone curve (a perfectly inverted linear one with no camera
curve) to make a DCP file and I can get similar results, but I have to apply my own
tone curve first when I use my own invert code.  (I'm applying a base concave curve
on tone curve 1, and my S curve for personal taste on tone curve 2)  The invert I wrote
doesn't apply any curve at all so, that's why it may look flat.  I decided not to add
a typical digital camera curve after because film has its own characteristics and I
thought it better to apply your own custom curve to allow you to get the most accurate
results.  This is also where my knowledge drops off, perhaps someone who knows the
DCP code well can take a look at my code to see if I'm not doing a step I should be
doing.  I was expecting to do simple flip of the bits in the raw image, but it is converted
to 16 bits fp before I can do that which makes things a little weird since the image
is only really 14 bits of data...so I may be misunderstanding how RT scales the data
to fit.  I have no clue how RT knows where the white and black points are...I couldn't
find it in the code and RT seems to differ from other programs that have white and
black sliders.  So I had to find a way to reposition the colors.

There is one assumption/limitation I made because when you invert the colors the entire
image density becomes shifted and you must somehow correct for that.  My assumption
is that all image data starts at the black point so I automatically shift the image
down to start at that region which means as a user you might have to push the exposure
up to get bright whites.  It is the only way I could do it without really complicating
things and it preserves the data in its most basic form so you are always editing it
from a defined starting point.   

Reported by smartyrawtherapee on 2014-02-07 20:19:40

[Beep6581]

Does not work for me.

What we really need is to map the range of each color channel R, G, B from 0-65535
to 65535-0 

for reference the a sample file and the dcp profiles (normal and negative)
http://filebin.net/hu8s9gjpct

Reported by iliasgiarimis on 2014-02-08 00:04:04

[Beep6581]

To clarify .. by "does not work" I mean it's very flat and bright, while the colors
ARE inverted .. 

I forgot to mention that you need to check "use dcp's tone curve" with the negative
dcp for "correct" results on the reference samples.

I think we should call it "Negative" instead of "Invert colors"..

The "Negative colors" option should operate in addition to the rest options. If we
use a dcp profile, or camera standard or any .. it should apply on the resulting by
the profile colors, the inverted linear curve when the data are linear or the inverted
gamma when the data are gamma encoded.

Reported by iliasgiarimis on 2014-02-08 03:14:07

[Beep6581]

I can test once a few negative scans are provided.

Reported by entertheyoni on 2014-02-08 19:34:12

[Beep6581]

Hi all...  I can profile some sample .CR2 files, but probably not for a couple days.
 The ones I used have people in them who might not approve of me distributing them...
so, I'll have to pick a few, copy them etc...  

I did some investigating on the DNG tone curve method, my solution was modeled after
that...but I wanted something that was all in one and that doesn't rely on their tone
curve.  So you provide that in your own tone curve in RT.  Anyway, I wanted to mention
this part...there is a bug in the Adobe DNG Profile Editor when you try to make an
inverted tone curve using their "linear base curve".  You would expect the resulting
.DCP file to have linear data in it, but it does not.  It includes their default base
curve.  I believe it is an error because if you use a non-inverted curve they do not
apply any base curve and you do get a linear dataset.  The reason I mention this, is
because if you do a side-by side comparison, the adobe method will not look as flat
because it has a tone curve already included.  

Reported by smartyrawtherapee on 2014-02-09 01:44:09

[Beep6581]

Great! that explains the "unnaturalness" of the inversion I referred to. 
All that is needed, I think, is a simple linear inversion operation.
However, inversion has to happen after the input ICC/DCP/matrix of the capture device
is applied, on top of the capture device calibration.

Reported by michaelezra000 on 2014-02-09 01:50:25

[Beep6581]

The image I take from the invert option looks like not linear negative. The concave
curve needed is in fact imitating a gamma curve.

Compare with the NegativeLinear.dcp both neutral and with a concave curve ..

http://filebin.net/hu8s9gjpct/Canon_PowerShot_A720_IS_a720_v2-negative-linear.dcp
http://filebin.net/hu8s9gjpct/1_CRW_8811b-InvertPatch.jpg
http://filebin.net/hu8s9gjpct/1_CRW_8811b-InvertPatchcage10-0.jpg
http://filebin.net/hu8s9gjpct/1_CRW_8811b-NegativeLinearDCP.jpg

The colors differ because the dcp profile is a bad one and I just use it for the negative
effect ..

One more problem that remains with the current approach(es) is that WB controls work
inversely ..

Reported by iliasgiarimis on 2014-02-09 12:45:37

[Beep6581]

There is a possible race condition in the parallelization in invertPixelColors(..).
It's extremely unlikely, but it can happen that the variable 'newlowest' doesn't contain
the value it should. Fixed with this patch.

Ingo

Reported by heckflosse@i-weyrich.de on 2014-02-09 17:18:37

[Beep6581]

Oups, small bug in patch from #16. Will make a new one.

Reported by heckflosse@i-weyrich.de on 2014-02-09 17:36:40

[Beep6581]

This one is better.

Reported by heckflosse@i-weyrich.de on 2014-02-09 17:41:19

---

• _Attachment: issue2193_rc.patch_

[Beep6581]

Thanks Ingo, I will try later as I had some problems with the inverted colors patch
patch, (the edit preview at 100% darkening when displaying the middle of the frame
..) 

A useful synthetic raw ramp 
http://cl.ly/3F2v0T0j2x3P/RGB_255to0_52steps_Bayer.dng.zip
http://www.dpreview.com/forums/thread/3371013

We have to first disable the "Avoid color shift" option in Lab adjustments to make
RT totally neutral .. 

Reported by iliasgiarimis on 2014-02-10 10:39:07

[Beep6581]

Ilias, no need to try. It's really very unlikely that you ran into this possible error.
I only wanted to have it fixed, because this things are hard to find, if they occur.

Reported by heckflosse@i-weyrich.de on 2014-02-10 10:43:59

[Beep6581]

I put together some sample images and files from my testing.  The link to download is
https://dl.dropboxusercontent.com/u/63600569/Sample_stuff_for_RT.zip  Within the folder
is a bunch of stuff.  In the jpg images and profiles, I applied the most basic operation
as a starting point to compare the images.  I confirmed the Adobe DNG is adding a base
curve...I included a bunch of excel graphs to show what each .dcp file has for a tone
curve. ...its interesting.  My conclusion so far...in trying to compare apples with
apples its really easy to get oranges mixed in there too...  Some images work well
and others seem to be very difficult.  At this point I don't know what needs to be
done.  I did try something with the card test jpg in the folder and its exact inverse,
if you load it up and invert it with a tone curve (being very careful to select "no
profile" and "neutral exposure", you can generate the other image exactly... so the
tone curve method is correct...the automatic profile setting and exposure is a little
frustrating to figure out what RT is doing sometimes.  If I could match the tone curve
functionality it would be good...I'm not sure why the exposure seems so far off.

Reported by smartyrawtherapee on 2014-02-11 04:53:43

[Beep6581]

Exposure should not change... but we perceive it differently due to the nature of our
perception:)

This shift is likely due to what you are explaining above: "I automatically shift the
image down to start at that region which means as a user you might have to push the
exposure up to get bright whites."

Reported by michaelezra000 on 2014-02-11 05:04:03

[Beep6581]

I've got a pretty good grasp on what is conceptually happening.  The digital camera
captures brightness linearly.  In other words, if you double the exposure time, the
pixel data will double.  This I confirmed in RT by dumping out the values where I placed
the invert function, the data is pretty much linear.  This tells me no correction is
being done at that point which I think is a good thing, because the negative already
has a response curve which we want to correct for later on using the regular process...
 When you copy a negative, if you include part of the border, you will have a place
to set your white balance eye dropper and the invert code uses the brightest value
to set the minimum black point.  The hard part is finding the maximum white point.
 I would like to set it automatically based on the darkest part of the negative, I
can do that by scaling the data, but maybe there is already a way to set it automatically...using
the existing white point correction...I need some assistance to figure that part out.
 The second part, after thinking about it some more, I'm now not sure if I'm inverting
the values completely correctly because although the data is linear compared to exposure
it is logarithmic compared to each step in tone...so black values are closer together
in range than white so I think I may need to scale the results accordingly.  I'll have
to play around with the code some more... thanks for all the help so far.  

Reported by smartyrawtherapee on 2014-02-12 22:39:29

[Beep6581]

I think that setting white and black during the inversion process is equivalent to the
auto levels operation, not just an inversion, so this operation would contradict the
meaning of the label on the checkbox.

Would could be more useful, is to 
1. Do a simple inversion. 
2. Add a new auto-levels method to drive the tonecurve 1 only and set its black and
white point in accordance with the indicated clipping level. The tonecurve would keep
the linear shape.

In this case #2 becomes more universal and useful for both positives and the negatives
(which become positives due to #1.

What do you think?

Reported by michaelezra000 on 2014-02-13 02:18:42

[Beep6581]

#2 is a separate issue, however.

Reported by michaelezra000 on 2014-02-13 02:19:33

[Beep6581]

#23 smartyra.. , the inversion is correct and the picture becomes so bright and flat
because it acts on linear data (I was wrong in #15).

If you try to export in linear space (use RT_sRGBg10 as output profile) then try in
Photoshop to invert, the result is the same bright flat picture. By changing to a gamma
encoded space the picture is the usual "correct" positive.
So I think that to have a perceptually good starting point the inversion must act on
gamma encoded data (or use the gamma curve during the inversion) and I suppose that
the gamma used should the the gamma of the working space.

Regarding the Black and white level of the camera, they are already used for the scale
color calculations so no need to worry for these just use 0/65535. Any additional black/white
point offset is really depended on inaccurate "scanning" or film development / aging
and should be corrected afterwards. It is a part of "film response curve" ie an S shape
curve with black/white offsets.

Reported by iliasgiarimis on 2014-02-13 10:46:58

[Beep6581]

The idea to put a black and white slider on the tone curves may be a good one, if you
look at how Canon's DPP program works, the black and white sliders don't seem to operate
on the data, but they define where the filtering begins and ends.  In RT the curve
is applied to the entire range of data even if the data is not relevant.  You can do
the same by pulling in the end points on the curve, but then you are changing the curve...
 I'm not sure what's better, but in DPP I always start with 1. adjust the exposure
which slides the histogram left or right, 2. pick the curve, 3. slide the black point
right which determines where the curve begins.  Then contrast and brightness etc, changes
the shape of the curve including the knee and shoulder which also hit the data you
want it to hit, otherwise the knee would be applied to black space.

Reported by smartyrawtherapee on 2014-02-13 17:35:12

[Beep6581]

Good news!  I coded up a new version of the raw invert function.  The working version
is attached.  I tested it and it works on every image I have.  It is not
fully automatic, you have to sometimes drag the exposure quite far to the right, but
the easy method is invert the image, do auto white balance, then auto exposure level,
then drag the exposure level to where you want... after that, adding contrast or using
a tone curve etc.  Give it try... 

Reported by smartyrawtherapee on 2014-02-14 18:54:52

---

• Attachment: working2193.patch

[Beep6581]

I see a couple of problems:
1. some images open almost entirely black.
2. if image has a white and black borders, then in the preview the inversion effect
changes depending on whether these borders are visible in the preview area or not.
This is with neutral profile and no auto levels.

Reported by michaelezra000 on 2014-02-14 21:24:45

[Beep6581]

Item 1 might be expected... if you slide the exposure over does that display the image?
Item 2 ...sounds like some unrelated issue especially if the invert image button is
not selected because the new code won't ever be called.

I had good results.  Can you upload your test image for me to try?

Reported by smartyrawtherapee on 2014-02-15 07:30:04

[Beep6581]

Conversion of your IMG_0368.cr2 .. nice

http://filebin.net/1uz2gpc379

In the previous link you will find 2 snapshots of the edit window preview showing the
problem mentioned by Michael.
As soon as there are any bright spots in the visible frame it becomes dark in the 100%
preview. Most times the darkening remains in 50% or/and 33%. In lower zoom presets
it gets normal.

The problem disappears when we use the dcp method (I used curve2.dcp all else settings
same as with "invert colors") 

Reported by iliasgiarimis on 2014-02-15 14:12:00

[Beep6581]

The same problem existed with the previous patches also ..

Reported by iliasgiarimis on 2014-02-15 14:19:33

[Beep6581]

Bump, patch is going stale.

Reported by entertheyoni on 2014-03-08 20:50:09

[Beep6581]

Pity

Reported by entertheyoni on 2014-11-06 13:19:00

• Status changed: WontFix

[Beep6581]

iliasG if you have a DCP for inversion for treating negatives, could you make it available,
so we can bundle it with RawTherapee?

Reported by entertheyoni on 2014-11-06 13:22:57

[Beep6581]

It would need a dedicated negative-dcp for each normal dcp so this is no option I think.
The idea here was to auto-inverse the tone curve by a click. This patch was close to
the desired effect just some refinements needed regarding black-white levels and color
scaling.

But in the meantime we have available the negative haldCLUT :) doing the same job :)
so anyone interested in negatives can use this :)

I have not yet tested the inversion quality of this option .. 

A problem could be that this option is under a "film simulation" label which is not
easy to be comprehensive by a normal user.
Maybe a more user friendly approach would be a check box in the color management group,
which when enabled will make use of the negative haldCLUT ..

Reported by iliasgiarimis on 2014-11-06 14:05:25

[Beep6581]

Issue 1275 has been merged into this issue.

Reported by iliasgiarimis on 2015-01-08 03:41:34

[Beep6581]

Issue 1275 has been merged into this issue.

Reported by entertheyoni on 2015-01-10 23:15:38

[Beep6581]

I started a RawPedia page for processing negatives, please expand (don't worry about
poor English, I can revise):
http://rawpedia.rawtherapee.com/Negative

Reported by entertheyoni on 2015-01-10 23:52:27

[Beep6581]

Issue 1275 has been merged into this issue.

Reported by entertheyoni on 2015-01-10 23:54:12

[Beep6581]

Issue 1275 has been merged into this issue.

Reported by entertheyoni on 2015-01-11 14:00:41

[Beep6581]

# 39, answered by mail :) 

Reported by iliasgiarimis on 2015-01-12 23:58:50

[Beep6581]

Issue 1275 has been merged into this issue.

Reported by entertheyoni on 2015-01-13 01:22:12

[Beep6581]

Roger

Reported by entertheyoni on 2015-01-13 01:23:06

[Beep6581]

p.s. I don't know why it keeps merging 1275 into this, I didn't touch that field. GC
quirk.

Reported by entertheyoni on 2015-01-13 01:24:08