New noise reduction algorithm

[Beep6581]

Originally reported on Google Code with ID 1052

Purpose of code changes on this branch:
To address complaints on the effectiveness of luma NR

When reviewing my code changes, please focus on:
Whether it's better.  A few tweaks are still possible, in particular an attempt was
made to preserve detail; more NR can be had at the cost of softening detail more.

The new code requires a new library to be installed: fftwf3, a fast Fourier transform
utility.  You can obtain it from 

http://www.fftw.org/download.html

For OSX, there is an easy installation via MacPorts.  

Reported by ejm.60657 on 2011-10-16 16:43:38

• Blocking: #1307, #1130

---

• Attachment: FTdenoise-5.patch

[Beep6581]

Forgot to mention, thanks to Michael for supply the necessary UI modifications.

Reported by ejm.60657 on 2011-10-16 16:44:58

[Beep6581]

any examples on how well it performs?

Reported by janrinze on 2011-10-16 18:34:00

[Beep6581]

http://theory.uchicago.edu/~ejm/pix/20d/posts/ojo/FTdenoise_compare.jpg

Reported by ejm.60657 on 2011-10-16 21:06:52

[Beep6581]

Are the A/B images images reversed? When I ran an ImageJ line profile on the black bottle
tops the before image has about 2.5X less noise than the after image.

Reported by scribble1@charter.net on 2011-10-18 13:34:24

[Beep6581]

Emil, you can try using the latest version of RT - I added a user-controlled lock for
 the Before view, this way the After view can be changed at will to also preview a
compound effect of multiple tools.

Reported by michaelezra000 on 2011-10-18 14:25:49

[Beep6581]

Unfortunately I did not manage yet to install fftw under MinGw64 to compile and contribute
to testing. If anyone succeeds, please share the details.

Reported by michaelezra000 on 2011-10-18 15:49:52

[Beep6581]

I have started writing another tutorial comparing RT to ACR with, unfortunately, ACR
coming out on top. Any idea when a build with the new algorithm will be done? And do
you have any links to info on the new algorithm? 

Reported by scribble1@charter.net on 2011-10-18 16:02:26

[Beep6581]

In what areas does ACR excel relative to RT?

Timescale for the new NR is probably a few weeks, to get feedback and tweak the parameters.

Reported by ejm.60657 on 2011-10-18 16:18:28

[Beep6581]

With my first pass at testing ACR cut about 3X more noise than RT. However I was measuring
white, grey and black patches which for wavelet noise reduction should be rather easy.
I'm going to redo the test with a target containing items with more detail so I have
a better idea what happens there. 

What information can you use? I'm planning to look at a range of ISOs starting a 6400
using both my 32 and 64 bit machines. I'll be using my Nikon 7000 and could post NEF
and jpg images if that would be useful.

Reported by scribble1@charter.net on 2011-10-19 13:57:06

[Beep6581]

Yes that would be a help.  There are a couple of things I'm thinking about that might
be done.  One is that I can certainly increase the strength of the NR; I am always
conservative on that front, I hate smeared detail but some people don't care and just
want to hammer the noise.  Another is that on many cameras the conversion from camera
colors to a standard color space amplifies noise (esp with weak color separation on
newer cameras) so it might be better to revert to the camera 'color space' for the
NR and then transform back.

Reported by ejm.60657 on 2011-10-19 14:23:50

[Beep6581]

In the algorithm would it be easy to implement a slider under user control, which would
determine the degree of detail retention vs. NR strength?

Reported by augusto.amatori on 2011-10-20 12:36:45

[Beep6581]

That's the slider labeled 'Luminance NR strength' or similar.

Reported by ejm.60657 on 2011-10-20 12:55:33

[Beep6581]

BTW still waiting for feedback.  Are people (besides Michael) having trouble installing
the FFTW libs?

Reported by ejm.60657 on 2011-10-20 12:56:52

[Beep6581]

Will try tonight...

Reported by natureh.510 on 2011-10-20 13:05:13

[Beep6581]

Emil, I was referring to the changes that you were considering for the algorithm in
order to "increase the strength of the NR" at the expense of "smeared detail".  Could
those choices be turned into a user controlled variable?

Reported by augusto.amatori on 2011-10-20 13:41:42

[Beep6581]

Smearing the details is the direct consequence of increasing the strength, so the Strength
slider is what you're looking for ;)

Reported by natureh.510 on 2011-10-20 13:46:11

[Beep6581]

Got it, and perhaps I am misunderstanding.  What under the hood changes was Emil thinking
of implementing in order to better compete with ACR  - that would make the NR stronger
at the expense of smeared detail?  Simply allowing a bigger range in the 'Luminance
NR strength'?

Reported by augusto.amatori on 2011-10-20 13:55:38

[Beep6581]

Yes, a bigger range. There is also a point in the algorithm where Fourier coefficients
are preserved more if they are larger than the neighborhood average; removing that
smooths things more at the expense of lost detail contrast.  One could also try adding
an edge mask with strength slider, a la ACR, though that would take a bit of work to
set up and cost more processing time.

BTW, since I still don't have OpenMP working, the OpenMP commands are just a guess,
I would like help making sure there are no bugs there.

Reported by ejm.60657 on 2011-10-20 14:05:47

[Beep6581]

Emil, by looking at the library you linked in (fftw3f), i guess that you're using "float"
precision, right?

For those interested, here is my configure script setting to build the library under
windows 32bits (in an MSYS console):

./configure --prefix=/mingw/ --enable-float --enable-openmp --disable-fortran --enable-sse2
--with-our-malloc16

And...

make
make install

...will do the rest.

I managed to compile RT4 with your patch, but i had to tweak and ignore some region
in main.cc(!?), and now the processing time is loooooooong and give a blue image with
the neutral profile.

Could you provide an updated patch in sync with Default?

Reported by natureh.510 on 2011-10-20 20:08:32

[Beep6581]

Yes float precision; double (which is the FFTW default) is overkill and costs more processing
time.

I will work on a patch update.  I probably included some gunk that cleans out the rtstart
changes from a month or so ago, we still have to get together to fix default so that
it works for me in XCode.

Reported by ejm.60657 on 2011-10-20 20:26:03

[Beep6581]

Hombre, thanks, I will try this to compile. My previous attempts resulted in compilation
errors under msys. 

Emil, does the current version of the Cmakerlists work for you on Mac (it's version
in the patch is rather old)?

Reported by michaelezra000 on 2011-10-20 21:21:25

[Beep6581]

This seems to work for me (but not in XCode):

Reported by ejm.60657 on 2011-10-20 21:43:20

---

• Attachment: FTdenoise-8.patch

[Beep6581]

Hombre, I get this error:

Michael@MK-NY-D1 /C/msys/1.0/home/Michael/fftw-3.3
$ make
make  all-recursive
make[1]: Entering directory `/C/msys/1.0/home/Michael/fftw-3.3'
Making all in support
make[2]: Entering directory `/C/msys/1.0/home/Michael/fftw-3.3/support'
make[2]: Nothing to be done for `all'.
make[2]: Leaving directory `/C/msys/1.0/home/Michael/fftw-3.3/support'
Making all in kernel
make[2]: Entering directory `/C/msys/1.0/home/Michael/fftw-3.3/kernel'
/bin/sh ../libtool --tag=CC   --mode=compile gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I..
 -I../simd   -O3 -fomit-frame-pointer -mtune=native -malign-double -fstrict
-aliasing -ffast-math -MT align.lo -MD -MP -MF .deps/align.Tpo -c -o align.lo align.c
libtool: compile:  gcc -std=gnu99 -DHAVE_CONFIG_H -I. -I.. -I../simd -O3 -fomit-frame-pointer
-mtune=native -malign-double -fstrict-aliasing -ffast-math -MT ali
gn.lo -MD -MP -MF .deps/align.Tpo -c align.c -o align.o
In file included from align.c:22:0:
ifftw.h:200:6: error: #error sizeof void* is unknown!
make[2]: *** [align.lo] Error 1
make[2]: Leaving directory `/C/msys/1.0/home/Michael/fftw-3.3/kernel'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/C/msys/1.0/home/Michael/fftw-3.3'
make: *** [all] Error 2

Michael@MK-NY-D1 /C/msys/1.0/home/Michael/fftw-3.3

BTW, I am trying to compile 64 bit version. Is this the one you compiled?
If so, could you share the result?

Reported by michaelezra000 on 2011-10-20 23:33:08

[Beep6581]

Also, I am on Athlon 64X2, could that make a difference?

Reported by michaelezra000 on 2011-10-21 00:01:57

[Beep6581]

I guess that you launched the ./configure process with my parameters!? I didn't read
anything about 64bits specificities, so i gues that it's the same process.

If you already made an unsuccessful attempt to build fftw, you should clean everything
with "make distclean", then relaunch the configure process.

I think that it may be related to the compiler, i use TDM-Gcc 4.5.0/32bits, i guess
you're using MinGW-64. I don't know much what's going on here, sorry.

Reported by natureh.510 on 2011-10-21 00:14:40

[Beep6581]

Try removing "--with-our-malloc16" in the configure process, hope that helps.

Reported by natureh.510 on 2011-10-21 00:15:53

[Beep6581]

yes, of course I ran ./configure :)
retrying now without --with-our-malloc16

Reported by michaelezra000 on 2011-10-21 00:43:55

[Beep6581]

same problem...

Reported by michaelezra000 on 2011-10-21 00:47:53

[Beep6581]

Compiler: TDM GCC 4.5.2 x64

Reported by michaelezra000 on 2011-10-21 00:49:19

[Beep6581]

No fftw lib problems in gentoo, havent tried it in ubuntu yet.

Reported by entertheyoni on 2011-10-21 11:15:20

[Beep6581]

Sorry, i missed your patch from comment 23. I applyed it and it failed to build: it
looks like you forgot to include FTblockDN.cc

Reported by natureh.510 on 2011-10-21 15:38:44

[Beep6581]

Must have reset the status when doing some Mercurial operations; it was added previously.
 Anyway, here it is with the offending file added:

Reported by ejm.60657 on 2011-10-21 16:14:18

---

• Attachment: FTdenoise-10.patch

[Beep6581]

@Michael:

The key error message is

"ifftw.h:200:6: error: #error sizeof void* is unknown!"

which mean that the compiler don't know if pointers are 32 or 64 bits (i.e. if you
want a 32 or 64bit application). By looking at this page (http://sourceforge.net/apps/trac/mingw-w64/wiki/MSYS),
you may need to add

--host=x86_64-w64-mingw32

to the configure parameter (not tested though).

Reported by natureh.510 on 2011-10-21 16:27:15

[Beep6581]

No more chance with the last patch: i still have a completly blue image! All pixels
from red and green chanels are set to 0. Do you have any idea where to look? Also,
is it worth finding this bug now if you plan to use the xyz_cam instead?

Reported by natureh.510 on 2011-10-21 17:42:45

[Beep6581]

Which patch are you talking about?  FTdenoise-10?

And yes it is worth finding this bug, it is independent of what I hope to test (which
might not even bring improvements enough to make it worthwhile).

Reported by ejm.60657 on 2011-10-21 18:19:56

[Beep6581]

Yes, i'm speaking of FTdenoise-10.

Reported by natureh.510 on 2011-10-21 18:56:12

[Beep6581]

Hmmm.  I just rebuilt from the command line using FTdenoise-10, and it works for me.

Reported by ejm.60657 on 2011-10-21 19:35:50

[Beep6581]

I thought that i was an OpenMP problem, but it doesn't seem to be the case: with FTdenoise-9b_before4Hombre.patch
as is, it's even worse as i get a pure black image.

What should i set on line 263 to test with cam_xyz by the way?

Reported by natureh.510 on 2011-10-21 20:09:06

[Beep6581]

I forgot to mention: i tested on an image loaded with neutral profile, so without any
noise reduction activated... That's what puzzle me the most!

Reported by natureh.510 on 2011-10-21 20:10:50

[Beep6581]

@ #39: My problem is that I get compile errors if I uncomment the appearance of cam_xyz
it is out of scope.  How do I get it in scope in ImProcFunctions, given that its scope
is in RawImageSource.  Anyway, please do not use that patch as a test ground for the
algorithm, it is very much bugged.  The discussion here should be about FTdenoise-10,
which for me works beautifully, albeit with OpenMP switched off in compilation.

Reported by ejm.60657 on 2011-10-21 22:22:11

[Beep6581]

Okay, then i'll use FTdenoise-10 as a test ground to introduce cam_xyz, then you'll
easilly report the modification wherever you want.

I'll then test FTdenoise-10 without OMP right now.

Reported by natureh.510 on 2011-10-21 22:31:58

[Beep6581]

I tried all kinds of combinations and also this one, but got the same error (with "make
distclean" every time between tries).

$ ./configure --prefix=/mingw/ --enable-float --enable-openmp --disable-fortran
 --enable-sse2 --with-our-malloc16 --build=x86_64-w64-mingw32 --host=x86_64-w64-mingw32

Olli, you had a better luck compiling x64 dependencies before, if you find some time,
could you take a look?

Reported by michaelezra000 on 2011-10-22 02:01:35

[Beep6581]

OK, I did a test by manually inserting a camera matrix and doing the NR before input
profile application.  Here is the standard pipeline

http://theory.uchicago.edu/~ejm/pix/20d/posts/ojo/FTdn_after.jpg

and here is what one gets by doing NR before input profile (ie in the 'camera colors')

http://theory.uchicago.edu/~ejm/pix/20d/posts/ojo/FTdn_before.jpg

This convinces me that it is worthwhile rearranging the pipeline to put NR much earlier,
or to make a roundtrip through the camera 'color space' for the purpose of NR.

Reported by ejm.60657 on 2011-10-22 22:11:51

[Beep6581]

Just for the record, i've investigated the cma_xyz problem and i've almost found a solution
(if that's related to comment #44), but i'm actually concentrated in Jacques' Vibrance
tool. I should be able to provide a solution tomorrow.

Reported by natureh.510 on 2011-10-22 22:22:00

[Beep6581]

Emil, could you show the same comparison when all NR sliders are set to the same values,
e.g. like in the FTdn_before.jpg ?

Should we look at the difference in the chroma only?

Reported by michaelezra000 on 2011-10-22 22:49:19

[Beep6581]

The chroma is much worse if FTdnafter is set to the same values; there are a number
of impulsive pixels with colors that are way off (typically red).  I had to crank up
the NR to get rid of them (only mostly).  I adjusted the chroma slider until the two
results looked about the same as far as chroma noise; therefore you can judge by how
much more was required for FTdn_after.  Luma sliders are the same, so if you want sliders
the same you should be looking at the luma noise and notice all the textural artifacts
left behind in FTdn_after that are not there in FTdn_before.

All this is related to a common phenomenon of red-green color blotching in digital
files, especially in deep shadows.  I am not sure what causes it, but it seems to be
related to some sort of correlation in these channels over small regions; typical input
profiles amplify it a lot, especially on cameras with weak color separation in the
CFA.  By doing the NR before application of the input profile (or, I am hoping, by
using the matrix profile to go back to an approximation of the camera space for the
purposes of NR, then returning), the effects of this blotching/correlation are suppressed
and the NR can be more effective because it doesn't see false correlations and interpret
it as detail.

Reported by ejm.60657 on 2011-10-22 23:14:11

[Beep6581]

Looks like a substantial difference. 
In the "before" version" the color of the green leaf is much better preserved.

Could making a round trip into camera space introduce any global color shifts?
Is it more difficult to inject NR while image is still in camera colors?

Reported by michaelezra000 on 2011-10-22 23:37:36

[Beep6581]

I think the point is to align the color space so that artifacts are minimized, so that
NR doesn't have to deal with them.  Then with the image smoothed, the artifacts are
not as prominently generated by the transform to the working space through the input
profile.  Shouldn't lead to color shifts except insofar as any NR averages color over
patches.

I have been thinking about whether it would be best to move NR closer to the beginning
of the pipeline.  I think it might, but that is a more extensive change.  Maybe a lot
easier under the code rewrite which I hope we will do someday.

Reported by ejm.60657 on 2011-10-23 00:27:00

[Beep6581]

Would then the execution sequence of the simpler approach be:
 demosaic
  - camera color
 lcms transform to working space
  - working space
 reverse lcms transform
  - camera color
 NR
 lcms transform to working space
  - working space

Reported by michaelezra000 on 2011-10-23 00:38:43

[Beep6581]

Hombre, that's great news.  In the process of generating the demo, I realized that it
might be more convenient to have rgb_cam and cam_rgb passed to ImProcFunctions than
cam_xyz.  Of course it is straightforward to get any of these from any other one, so
once the code is in place we can adapt it.  Let me know if you need any clarification
on what I need.

Michael, the simpler version would be 
1. Demosaic -- camera color
2. Exposure compensation (so data has proper normalization)
3. Assign working profile and transform to Lab 
(just like using 'no profile' in the ICM tab)
4. Perform NR
5. Transform Lab to RGB; Assign camera profile (back to colorimetric correctness)
6. Apply inverse of exposure compensation to restore data normalization
6. Proper transform (lcms or internal matrix) to working space RGB and on to rest of
pipeline

Here the issues are to pass EC setting to RawImageSource, and also all the methods
for conversion Lab->RGB and back are in ImProcFunctions, so RawImageSource would need
access to them.  I also think it best to keep the NR function in ImprocFunctions, to
avoid having to duplicate the code in both RawImageSource and SimpleProcess etc.  But
again how to give RawImageSource access to the function defined in ImProcFunctions?
 If this could be done, I would place the NR operation right before the color space
conversion in RawImageSource::getImage.  From my meager understanding of C, I somehow
need ImprocFunctions to be a 'friend class' to RawImageSource, but every time I try
to implement that it doesn't work.

Reported by ejm.60657 on 2011-10-23 01:24:19