Hot/Dead pixel filter

[Beep6581]

Originally reported on Google Code with ID 2470

Though the actual Hot/Dead pixel filter in RT does a good job in removing hot pixels,
it does a bad job in removing dead pixels (see Issue 2281) and also introduces artifacts
by removing dead pixels.

I changed the method to filter only hot pixels. This way the most (if not all) artifacts
produced by dead pixel filter are gone now and I also got a speedup of about 25% for
hot pixel filter at my machine.

In a following patch I will address the dead pixel filter and make changes to get the
hot/dead pixels filter working correctly on images with xtrans sensor.

Ingo

Reported by heckflosse@i-weyrich.de on 2014-08-11 18:29:00

---

• _Attachment: hotdeadpixel_00.patch_

[Beep6581]

Here is a comparison between no, old and new (from left to right) version of hot/dead
pixel filter : http://www.i-weyrich.de/rt/hotbadpix_comparison.png with 'hot_pixels_long_exposure_381s.pef'

Ingo

Reported by heckflosse@i-weyrich.de on 2014-08-11 18:54:10

[Beep6581]

Issue 2265 has been merged into this issue.

Reported by heckflosse@i-weyrich.de on 2014-08-11 19:19:02

[Beep6581]

Ingo I'll test your patch for xtrans when available. 

Reported by sguyader on 2014-08-11 20:57:54

[Beep6581]

Before I continue work on this Issue, I want to discuss some things:

Actually (without the patch) hot and dead pixel removal is executed after flat field
correction. Also it's not called for the flat field itself. Though it's very unlikely
that a ff shot will have hot pixels it is also very likely that it will have dead pixels
if the cam has dead pixels. If we don't remove this dead pixels from flat field they
are treated as small dust spots when ff correction is used for dust removal.

So for a better work of hot/dead pixel removal combined with flat field correction
we should

1.) remove dead pixels from flat field?
2.) remove hot and dead pixels from image before applying the flat field and not afterwards?

If someone has images with bad pixels, please upload some to http://filebin.net/ .
The more images I can use for tests, the better the result will be.

Ingo

Reported by heckflosse@i-weyrich.de on 2014-08-12 17:38:40

[Beep6581]

I meant 'DEAD', not 'BAD', in last paragraph

Reported by heckflosse@i-weyrich.de on 2014-08-12 17:57:58

[Beep6581]

I made some tries with the actual 'hot' pixel detection:

Here are three images from different cameras, all completely overexposed:

Nikon D700:  http://www.i-weyrich.de/rt/DSC_5045.NEF
Nikon D200:  http://www.i-weyrich.de/rt/DSC_9005.NEF
Pentax K10D: http://www.i-weyrich.de/rt/imgp2663.pef

In none of this images 'hot' pixels should be detected, because they all are completely
overexposed, but in the D700 and K10D image RT detects a lot of hot pixels.

Actually, I don't know why, but I guess it's because colors are scaled before hot pixel
detection (only a guess)
Ilias, could you have a look at these images?

Ingo

Reported by heckflosse@i-weyrich.de on 2014-08-13 20:52:58

[Beep6581]

Hoping it could be useful i post a link to a file made with a Canon compact camera with
CHDK where i forgot to create a valid deadpixel file, seems to have a lot of problematic
pixels:

http://goo.gl/W8RmKM (download with the arrow on the top)

Thank you.

Reported by fotogiuliopv on 2014-08-14 18:33:43

[Beep6581]

re #8: That's a very useful example! Thank you very much!

Ingo

Reported by heckflosse@i-weyrich.de on 2014-08-14 21:24:49

[Beep6581]

I had a look at the image from Issue 2265 (panasonic file).
Detecting dead pixels in Panasonic files is very easy. dcraw sets the variable 'zero_is_bad'
to '1' for panasonic files. We can simply mark all pixels with value '0' as bad when
'zero_is_bad' is set to '1'. But we have to do this before color scaling. And this
won't solve the problem for other cams. I'll make a patch for this case today.

Ingo

Reported by heckflosse@i-weyrich.de on 2014-08-15 10:49:54

[Beep6581]

#7 Ingo, I looked at the D700 sample .. it clips lower than Dcraw's/RT's default 16383
so a camconst.json entry could help

One little problem is that your sample and the samples of Russel Cottrell seems to
come from different variations of D700 !!
Different white clipping and different sensor construction (seams - uniformity) 
We can use Russell's more conservative setting for safety
http://rawtherapee.com/forum/viewtopic.php?f=1&t=5537#p38136

http://filebin.net/vy9pthck1s

or use the following to see the uniformity of your sample in RT
  { // quality B; sensor is not uniform
        "make_model": "Nikon D700",
        "dcraw_matrix": [ 8139,-2171,-663,-8747,16541,2295,-1925,2008,8093 ],
        "ranges": { "black": 15700 } 
       // "ranges": { "white": 15427 }
   },  

#10 indeed Panasonic bad pixels are a somehow different case from hot/dead pixels ..
and easy to correct :)
I think such black pixels (panasonic's and I think that there is a similar case with
Canon 650D) should be corrected before any other operation

Reported by iliasgiarimis on 2014-08-17 19:15:19

[Beep6581]

Hi Ilias, thanks for the camconst.json entry for D700. I will try it.

About the hot/dead pixel removal:

Old Hot/Dead pixel removal was one function, which did both. There was no possibility
to enable only one of them. The dead pixel removal killed a lot of detail and also
caused visible artifacts.

New Hot/Dead pixel removal will be two functions, which can be enabled separately.
The hot pixel detection will use the same algorithm as before. Dead pixel detection
will use a special (very simple) algorithm for the case zero_is_bad is set by dcraw
(mainly panasonic and Leica cams). For other cams an also very simple, but different
algorithm will be used for Dead pixel removal (It's a modified version of an algorithm
Martin Burri sent me by email and it works quite good in my tests, thanks Martin:).
I also changed the interpolation method for dead pixels in green channels of bayer
sensors. Old Interpolation used 8 pixels from 5x5 square to interpolate the value of
a dead pixel of a channel independent on the color of the channel. New version uses
12 pixels from 5x5 square for the green channel and 8 pixels for red and blue channels.

I'll try to post a new patch with all this changes including gui, pp3 and backwards
compatibility tomorrow. I'll give more detailed explanations when I submit the patch.

In a subsequent patch I'll make the changes for x-trans sensor.

Ingo

Reported by heckflosse@i-weyrich.de on 2014-08-17 20:59:42

[Beep6581]

Ilias, I tried your camconst.json entry to see the uniformity of the D700 sensor. It's
really impressive to see how the sensor is made from 4 parts http://www.i-weyrich.de/rt/D700_ingo.png
. Then I tried the camconst.json entry you posted in forum, but the result concerning
hot pixel detection was almost the same (only a few pixels difference). I'll investigate
further.
Thank you very much, Ilias ! I learn something new by every post you make to RT GC.

Ingo

Reported by heckflosse@i-weyrich.de on 2014-08-17 22:05:22

[Beep6581]

Update to #12: I'll need some more days for the patch, because I try some methods actually.

Ingo

Reported by heckflosse@i-weyrich.de on 2014-08-20 22:15:59

[Beep6581]

Reported by heckflosse@i-weyrich.de on 2014-09-03 18:52:15

• Status changed: Duplicate
• Merged into: #2469