Support for Pentax pixel shift files

heckflosse commented 7 years ago

Pentax K70 supports pixel shift mode. In this mode 4 shifted frames are recorded and available in the raw file. How it works In first step we could allow access to each of the frames in rt. That's easy, as dcraw already supports that. We would need a frame selector somewhere in gui of the raw demosaic tab and pass the value (1,2,3,4) to dcraw.

I will make a new branch with a prototype to allow testing.

In second step we can define a new 'demosaic' method pixel shift which combines the 4 frames to one image.

iliasg commented 7 years ago

May be this will help .. https://github.com/tomtor/dcrawps

heckflosse commented 7 years ago

@iliasg :+1:

heckflosse commented 7 years ago

@Hombre57 Can you help me with the gui part (a combobox somewhere in raw processing tab to select the number of the frame) when I pushed my changes to allow extraction of individual frames to a new branch (already coded and tested)?

Ingo

heckflosse commented 7 years ago

Prepared engine to handle individual frames

Hombre57 commented 7 years ago

@heckflosse

Sure ! Should I add the pp3 parameter as well, or will you do it ?

Hombre57 commented 7 years ago

Btw, could you look at the files from Fuji S4/S5 bodies, where 2 images are embedded with different sensor sensibility, it should work the same way. I remember having made some test, and the change was rather trivial.

heckflosse commented 7 years ago

@Hombre57 Great, yes, please add pp3 parameter as well. I'll have a look at S4/S5 too

heckflosse commented 7 years ago

@Hombre57 I pushed two bug fixes. Now it's possible to extract 2nd frame for Fuji S5Pro (did not test yet with S4 RAF)

heckflosse commented 7 years ago

@Hombre57 please don't pull pixelshift atm. I have to make a few corrections to get it working correctly for Fuji cams

heckflosse commented 7 years ago

@Hombre57 Now it should work correctly. I had to fix a bug in dcraw.

heckflosse commented 7 years ago

@Hombre57 Thanks for the gui mockup. I just fixed the segfault we talked about in irc. There's still one issue remaining: We have not only to demosaic from scratch after changing the frame number, but also to load the file again (iirc there's no Ev... for this case atm)

heckflosse commented 7 years ago

@Hombre57 Also frame number is not passed to load function resulting in always loading frame 0

Hombre57 commented 7 years ago

@heckflosse Will check that tonight.

We have not only to demosaic from scratch after changing the frame number

Isn't this what it's supposed to do ?

but also to load the file again

I guess that the file is closed once the (sub-)image has been loaded, why keeping it open ? and in your case, when to close it ?

heckflosse commented 7 years ago

@Hombre57 The problem is, that the event EvRawImageNum which is triggered when you select an Image number, does not load the raw file again, but it has to.

heckflosse commented 7 years ago

@Hombre57 Please wait with your modifications. I make some modifications first ...

heckflosse commented 7 years ago

@iliasg Ilias, do you have some insight into pixelshift files? I tried with a pixelshift file from K-70. Dcraw reports 4 frames and I can extract 4 frames too. The first 3 frames are different (as it should be) but the 4th frame is identical to the 1st one. This also fits to the size of the file which is about 3 times the size of one normal uncompressed file.

heckflosse commented 7 years ago

@iliasg Please ignore my last comment :)

heckflosse commented 7 years ago

I updated the pixelshift branch. Now it's possible to view in editor and to process individual frames of raw files which contain more than one frame (i.e. Pentax pixelshift and Pentax hdr). For Fuji S5 files I kind of disabled it atm because 1st and 2nd shot have different dimensions which is not solved yet and would crash. That does not mean that Fuji S5 is not supported. It's supported the same way as in master branch.

heckflosse commented 7 years ago

I made a hack to test pixelshift implementation. I'll push later today. Here's a screenshot. Left is amaze, right is pixelshift. Edit: Here's another screenshot

heckflosse commented 7 years ago

I pushed to pixelshift branch. Now there is a new demosaic method pixelshift simple available for bayer sensors. If you select it on a non pixelshift file it falls back to amaze demosaic automatically. At the moment it completely bypasses raw preprocessing and simply combines the four sub frames without checking for motion. It's only for testing, code is also not clean.

heckflosse commented 7 years ago

I tested with a few other images. While the result I got with the image I used for the screen shots above is good, there are images where the result is bad. Seems like the shift is not always the same. I'll investigate.

Edit: Or the other shots I tested are no tripos shots...

iliasg commented 7 years ago

Ingo, could we have a user controlable way of defining the shift order ?

heckflosse commented 7 years ago

Ilias, yes, that can be done if really necessary. Isn't that order fixed? Currently I'm not sure. Maybe my test images were shot without tripod....

heckflosse commented 7 years ago

@iliasg Can you make a camconst.json section for Pentax K-1 files? Currently the PEF files from that cam have wrong crop.

iliasg commented 7 years ago

@heckflosse Ingo, OK .. it will ready in a few minutes if you are in a hurry (only color matrix and raw crop) or a few hours to also fill proper White Levels ..

heckflosse commented 7 years ago

@iliasg no Hurry, Thank you :+1:

heckflosse commented 7 years ago

I pushed some changes to pixelshift branch. I fixed a bug, implemented parallel decode of frames and cleaned the code a bit. No change to algorithm this time.

iliasg commented 7 years ago

No hurry meaned 5 days :( https://github.com/Beep6581/RawTherapee/issues/3298#issuecomment-259758936

heckflosse commented 7 years ago

In case you want to follow the current pixelshift development

Hombre57 commented 7 years ago

Updated GUI done. Sorry, I've forgot to reference the issue number. See commit b1e7dcb.

the new Motion Correction combobox needs a better tooltip
the History messages needs to be set correctly in 'default' language files
the Fast Export and Partial Paste feature are not updated yet, will be done before the finale merge
the show/hide enable/disable logic seem quite broken, I'll look at this tomorrow, unless you fix it yourself
the pixelshift method probably need to be updated to take care of the new enum

heckflosse commented 7 years ago

@Hombre57 Thanks a lot :+1: I pushed the enum related changes

Desmis commented 7 years ago

I read the 2 texts (in Pixls.us and issue 3489)...and download some images.... I am not sure to all well understand.

1) I cannot download first files in Pixls.us - they are expired : IMPG0003.dng, etc. 2) one of the problems is to detect motion...and noise

The noise does not correspond to a true Gaussian distribution - one of the best noise evaluation is MAD (with wavelet, see Ftblockdn.cc). In fact the curve in bell is very deformed, to the left of the average the curve is compact, and right very dilated. This means that using sigma is not very appropriate in this case, normally with 3 sigma we get about 99.5% of the data, in the case of noise it will be much less, especially in the reds. Mathematically this results in high KHI2 values. Perhaps can we reduce the noise by a Median (at least 7x7 for chroma...).

But the way in which the problem is approached seems to me to be relevant :)

for motion, our eyes are very sensitive to luminance, which in the case of images close to the raw, is fairly well represented by the green channels

In branch newwavelet for "merge HDR", I use wavelet to merge images, with suppression of the lowest levels on one of the images, and keep them for the other, at the choice of the user. By adjusting the number of levels difference, the motion problem is solved: nothing, 1, 2, 4 pixels, etc. But here, it seems to me, that wavelet will be complex, and will cost a lot of processing time and memory

jacques

Desmis commented 7 years ago

Oh I forgot, congratulations for this great work :)

iliasg commented 7 years ago

Jacques, thanks for the review ..

In fact the curve in bell is very deformed, to the left of the average the curve is compact, and right very dilated. This means that using sigma is not very appropriate in this case, normally with 3 sigma we get about 99.5% of the data, in the case of noise it will be much less, especially in the reds. Mathematically this results in high KHI2 values.

Why "especially on the reds" ?. We work on raw data without WB applied

This bell curve deformation only happens for very low levels and I think it is not significant .. It comes from ..

the nature of the photon noise ( Poisson distribution)
raw-denoise operations (like removing outliers)
clipping of raw values at Black level (K-1/K70 do not clip at low ISOs clip slightly at high ISOs, K3II clips everywhere)

We have 3 noise sources in our stdev calculation

Photon noise which follows Poisson Distribution .. Poisson distribution is very-very close to Gaussian for electron counts > 10 .. even for λ=5 it's close .. 5 is very-very dark relative to Full Well of around 50000 electrons fro ISO100
Read noise which follows Gaussian distribution by nature .. except if raw_denoise and/or clipping occur. Read noise is significant only for very low levels for ISO100 a bit more as ISO rises .. We already take care of raw_values_clipping at Black Level and possible raw_denoise with our measures of read noise (black frames) by using only the positive values, reconstructing the distribution by mirroring and measure stdev on the reconstructed data. We talked with Ingo about clipping at BlackLevel in RT and he says that RT doen't clip below BlackLevel values if they exist (i.e. if they are not already clipped by the camera).
Pixel Responce Non Uniformity which is again gaussian_like but with fatter tails but it is symmetrical
The Central Limit Theorem says

when independent random variables are added, their sum tends toward a normal distribution (commonly known as a bell curve) even if the original variables themselves are not normally distributed

So I think we would be fine with supposing a gaussian disribution and relly on stdev as criterion .. if we have a robust estimation of mean. The real problem is that we have no robust estimation of mean, because for Green differences we only have two measures at each place of the grid and we use their average as mean. Unterestimating the mean gives us smaller stdev and this drives to overestimation of move i.e. we detect movement (or light change) when in fact the difference between the two greens comes from natural noise. The inverse happens when we overestimate the mean ..

If we could have a mathematical approach for predicting (and couteract accordigly) the mean_error we would be in better shape. You mentioned chi_square .. can we use this (or any other statistic tool) somehow for better mean estimation ?. Or any other method like MAD .. I am not good with statistics so I need help from a math/stats guy on this. If not, I am inclined to additionally use data from the neighbouring pixels .. either a weighted average or maybe median of 3X3 grid area or median of 3X3 cross

ByTheWay .. we can fine tune the stdev responce parametrically in the current GUI .. increase the read noise slider to decrease the response at the darks, increase PRNU to decrease the responce at highlights

normally with 3 sigma we get about 99.5% of the data

This is the probability of a sample being within -1.5 to + 1.5 stdevs .. we don't use this but the probability of a pair of samples to differ by more than 3 stdevs. I calculated this in excel by using 0.1 stdev wide sectors and calculate the sum of probabilities for our pairs .. it's around 96.5% for sigma = 3.0 stdevs .. So I think the default criterion is fine at 3.0 stdevs .. if the difference of a pair is more than 3.0 stdevs then motion is detected.

Desmis commented 7 years ago

We will not fight like ragpickers, about the problem of noise. :) I know the different types of noise for having worked deeply the algorithm of Denoise

Although the photon distribution is Gaussian, but when examining an entire image with over exposed areas, under exposed areas (with large amplitudes of dynamics - up to 13IL) and color differences, there are significant variations in total noise.

Each color photon has an energy which depends on the frequency according to Planck's formula. E= h*f (h planck constant, f frequency). Which brings about a difference of energy which strikes the sensor according to the color. The relationship between blue (sky) and red (flower) areas can reach very high values, which take into account all the noises due to photons, exposure, etc. on images that appear not very noisy !(eg colorspace_flowers.pef where maxima are between 10 and 300 for MAD) To check this try "Noise reduction" - "preview" on different areas (zoom about 500% or more)of an image. This ratio can reach a value of 30 (or more) for the maxima.

With 3 standard deviations, if the distribution is normal, exactly 99,865% of the total population is covered, between the beginning of the left part of the curve and the right part with 3 sigma.

I did not look at the code in detail, but it seems important to be able to differentiate the sigma as a function of the real noise (reminder: that on an "normal" image has maxima that vary in a ratio of 1 to 30). But this is another matter, which involves evaluating the noise in a very small area around the pixel to be processed... Maybe it is that realizes the current code, but I do not know I only started looking at it since last night.

Go further and use local KHI2 (which is statistical) or MAD emerges from complex problems, which risk making the constants configurable, considerably increase complexity and processing times. In this case (of complexity) why not use wavelet which are fantastic tools for analyzing and processing noise, and isolating parts of images (1, 2 , 4 pixels)...but it will be very very complex...and time processing ??? jacques

heckflosse commented 7 years ago

Status report:

We are currently working hard on some new methods to reduce the number of false motion detections while increasing the number of true motion detections. As soon as the the stuff is worth to be pushed to pixelshift branch I will do that...

Desmis commented 7 years ago

Happy holidays to everyone :)

heckflosse commented 7 years ago

@Desmis Thank you. Happy holidays to you as well.

heckflosse commented 7 years ago

I pushed a lot of new stuff to pixelshift branch and André already made a win64 build (the last in the list).

Hombre57 commented 7 years ago

I don't know what the gazillions of options does and how to use them but still, I managed to have very detailed images with this new feature, so thanks a lot @heckflosse and @iliasg for have done such a great tool.

Can I ask what the "Median" option does ?

heckflosse commented 7 years ago

@Hombre57

I don't know what the gazillions of options does and how to use them

I will write a tutorial soon. Some of the options surely will not be in the final release.

Can I ask what the "Median" option does ?

Isn't the tooltip understandable?

Use median of all frames instead of selected frame for regions with motion.
Removes objects which are at different places in all frames.
Gives motion effect on slow moving (overlapping) objects.

One use case for 'Median' are shots with water (e.g. the waterfall example from dpreview)

heckflosse commented 7 years ago

@Hombre57

Left is without Median using 4th frame for regions with motion and right is using Median: ps_waterfall_median

Hombre57 commented 7 years ago

First, I didn't even looked for a tooltip, sorry for that. But even with the tooltip, I couldn't imaging the use case (in fact I don't fully understand the tooltip, but once translated it should be ;) ).

That median feature is in itself a very cool feature ! I guess that someone will ask to add that feature even for non PS images within a week after the Release :)

Beep6581 commented 7 years ago

Some feature requests based on latest dev 5.0-r1-gtk3-259-g2228159f:

The Editor Toolbox now shows a "Sub-image" combo with values 1-4, even when I edit a plain old raw file. It would be nice if this combo was hidden or grayed-out when editing an image without any sub-images, and if the numbers it shows corresponded to the sub-images the image contains.
The File Browser Toolbox shows all the new Pixel Shift options in the "Sensor with Bayer Matrix > Demosaicing" frame. It would be cleaner and more user-friendly if all of these went into a "Pixel Shift" frame (or some other generic word if these things are useful for more than just Pixel Shift files). Either a sub-frame in "Demosaicing" or a new "Pixel Shift" frame under "Demosaicing".
We should spell this feature consistently. Pentax calls the technology "Pixel Shift", so if we use that name then it should be "Pixel Shift" everywhere.

heckflosse commented 7 years ago

@Beep6581 The spelling problem can be solved by making the demosaic method names translatable. Currently they are not translatable and the method name is also used as value in pp3 file. Making them translatable would also allow to translate 'none' method and use 'AMaZE' instead of 'amaze' ;)

I will take that part now.

Beep6581 commented 7 years ago

@heckflosse :+1: I can help clean up default - let me know when is a good time.

heckflosse commented 7 years ago

@Beep6581 I'm already working on that and also on the Sub-Image issue. I will let you know when it's ready.

heckflosse commented 7 years ago

@Beep6581 Status report: I made the changes for translation of demosaic methods and also for sub-image selector locally and they work fine. I want to use this opportunity to also hide xtrans gui-frame for bayer files and bayer gui-frame for xtrans files. What do you think?

heckflosse commented 7 years ago

Status report: Now also hiding xtrans frame when bayer files are opened and hiding bayer frame when xtrans files are opened works fine here. I will push tomorrow.

heckflosse commented 7 years ago

Further update: Now also hiding the whole raw tab when opening a non-raw file works :)

Beep6581 / RawTherapee

Support for Pentax pixel shift files #3489