film negative: add features to spot picker

[nomar500]

Hi everyone,

Environment: RT-5.8 dev, module "Film Negative", by @rom9 See forum discussion here posted here following my sad attempt 😆 in #6175

I suggest two evolutions to the spot picker in the Film Negative tool. Let me know if I should file two separate "Feature Request" cases.

A. Spot size:

• current behaviour: size is fixed (32*32 pixels square?)
• suggested behaviour:
  • size can be set from a drop-down list, just as in the current White Balance pipette tool.
  • spot can be made into an arbitrary rectangle, just as a selection rectangle (see darktable's negadoctor selection tools).

B. What the spot samples:

• current behaviour: spot area average pixel value is evaluated (to be confirmed?).
• suggested behaviour: due to the nature of the signal (scanned film, with grain, dust and scratches, crudely compared to salt-and-pepper noise), spot needs to be smooth:
  • (quick win?) spot returns the median pixel value;
  • (more complicated) to smoothen the spot area, its content first goes through a median filter (on the spot selection, with a certain kernel size, say 5 or 9) and then the average pixel value is sampled.

Let me know if something needs to be rephrased 😁

[nomar500]

Hmm, what if I filed a new feature request for the WB module spot picker (the pipette): it already has adjustable size, but I don't think I've found a reference explaining how it processes the square (Rectangle, if I read some of the code).

What do you think? Would that make it easier to derive from this class, further down the line, in Film Negative?

should the user be able to choose from several sampling algorithms, when using the WB spot picker?

[rom9]

Hi! Sorry for the delay on this, it's been a super busy year at work :_(

Hmm, what if I filed a new feature request for the WB module spot picker (the pipette)

There's already issue #709 somewhat related to this for normal, non-filmnegative WB (look at the last comment). That issue suggests using an arbitrary rectangular selection in Auto WB mode, which makes sense: this way, the user could choose any of the auto wb methods on the region.

it already has adjustable size, but I don't think I've found a reference explaining how it processes the square (Rectangle, if I read some of the code).

To me, the hard part is that, in order to map from the square spot on screen to the actual raw pixels, the sampling procedure needs to take into account all of the possible "distortion" tools applied (like the Perspective tool for example), so you might end up with a raw selection which is not rectangular. To solve this, the Spot WB tool fills a vector with the coordinates of each individual raw pixel to sample. See here. That said, going from a fixed 32x32 square, to an arbitrary selection (potentially as big as the whole image), this method could become quite memory and computation intensive...

What do you think? Would that make it easier to derive from this class, further down the line, in Film Negative?

No, unfortunately i don't think the same code could be shared between normal WB and film negative sampling, the problem to solve is completely different.

should the user be able to choose from several sampling algorithms, when using the WB spot picker?

do you mean, choosing between average or median or something else ?

[nomar500]

Hi @rom9, I understand. Busy, here too.

😄 #709 is of huuuge interest, to me, indeed.

off topic: in some way, especially if Film Neg inversion is sensitive (currently: from a default initial WB perspective) to black or washed out borders, then I want it to be able to reset the inversion based on some crop.

To me, the hard part is that, in order to map from the square spot on screen to the actual raw pixels, the sampling procedure needs to take into account all of the possible "distortion" tools applied (like the Perspective tool for example), so you might end up with a raw selection which is not rectangular.

Thanks for explaining that! quite insightful. So it's partly a pipeline question, I figure.

About the sampling algo: yes, that's what I mean; esp. when sampling from noisy images, I want to have something much more robust than a plain average value calculation. Right now, I'm not even sure how that arbitrarily sized square behaves. What kind of "average" is it computing? Is that even documented anywhere, in the inline code comments?

Maybe it does not have to be a sampling algo added to the picker... but a separate "blur" (avg, median, gaussian, you name it) module, correctly placed in the pipeline, just as a layer we can pick samples from.

[rom9]

Right now, I'm not even sure how that arbitrarily sized square behaves. What kind of "average" is it computing? Is that even documented anywhere, in the inline code comments?

Regarding the normal (non-filmneg) WB, i've checked the code and it seems to do a plain average (see the lines below this one for the Bayer part). The code loops through all the coordinates collected before, and seems to do a clever trick to also take in to account the surrounding pixels in the CFA (sort of a quick and dirty demosaic). But at its heart, you can see that this piece of code sums all the channel values and then divides by the value count, so it looks like an average.

Regarding the filmneg picker, yes, i confirm i'm doing a plain average :-)

Maybe it does not have to be a sampling algo added to the picker... but a separate "blur" (avg, median, gaussian, you name it) module, correctly placed in the pipeline, just as a layer we can pick samples from.

I don't know... i agree that upgrading from a plain average to a median could bring some benefits, but beyond that, do you think that also applying gaussian blur would really improve much over the median? Anyway, sampling from a blurred layer would not be the same thing as computing some statistics over a portion of the original image... think of picking a sample right next to an edge... in this case you want to read your values from the initial sharp image, because otherwise your sample would be biased by colors bleeding in from over the edge.

[nomar500]

hi @rom9, Thanks for reading it for me 👍 (not kidding)

i agree that upgrading from a plain average to a median could bring some benefits, but beyond that, do you think that also applying gaussian blur would really improve much over the median?

From a noise processing perspective, no, it's not essential (to me) to have something else than an adequately sized median box picker. I was merely mentioning the gaussian blur filter in the logic of making a cheaply denoised view that would nicely fit in the pipeline. Also, when applying a median to an entire image, depending on the kernel size, additional smoothing may be required (hence the gaussian blur addition).

think of picking a sample right next to an edge... in this case you want to read your values from the initial sharp image, because otherwise your sample would be biased by colors bleeding in from over the edge.

what kind of edge are you specifically thinking about? I think I understand your concerns about bleeding, though sometimes I'm counting on it (hello, black and white car license plate - makes for perfect greys, in theory). If however you're concerned with the image boundaries, then it's a different topic, related to "padding" (i.e. how do you populate the values of pixels right outside the border of your sliding window for convolution?) https://medium.com/dataseries/border-effects-and-transition-steps-in-convolutional-neural-networks-75bc6a44fd43 https://www.mathworks.com/help/images/imfilter-boundary-padding-options.html

(off topic: I thought you had concerns related to borders when it came to the initial median WB not being based on a crop, so maybe the links I shared will help)

🙂