Sherloq 0.89 Issues with auto_bright_thr=0

[JohnConnor01]

I have been testing the new version of Sherloq 0.89 after the recent changes. There are some observations I have noticed since "auto_bright_thr=0" has been added. From what I've seen, this is making it very difficult/impossible to to separate the signal and noise. My post will be in reference to the initial post. Initial Post: https://github.com/GuidoBartoli/sherloq/issues/86 Isolated patches were showing up on a photo, and this was being cited as areas affected by clipping. There was also a Histogram comparison being shown. As the RGB intensity values get pulled significantly to the left, it becomes harder and harder to separate the signal from the noise for specific images. So I wanted to adjust the settings to see if these areas still appeared when there was barely any clipping of the RGB intensity values (anything over 255 gets clipped & autobright is activated). I will show these examples below. First there are some important points to make: 1.) Adding this 1 setting, and seeing the patches go away. Does not mean clipping was causing the patches to appear in the first place. 2.) These very patches were in almost identical areas on this same photo in Dark Table - a completely separate application. I will illustrate this below. I also did not see the original poster mention the Dark Table results. I am not sure if he was unaware, but checking other applications for corroborating results would be one of the first things I would do in a situation like this. 3.) Autobright was not being called in Sherloq 0.87. Even though it was being cited as the reason for clipping, and subsequently the reason for patches showing up on a photo. To address this, auto_bright_thr=0 was added to Sherloq 0.89. This will potentially cause a wholesale shift in RGB intensity values of many photos read by Sherloq 0.89. And it will not prevent clipping with all RAW photos.

The next point is very important 4.) This is also going to default to the settings that were present in the original RAW File. Because in this case, the settings in the original RAW file were the thing that caused the RGB intensity values to shift to the left significantly. If I am trying to critically analyze a RAW file, why would I also want to 100% trust the RAW file? This is another reason why it would be great to have a user option.

This point is also very important. 5.) Adding auto_bright_thr=0 will NOT prevent clipping. It will prevent clipping in the photo (atadams) shared. But it will not prevent clipping universally. And this was implemented as a way to eliminate clipping- which it will clearly not do universally as highlighted below. Which makes a stronger case for a user option.

I used the same image from the original post. And I tried 3 different settings: auto_bright_thr=0.0015 auto_bright_thr=0.003 auto_bright_thr=0.005

As the RGB intensity values get pulled to the left - more and more noise is being introduced to the analysis. I will start with (auto_bright_thr=0) on Signal Separation so you can see the progression.

This is what the same image looked like on Sherloq 0.87 with Signal Separation & Bit Plane. These same areas showed up on Dark Table (below)- an entirely different application.

This is the same image compared with Sherloq 0.87 and Sherloq 0.89.

Below. I will show auto_bright_thr=0.0015. Signal Separation, Bit Plane, Histogram. As you can see even with 0.0015, the patchy areas still appear. Histogram on auto_bright_thr=0.0015 Now I will move to auto_bright_thr=0.003 Histogram on auto_bright_thr=0.003 Lastly, this is auto_bright_thr=0.005 Histogram on auto_bright_thr=0.005

I mentioned earlier that adding auto_bright_thr=0 and seeing these patches go away. Does NOT mean that clipping from an autobright setting was causing the patches to appear in the first place in Sherloq 0.87. There is a more plausible answer and that is auto_bright_thr=0 is pulling some photos RGB intensity values to the left and making it very difficult/impossible to separate the signal from the noise. This is simply causing the image to be flooded with noise, making forensic analysis impossible. (This does not happen with ALL RAW photos. And we have already demonstrated how numerous RAW photos will still be clipped with this setting)

Given there is a second application, Dark Table, that shows the same areas as Sherloq 0.87. And given how significant these changes have been to all photos analyzed with Sherloq. I would greatly appreciate if there could be a user option for this. (typo was corrected)

Thank you.

[Ray9T]

It only does that for your image, it's not the intended purpose of the setting. No one should disable critical auto brightness scaler to get full data, that is wrong.

Auto brightness isn't disabled. auto_bright_thr=0 only assures no clipping (I.e., data loss) takes place when auto brightness is applied.

This issue has over 40 comments. Don't you think it's been discussed enough?

Auto brightness isn't disabled

That's an erroneous statement, are you saying auto brightness is still working? Auto_bright_thr=0documentation says at this value, even if one pixel is outside the 255 range/clipped then DISABLE Auto brightness.

Now it's established that you misinterpreted the settings and applied the setting universally!

[atadams]

For the record please see that TWO other applications have corroborated the results.

This has been dismissed as nonsense by the person who opened this issue.

That is only corroboration that other apps can highlight the brightest 1% of pixels, not that, from a forensic standpoint, those pixels have any significance. That is why it is nonsense.

[Ray9T]

For the record please see that TWO other applications have corroborated the results. This has been dismissed as nonsense by the person who opened this issue.

That is only corroboration that other apps can highlight the brightest 1% of pixels, not that, from a forensic standpoint, those pixels have any significance. That is why it is nonsense.

Ah now you are indicating that there is a correlation between clipped pixels and patches! Which was asked many times and never provided. I also showed you via Raw therapee how the clipped pixels are not part of the patches.

Patches going away can be for many reasons,

[GuidoBartoli]

Wow, you guys wrote more than 60 comments in two days, it's really a lot and I'm just noticing it now when I get back from the weekend... My program had never received this much concentrated attention, and I don't know if that is good or bad 😆

There is too much to read and understand to quickly respond to you knowledgeably, so loosen up for a moment and give me a couple of days to analyze it all, do some testing, and draw some conclusions, okay?

However, I thank you all anyway for taking the time to use and test Sherloq, I'm really pleased about this, it's in the spirit of open-source development!

Some initial notes based only on my perception:

@JohnConnor01, @SacredCowBBQ, @BakersTutz, @Ray9T, @vincent-wmtm, @TJPofTexas, @GreatRevealer, @NaturalML, all of you created your accounts just for commenting here and none of you has ever committed a single line of code in Github (are you the same person?). I hope your findings are meaningful and useful, not a waste of time for the community, note that some basic reputation and credibility are needed to take this kind of criticism seriously, whether constructive or not.

@atadams, @PeterWem thanks for replicating results, your politeness and objectiveness is appreciated.

[JohnConnor01]

Thank you for your response @GuidoBartoli

We are happy to create a more streamlined version of the points made. We are simply looking for a user option. I understand going through threads like this takes time and I greatly appreciate you getting back with us.

[GuidoBartoli]

Okay, here I am after analyzing this very long thread that basically just revolves around what options to choose when importing RAW files via the rawpy library.

Here are my conclusions on the matter:

• I have done some personal research on the people involved in this discussion and I think I have a pretty accurate idea of ​​the reality of the facts, but this in no way influences the technical decisions regarding development on Sherloq, which I would like to remain a valid open source tool useful for carry out research in the field of forensic analysis of digital images.
• Regarding standard raster images (JPG, PNG, TIF, ...), the program simply reads the RGB values present so all subsequent analysis is consistent with the original content. RAW images, on the other hand, contain only sensor light readings that require post-processing to obtain the "final" RGB values, which depend on many steps (demosaicing, denoising, gamma correction, white balance, black levels, and so on...).
• A forensic analysis on RAW images is in itself very rare and peculiar, because potential alterations are in almost all cases carried out by programs working in RGB, so I had initially thought of completely removing the possibility of importing RAWs, but then I felt that it could still be useful in some cases, so I decided to keep it.
• In this scenario in which post-processing of the RAW data is still necessary to obtain the RGB data on which to perform subsequent analysis, I think that the best compromise is to reduce it to a minimum to avoid biased results.
• A premise must be made: if a photo was taken overexposed (or underexposed), balancing the exposure from the RAW is used to recover the areas where clipping occurred (which is not possible with 8-bit raster images due to the much smaller dynamic range); this is a common practice of photographers (myself included) and totally legit.
• That said, assessing the presence of potential digital manipulation based on the RAW preprocessor's ability to recover these artifacts makes absolutely no sense from a forensic researcher's point of view, so I kindly ask you to avoid posting additional images comparing the effect of the different options on the clipped areas.
• I looked at the rawpy default options and saw that to minimize post-processing operations, it is necessary to disable auto brightness adjustment, auto white balance and pixel value scaling, so please take a look at the update on pca.py inside commit 5c8684a1072f6d2ea2bebb816eab914a3fd6707e where you can find these new behavior together with some more Black formatted code for the whole project.