threshold option not working

[UPuXA]

Hello, i experimented a little bit with the threshold option and it seems that it has no effect whatsoever.

In my final test run i created 3 files with a threshold of 0.0, 0.5 and 1.0. And every time the result and the runtime was the same. I used cli.py and these options:

weights 1080p_medium_v8
blur_workers 5
blur_size 9
roi_multi 1
quality 5
feather_edges 10
batch_size 8
export_mask

I merged all my test files in one video with the different clips shifted by 40px to make them all visible (because they would be on top of each other). You can see that there is always 1 element for each threshold value and always in a triplet.

https://github.com/tfaehse/DashcamCleaner/assets/79371648/2db3bc8c-1e23-418f-ab56-ed3c4e045c17

For reference, this is the environment of the video.

[joshinils]

mh, I remember something about using the confidence as a pixel value for the mask output, ill have a look at the code.

[UPuXA]

There is this option which i didn't use for this comparison:

-mc   (Default: False)
--export_colored_mask
    Export a colored mask video of the blur-mask without applying it to the input clip.
    The value represents the confidence of the detector.
    Lower values mean less confidence, brighter colors mean more confidence.
    If the --threshold setting is larger than 0 then detections with a lower confidence are discarded.
    Channels; Red: Faces, Green: Numberplates.
    Hint: turn off --feather_edges by setting -fe=0 and turn --quality to 10

[joshinils]

uh, yeah, so...

https://github.com/tfaehse/DashcamCleaner/commit/543389444ad4d31672f0163e22621d630b7c1ead#diff-dc15d6e41ef6c6b5e2ae45f31eb3dd289b44ea277afa9438d0d8e4cb1ee90090 tfaehse removed the colored mask option, which also got rid of the use of the detection.score use in code, I don't see how this could have any influence on the mask export option now.

But, I don't yet know when the threshold value is used for the detector.

As I remember, the detector always spits out the same confidence for every detection, and only afterward are ones with a confidence which is too low thrown away. But I do not know if something has changed since I last contributed code.

[joshinils]

if you want, you can try this patch:

diff --git a/dashcamcleaner/src/blurrer.py b/dashcamcleaner/src/blurrer.py
index bfa3baf..ada2ba5 100644
--- a/dashcamcleaner/src/blurrer.py
+++ b/dashcamcleaner/src/blurrer.py
@@ -211,6 +211,8 @@ def apply_blur(frame: cv2.Mat, new_detections: List[Detection], parameters: Dict
     mask_color = [1, 1, 1]
     for detection in detections:
         bounds = detection.bounds.expand(frame.shape, feather_dilate_size)
+        if export_mask:
+            mask_color = [1, 1, 1] * detection.score
         if detection.kind == "plate":
             cv2.rectangle(blur_area, bounds.pt1(), bounds.pt2(), color=mask_color, thickness=-1)
         elif detection.kind == "face":

I hope that results in the brightness of the exported mask representing the confidence (detection.score) for each detection.

Just to sanity-check that the detections do have a score which is not always 1 (or very close to one, such that they are not visually distinguishable).

However I do not know where the threshold cut off actually happens, so I can't look at the piece of code which does that.

[UPuXA]

I needed to change mask_color = [1, 1, 1] * detection.score to mask_color = [*map(lambda c, t=detection.score: c*t, mask_color)]. The first one simply multiplies the lists length.

Sadly all 3 values stil behave the same:

https://github.com/tfaehse/DashcamCleaner/assets/79371648/21c61893-55f4-421a-836a-1cdddf60beb1

Edit:

Sadly all 3 values stil behave the same

I am a idiot, this was the more or less expected result.

[joshinils]

I needed to change mask_color = [1, 1, 1] * detection.score to mask_color = [*map(lambda c, t=detection.score: c*t, mask_color)]. The first one simply multiplies the lists length.

Yeah, I'm a dummy.

Now we have to ask @tfaehse why the threshold does not apply.

[tfaehse]

Thank you for reporting this @UPuXA! Looks like the newer ultralytics model class doesn't support setting .conf as an attribute anymore, you have to provide it in the predict() now. Fixed on main