Bogus raw value resulting from dark image assumed as valid measurement

teixeluis commented 1 week ago

The Problem

I found a couple of times in a 2 week timespan that sometimes the images are captured when the light is off. It appears to be some kind of synchronization issue between the camera and the LED light. This results in a black picture being captures.

On the other hand, instead of the software invalidating a picture in these conditions, it goes through the recognition cycle which results in a raw value always in the vicinity of 444.44 m3.

In one of the occurrences this raw value stood like that for 3 hours until the correct value was again recognized:

Still, even though I have the following rate validation,

main.MaxRateValue = 0.040
main.MaxRateType = RateChange

the bogus raw value ended up being assumed, causing the measurement to be incremented from 394.43 to 444.44:

Then after this hallucination period ended for the camera, the correct raw value was again recognized, but being a negative rate, it was no longer accepted:

Would it make sense to fail fast on black images or without any match with the markers, therefore avoiding these errors?

Also, is there anything it can be done regarding the LED light turning on everytime a capture is taken?

Thanks

Version

Development-Branch: main (Commit: 57db60f+)

Logfile

[9d07h23m15s] 2024-10-23T00:02:06 <INF> [TFLITE] Trying to load the model. If it crashes here, it ist most likely due to a corrupted model!
[9d07h24m21s] 2024-10-23T00:03:13 <ERR> [POSTPROC] main: Raw: 394.423, Value: , Status: Neg. Rate - Read: - Raw: 394.423 - Pre: 444.438
[9d07h24m22s] 2024-10-23T00:03:13 <INF> [MAINCTRL] Round #4468 completed (165 seconds)
[9d07h24m37s] 2024-10-23T00:03:28 <INF> [MAINCTRL] Round #4469 started
[9d07h26m15s] 2024-10-23T00:05:06 <INF> [TFLITE] Trying to load the model. If it crashes here, it ist most likely due to a corrupted model!
[9d07h26m31s] 2024-10-23T00:05:22 <ERR> [POSTPROC] main: Raw: 394.423, Value: , Status: Neg. Rate - Read: - Raw: 394.423 - Pre: 444.438
[9d07h26m31s] 2024-10-23T00:05:22 <INF> [MAINCTRL] Round #4469 completed (114 seconds)
[9d07h27m37s] 2024-10-23T00:06:28 <INF> [MAINCTRL] Round #4470 started
[9d07h29m15s] 2024-10-23T00:08:06 <INF> [TFLITE] Trying to load the model. If it crashes here, it ist most likely due to a corrupted model!
[9d07h29m31s] 2024-10-23T00:08:22 <ERR> [POSTPROC] main: Raw: 394.423, Value: , Status: Neg. Rate - Read: - Raw: 394.423 - Pre: 444.438
[9d07h29m31s] 2024-10-23T00:08:22 <INF> [MAINCTRL] Round #4470 completed (114 seconds)
[9d07h30m37s] 2024-10-23T00:09:28 <INF> [MAINCTRL] Round #4471 started
[9d07h31m15s] 2024-10-23T00:10:07 <INF> [SNTP] Time is synced with NTP Server pool.ntp.org: 2024-10-23 00:10:07

Expected Behavior

Bogus raw values (444.44) resulting from very poor images be ignored/not have a chance of being used for measurements, or ignore dark or otherwise invalid images (i.e. not containing digits or aligment markers);
MaxRateValue being always observed, preventing large measurement jumps from being accepted;
make sure the LED light is turned on in every capture.

Screenshots

No response

Additional Context

No response

SybexX commented 6 days ago

What did you set at preValueAgeStartup, not coincidentally 180 minutes? If preValue is older than preValueAgeStartup, it will be reset.

You may have to increase "Wait Before Taking Picture" a little because your camera needs some time to get the picture bright and sharp. Experience has shown that values between 3 and 5 seconds are ok. The higher the value, the better the image quality, as the auto functions (Auto Exposure Control............) take a certain amount of time. However, too long a time also has disadvantages (the ESP gets hotter...).

teixeluis commented 4 days ago

What did you set at preValueAgeStartup, not coincidentally 180 minutes? If preValue is older than preValueAgeStartup, it will be reset.

No, currently I have these settings:

[PostProcessing]
main.DecimalShift = -3
;main.AnalogToDigitTransitionStart = 9.2
;main.ChangeRateThreshold = 2
PreValueUse = true
PreValueAgeStartup = 43200
main.AllowNegativeRates = false
main.MaxRateValue = 0.040
main.MaxRateType = RateChange
main.ExtendedResolution = false
main.IgnoreLeadingNaN = false
ErrorMessage = true
CheckDigitIncreaseConsistency = true

You may have to increase "Wait Before Taking Picture" a little because your camera needs some time to get the picture bright and sharp. Experience has shown that values between 3 and 5 seconds are ok.

Currently 5 seconds:

WaitBeforeTakingPicture = 5

But I will jack it up to see if it increases the performance.

Another issue though, is that in spite of using the new model (built after I submitted my digits), it appears to still be unrealiable with some digits, such as digit 5 as the first decimal:

teixeluis commented 4 days ago

My ROI seem ok..

teixeluis commented 4 days ago

Ok..meanwhile by switching from

dig-cont_0710_s3_q.tflite to dig-cont_0711_s3_q.tflite

it is looking better:

teixeluis commented 4 days ago

Even having increased the camera delay to 8 seconds, sometimes the image is dark:

In this case however it did not interpret as 444.44, which is good.

[0d00h13m54s] 2024-10-26T12:32:17 <INF> [TFLITE] Trying to load the model. If it crashes here, it ist most likely due to a corrupted model!
[0d00h13m56s] 2024-10-26T12:32:20 <WRN> [CNN] Value Rejected due to Threshold (Fit: 0.296875, Threshold: 0.500000)
[0d00h13m59s] 2024-10-26T12:32:23 <WRN> [CNN] Value Rejected due to Threshold (Fit: 0.398438, Threshold: 0.500000)
[0d00h14m02s] 2024-10-26T12:32:25 <WRN> [CNN] Value Rejected due to Threshold (Fit: 0.421875, Threshold: 0.500000)
[0d00h14m05s] 2024-10-26T12:32:28 <WRN> [CNN] Value Rejected due to Threshold (Fit: 0.304688, Threshold: 0.500000)
[0d00h14m08s] 2024-10-26T12:32:31 <WRN> [CNN] Value Rejected due to Threshold (Fit: 0.292969, Threshold: 0.500000)
[0d00h14m11s] 2024-10-26T12:32:34 <INF> [POSTPROC] main: Raw: NNN.NN3, Value: 394.503, Status: no error
[0d00h14m11s] 2024-10-26T12:32:34 <INF> [MAINCTRL] Round #5 completed (118 seconds)
[0d00h15m13s] 2024-10-26T12:33:37 <INF> [MAINCTRL] Round #6 started

teixeluis commented 4 days ago

Could tuning of the CNNGoodThreshold help in weeding out these false recognitions (the values seem too close to the threshold in spite of being pitch black images)?

SybexX commented 4 days ago

Does the Live Stream (Light on) work, or do you just get a black picture there? I'm starting to suspect that your camera is getting too hot or is defective.

teixeluis commented 4 days ago

Does the Live Stream (Light on) work, or do you just get a black picture there? I'm starting to suspect that your camera is getting too hot or is defective.

Was not able to reproduce during live stream. Only dark for the first couple of seconds.

teixeluis commented 4 days ago

Wonder if it can be the cause, but while I was trying to reproduce the issue by opening the livestream several times, in the digitalization round I got another dark picture.

Can there be some kind of race condition between the livestream and the recognition loop, causing the dark image when there is concurrent access to the camera? I ask this because I have an integration with Home Assistant to show the camera stream..

SybexX commented 4 days ago

If you end the LiveStream at the wrong time, the LED goes out during the TakeImage phase or shortly before and then of course there are black images. Likewise, if you run the LiveStream during the TakeImage phase, the LiveStream will turn black because the LED will turn off after the TakeImage phase.

jomjol / AI-on-the-edge-device