How to improve image recognition

[LukeLR]

Hi everyone,

first things first: Thanks for this awesome project. I'm looking forward to read my completely analog power meter with this software.

However, I can't quite get it to work. I've been tweaking the settings for two days now, and I'm wondering what I'm missing. Currently, it looks like this:

As you can see, only one of the digits is recognized, and even that is incorrect. What are the general steps to improve recognition rate? Does the lighting look okay to you? Are the ROIs and the alignment areas plausible? Is the image sharp enough? Or are the digits from my power meter simply too different from what has been learned in the neural network?

As I'm quite new to this software, I'm not experienced in fixing such issues, so I wanted to reach out to the more experienced people to get some tips on how they would proceed. I'd gladly accept any recommedations, and I'm really looking forward to any ideas coming to your mind!

Thanks a lot in advance, LukeLR

[Kalli01]

maybe you can try to get closer to the counter with the camera. This way the digits would be bigger. The photo itself I find already good. Here as an example my photo

[jomjol]

You can check the setting of the ROIs with this more detailed explanation again: https://github.com/jomjol/AI-on-the-edge-device/wiki/ROI-Configuration

[makuser]

@LukeLR oh nice, you seem to have the same 'Zählwerk' (counter) in your meter as I do. I am currently in the process of collecting a bunch of images for each integer and submit them to the repo. I might need some input from @jomjol on how to do that though, now that it has been changed over to the new repo.

I picked out several reference images from my setup, that were regularily misidentified, added them to the training data set and trained the data on my machine. It works much better, so my hopes are that this will indeed help all of us with the same physical counter in our meters. I still have lots of issues with the integer 1, which just seems so off compared to the other current integers in the training data set.

@LukeLR let me know if you'd like to try out my tflite model. You can upload that manually via the file manager or via your computer directly onto the SD card and then pick that from the config file.

[makuser]

I have to say I have never seen a counter display this inaccurate... You can even see it on your picture, the 5 is higher, the 1 is also a bit higher and the 9 is much lower than the normal center position.

I now took several more pictures of them into the training set, but even then it really requires the MaxRate function in order not to yield completely wrong numbers. Unfortunately, sometimes even then a '1' gets recognized as a '2'.

[makuser]

@LukeLR if you like, you can test the model with this file.

Just unzip the file, and upload or store the tflite model file on your sdcard (in the config folder). dig-s2-q-iskra.zip

Then you can select the model in the configuration menu under Digits->Model (or in the config file directly)

[Digits]
Model = /config/dig-s2-q-iskra.tflite

After a reboot of the device, the new model will be used.

[LukeLR]

Hi, thanks for your helpful replies! @Kalli01 @jomjol: Your information regarding the camera and ROI settings helped me a lot. I have applied your tips by zooming in and adjusting the focus, and now it looks like this:

Thanks to @deadly667's explaination, I was able to add the ESP32 as a sensor in Home Assistant, giving me graphs like this:

As you can see, recognition isn't quite perfect yet: In the first screenshot, the 5 is often recognized as a 9, causing the entire reading to fail. At the last position, this is still within the allowed rate and therefore not recognized as an incorrect reading. However, all subsequent correct readings (6, 7, 8) are discarded because of negative rate, causing these bumps instead of a smooth curve in the diagram (second screenshot).

I've already decreased the allowed rate to 0.1 or 0.2 to mitigate this issue. However, when using much power e.g. by cooking, the last digit increases so fast that the ESP32 will not recognize the last digit correctly even just once once. In this case, subsequent correct readings divert by more than 0.1 or 0.2 from the previous correct reading, which in turn causes all following readings to fail. Setting the recognition timer to 0.5 minutes made this issue less frequent, but it still occurs, so I had to increase the allowed rate back to 1 and live with those bumps in the diagrams for now.

Therefore, I'm certainly interested in increasing the recognition performance. @makuser, nice to hear that you have the exact same power meter. I'll monitor the current setup a few days to get a good baseline, and then try your model to see if if it gives any improvement above the standard model. If you need more training images, I'd happily provide them.

Thanks a lot for the help so far, I'll keep you posted!

[makuser]

Actually now instead of the model I uploaded yesterday, you should rather check out the new model @jomjol trained right here: https://github.com/jomjol/neural-network-autotrain-digital-counter/pull/1#issuecomment-1050228236 I'll do the same and will comment on the PR about the success.

But in general, I had and to some degree still have the same issue as you do, but I think if the MaxRate is tuned to a sensible value, along with an improved tflite model for this type of counter, that should solve it for good.

Maybe a few adjustments/improvements need to be made to the logic that checks several things, one of them being the "Übertrag" of the counter, which was both a blessing and a curse at the same time. Eg if your meter is at 1.7 and then instead of 1.8 it recognizes a 1.3, you'll actually end up with 2.3 instead of an error and this one mistake will skew all of the future values, because once 1.9 is reached, it will assume 2.9 and then 3.0. But this issue should be part of a different discussion anyway.

So let's see how the newly trained model performs :)

[LukeLR]

Great, thanks for the hint! I'll try that and keep you posted!

[jomjol]

Then I'll close this issue - never the less you can post and I will read it.