jomjol / AI-on-the-edge-device

Easy to use device for connecting "old" measuring units (water, power, gas, ...) to the digital world
https://jomjol.github.io/AI-on-the-edge-device-docs/
5.75k stars 618 forks source link

Corrupted ROI images when ROI goes outside of image #3043

Open EwaldvanGeffen opened 5 months ago

EwaldvanGeffen commented 5 months ago

The Problem

In my case when my digit ROI delta Y exceeded beyond the bottom of the demo picture (so the bottom red horizontal box isn't visible because its outside the image). The image in image recognition for this digit got corrupted. I had other smaller sized digit ROI that would still function and not have visual corruption.

I think you need to limit digitroi.x+ digitroi.deltaX <= image.x and similar for y so it cannot go out of bounds.

Version

15.7

Logfile

[0d00h00m00s] 2024-04-20T00:14:15 <INF> [MAIN] =================================================
[0d00h00m00s] 2024-04-20T00:14:15 <INF> [MAIN] ==================== Start ======================
[0d00h00m00s] 2024-04-20T00:14:15 <INF> [MAIN] =================================================
[0d00h00m00s] 2024-04-20T00:14:15 <INF> [MAIN] PSRAM size: 8388608 byte (8MB / 64MBit)
[0d00h00m00s] 2024-04-20T00:14:15 <INF> [MAIN] Total heap: 4380199 byte
[0d00h00m04s] 2024-04-20T00:14:20 <INF> [MAIN] Camera info: PID: 0x26, VER: 0x42, MIDL: 0x7f, MIDH: 0xa2
[0d00h00m04s] 2024-04-20T00:14:20 <INF> [SDCARD] Basic R/W check started...
[0d00h00m04s] 2024-04-20T00:14:20 <INF> [SDCARD] Basic R/W check successful
[0d00h00m04s] 2024-04-20T00:14:20 <INF> [SNTP] TimeServer not defined, using default: pool.ntp.org
[0d00h00m04s] 2024-04-20T00:14:20 <INF> [SNTP] Configuring NTP Client...
[0d00h00m04s] 2024-04-20T02:14:20 <INF> [SNTP] Time zone set to CET-1CEST,M3.5.0,M10.5.0/3
[0d00h00m04s] 2024-04-20T02:14:20 <INF> [SNTP] time zone: +0200 Delta to UTC: 7200 seconds
[0d00h00m04s] 2024-04-20T02:14:20 <INF> [SNTP] Time is already set: 2024-04-20 02:14:20
[0d00h00m04s] 2024-04-20T02:14:20 <INF> [MAIN] CPU frequency: 160 MHz
[0d00h00m05s] 2024-04-20T02:14:20 <INF> [SDCARD] Folder/file presence check started...
[0d00h00m05s] 2024-04-20T02:14:20 <INF> [SDCARD] Folder/file presence check successful
[0d00h00m05s] 2024-04-20T02:14:20 <INF> [MAIN] Tag: 'v15.7.0', Release: v15.7.0 (Commit: 0d0b018+), Date/Time: 2024-02-17 00:15, Web UI: Release: v15.7.0 (Commit: 0d0b018+)
[0d00h00m05s] 2024-04-20T02:14:20 <INF> [MAIN] Reset reason: Via esp_restart
[0d00h00m05s] 2024-04-20T02:14:20 <INF> [WLANINI] SSID: m2m
[0d00h00m05s] 2024-04-20T02:14:20 <INF> [WLANINI] Password: XXXXXXXX
[0d00h00m05s] 2024-04-20T02:14:20 <INF> [WLANINI] RSSIThreshold: -75
[0d00h00m05s] 2024-04-20T02:14:20 <INF> [MAIN] WLAN config loaded, init WIFI...
[0d00h00m05s] 2024-04-20T02:14:20 <INF> [WIFI] Automatic interface config --> Use DHCP service
[0d00h00m05s] 2024-04-20T02:14:21 <INF> [WIFI] Set hostname to: iotatedge
[0d00h00m05s] 2024-04-20T02:14:21 <INF> [WIFI] Init successful
[0d00h00m07s] 2024-04-20T02:14:23 <INF> [WIFI] Connected to: wifi, RSSI: -68
[0d00h00m08s] 2024-04-20T02:14:24 <INF> [WIFI] Assigned IP: 10.20.1.300
[0d00h00m12s] 2024-04-20T02:14:27 <INF> [SNTP] Time is synced with NTP Server pool.ntp.org: 2024-04-20 02:14:27
[0d00h00m13s] 2024-04-20T02:14:29 <INF> [MAIN] Device info: CPU cores: 2, Chip revision: 301
[0d00h00m13s] 2024-04-20T02:14:29 <INF> [MAIN] SD card info: Name: SA16G, Capacity: 15193MB, Free: 15167MB
[0d00h00m15s] 2024-04-20T02:14:31 <INF> [MAIN] Initialization completed successfully
[0d00h00m17s] 2024-04-20T02:14:33 <INF> [LOGFILE] Set log level to DEBUG
[0d00h00m17s] 2024-04-20T02:14:33 <INF> [MAINCTRL] Starting Flow...
[0d00h00m17s] 2024-04-20T02:14:33 <DBG> [MAINCTRL] ----------------------------------------------------------------
[0d00h00m17s] 2024-04-20T02:14:33 <INF> [MAINCTRL] Round #1 started
[0d00h00m17s] 2024-04-20T02:14:33 <DBG> [FLOWCTRL] Status: Take Image (02:14:33)
[0d00h00m17s] 2024-04-20T02:14:33 <DBG> [PSRAM] Init shared memory for step 'Take Image' (STBI buffers)
[0d00h00m21s] 2024-04-20T02:14:37 <DBG> [PSRAM] Allocated 18456 bytes in PSRAM for 'STBI'
[0d00h00m21s] 2024-04-20T02:14:37 <DBG> [PSRAM] Freeing memory in PSRAM used for 'STBI'...
[0d00h00m21s] 2024-04-20T02:14:37 <DBG> [PSRAM] Allocated 18456 bytes in PSRAM for 'STBI'
[0d00h00m21s] 2024-04-20T02:14:37 <DBG> [PSRAM] Allocating memory (307215 bytes) for STBI (use shared memory in PSRAM)...
[0d00h00m21s] 2024-04-20T02:14:37 <DBG> [PSRAM] Allocating memory (153615 bytes) for STBI (use shared memory in PSRAM)...
[0d00h00m21s] 2024-04-20T02:14:37 <DBG> [PSRAM] Allocating memory (153615 bytes) for STBI (use shared memory in PSRAM)...
[0d00h00m21s] 2024-04-20T02:14:37 <DBG> [PSRAM] Allocated 643 bytes in PSRAM for 'STBI'
[0d00h00m21s] 2024-04-20T02:14:37 <DBG> [PSRAM] Allocated 643 bytes in PSRAM for 'STBI'
[0d00h00m21s] 2024-04-20T02:14:37 <DBG> [PSRAM] Allocated 643 bytes in PSRAM for 'STBI'
[0d00h00m21s] 2024-04-20T02:14:37 <DBG> [PSRAM] Allocating memory (921601 bytes) for STBI (use shared memory in PSRAM)...
[0d00h00m22s] 2024-04-20T02:14:38 <DBG> [PSRAM] Part of shared memory used for STBI (PSRAM, part of shared memory) is free again
[0d00h00m22s] 2024-04-20T02:14:38 <DBG> [PSRAM] Freeing memory in PSRAM used for 'STBI'...
[0d00h00m22s] 2024-04-20T02:14:38 <DBG> [PSRAM] Part of shared memory used for STBI (PSRAM, part of shared memory) is free again
[0d00h00m22s] 2024-04-20T02:14:38 <DBG> [PSRAM] Freeing memory in PSRAM used for 'STBI'...
[0d00h00m22s] 2024-04-20T02:14:38 <DBG> [PSRAM] Part of shared memory used for STBI (PSRAM, part of shared memory) is free again
[0d00h00m22s] 2024-04-20T02:14:38 <DBG> [PSRAM] Freeing memory in PSRAM used for 'STBI'...
[0d00h00m22s] 2024-04-20T02:14:38 <DBG> [PSRAM] Freeing memory in PSRAM used for 'STBI'...
[0d00h00m24s] 2024-04-20T02:14:40 <DBG> [C IMG BASIS] Not freeing (zwImage as there was never PSRAM allocated for it)
[0d00h00m24s] 2024-04-20T02:14:40 <DBG> [PSRAM] Deinit shared memory for step 'Take Image' (STBI buffers)
[0d00h00m24s] 2024-04-20T02:14:40 <DBG> [FLOWCTRL] Status: Aligning (02:14:40)
[0d00h00m25s] 2024-04-20T02:14:41 <DBG> [PSRAM] Allocating tmpImage (921600 bytes, use shared memory in PSRAM)...
[0d00h00m27s] 2024-04-20T02:14:43 <DBG> [PSRAM] Allocated 18456 bytes in PSRAM for 'STBI'
[0d00h00m27s] 2024-04-20T02:14:43 <DBG> [PSRAM] Freeing memory in PSRAM used for 'STBI'...
[0d00h00m27s] 2024-04-20T02:14:43 <DBG> [PSRAM] Allocated 18456 bytes in PSRAM for 'STBI'
[0d00h00m27s] 2024-04-20T02:14:43 <DBG> [PSRAM] Allocated 17935 bytes in PSRAM for 'STBI'
[0d00h00m27s] 2024-04-20T02:14:43 <DBG> [PSRAM] Allocated 4495 bytes in PSRAM for 'STBI'
[0d00h00m27s] 2024-04-20T02:14:43 <DBG> [PSRAM] Allocated 4495 bytes in PSRAM for 'STBI'
[0d00h00m27s] 2024-04-20T02:14:43 <DBG> [PSRAM] Allocated 105 bytes in PSRAM for 'STBI'
[0d00h00m27s] 2024-04-20T02:14:43 <DBG> [PSRAM] Allocated 105 bytes in PSRAM for 'STBI'
[0d00h00m27s] 2024-04-20T02:14:43 <DBG> [PSRAM] Allocated 105 bytes in PSRAM for 'STBI'
[0d00h00m27s] 2024-04-20T02:14:43 <DBG> [PSRAM] Allocated 44983 bytes in PSRAM for 'STBI'
[0d00h00m27s] 2024-04-20T02:14:43 <DBG> [PSRAM] Freeing memory in PSRAM used for 'STBI'...
[0d00h00m27s] 2024-04-20T02:14:43 <DBG> [PSRAM] Freeing memory in PSRAM used for 'STBI'...
[0d00h00m27s] 2024-04-20T02:14:43 <DBG> [PSRAM] Freeing memory in PSRAM used for 'STBI'...
[0d00h00m27s] 2024-04-20T02:14:43 <DBG> [PSRAM] Freeing memory in PSRAM used for 'STBI'...
[0d00h00m27s] 2024-04-20T02:14:43 <DBG> [PSRAM] Freeing memory in PSRAM used for 'STBI'...
[0d00h00m27s] 2024-04-20T02:14:43 <DBG> [PSRAM] Freeing memory in PSRAM used for 'STBI'...
[0d00h00m27s] 2024-04-20T02:14:43 <DBG> [PSRAM] Freeing memory in PSRAM used for 'STBI'...
[0d00h00m39s] 2024-04-20T02:14:54 <DBG> [OTA FILE] download_get_handler
[0d00h00m39s] 2024-04-20T02:14:55 <DBG> [MAIN SERVER] info_get_handler
[0d00h00m56s] 2024-04-20T02:15:11 <DBG> [PSRAM] Freeing memory in PSRAM used for 'STBI'...
[0d00h00m56s] 2024-04-20T02:15:11 <DBG> [PSRAM] Allocated 18456 bytes in PSRAM for 'STBI'
[0d00h00m56s] 2024-04-20T02:15:11 <DBG> [PSRAM] Freeing memory in PSRAM used for 'STBI'...
[0d00h00m56s] 2024-04-20T02:15:11 <DBG> [PSRAM] Allocated 18456 bytes in PSRAM for 'STBI'
[0d00h00m56s] 2024-04-20T02:15:11 <DBG> [PSRAM] Allocated 5135 bytes in PSRAM for 'STBI'
[0d00h00m56s] 2024-04-20T02:15:11 <DBG> [PSRAM] Allocated 1295 bytes in PSRAM for 'STBI'
[0d00h00m56s] 2024-04-20T02:15:11 <DBG> [PSRAM] Allocated 1295 bytes in PSRAM for 'STBI'
[0d00h00m56s] 2024-04-20T02:15:11 <DBG> [PSRAM] Allocated 78 bytes in PSRAM for 'STBI'
[0d00h00m56s] 2024-04-20T02:15:11 <DBG> [PSRAM] Allocated 78 bytes in PSRAM for 'STBI'
[0d00h00m56s] 2024-04-20T02:15:11 <DBG> [PSRAM] Allocated 78 bytes in PSRAM for 'STBI'
[0d00h00m56s] 2024-04-20T02:15:11 <DBG> [PSRAM] Allocated 11701 bytes in PSRAM for 'STBI'
[0d00h00m56s] 2024-04-20T02:15:11 <DBG> [PSRAM] Freeing memory in PSRAM used for 'STBI'...
[0d00h00m56s] 2024-04-20T02:15:11 <DBG> [PSRAM] Freeing memory in PSRAM used for 'STBI'...
[0d00h00m56s] 2024-04-20T02:15:11 <DBG> [PSRAM] Freeing memory in PSRAM used for 'STBI'...
[0d00h00m56s] 2024-04-20T02:15:11 <DBG> [PSRAM] Freeing memory in PSRAM used for 'STBI'...
[0d00h00m56s] 2024-04-20T02:15:11 <DBG> [PSRAM] Freeing memory in PSRAM used for 'STBI'...
[0d00h00m56s] 2024-04-20T02:15:12 <DBG> [PSRAM] Freeing memory in PSRAM used for 'STBI'...
[0d00h00m56s] 2024-04-20T02:15:12 <DBG> [PSRAM] Freeing memory in PSRAM used for 'STBI'...
[0d00h01m07s] 2024-04-20T02:15:23 <DBG> [OTA FILE] download_get_handler
[0d00h01m07s] 2024-04-20T02:15:23 <DBG> [MAIN SERVER] info_get_handler
[0d00h01m08s] 2024-04-20T02:15:23 <DBG> [PSRAM] Freeing memory in PSRAM used for 'STBI'...
[0d00h01m13s] 2024-04-20T02:15:29 <DBG> [PSRAM] Shared memory used for tmpImage (PSRAM, part of shared memory) is free again
[0d00h01m13s] 2024-04-20T02:15:29 <DBG> [FLOWCTRL] Status: Digitalization of ROIs (02:15:29)
[0d00h01m14s] 2024-04-20T02:15:30 <DBG> [OTA FILE] download_get_handler
[0d00h01m14s] 2024-04-20T02:15:30 <DBG> [CNN] doFlow after alignment
[0d00h01m14s] 2024-04-20T02:15:30 <DBG> [PSRAM] Allocating Tensor Arena (819200 bytes, use shared memory in PSRAM)...
[0d00h01m14s] 2024-04-20T02:15:30 <DBG> [TFLITE] CTfLiteClass::LoadModel
[0d00h01m14s] 2024-04-20T02:15:30 <DBG> [TFLITE] CTfLiteClass::ReadFileToModel: /sdcard/config/dig-class11_1700_s2.tflite
[0d00h01m14s] 2024-04-20T02:15:30 <DBG> [TFLITE] Loading Model /sdcard/config/dig-class11_1700_s2.tflite /size: 356488 bytes...
[0d00h01m14s] 2024-04-20T02:15:30 <DBG> [PSRAM] Allocating Model memory (1363148 bytes, use shared memory in PSRAM)...
[0d00h01m15s] 2024-04-20T02:15:31 <DBG> [MAIN SERVER] info_get_handler
[0d00h01m15s] 2024-04-20T02:15:31 <DBG> [TFLITE] CTfLiteClass::MakeAllocate
[0d00h01m15s] 2024-04-20T02:15:31 <DBG> [CNN] Processing Number 'main'
[0d00h01m15s] 2024-04-20T02:15:31 <DBG> [CNN] ROI #0 - TfLite
[0d00h01m15s] 2024-04-20T02:15:31 <DBG> [CNN] CNN Type: Digital
[0d00h01m16s] 2024-04-20T02:15:31 <DBG> [CNN] ROI #1 - TfLite
[0d00h01m16s] 2024-04-20T02:15:31 <DBG> [CNN] CNN Type: Digital
[0d00h01m16s] 2024-04-20T02:15:32 <DBG> [CNN] ROI #2 - TfLite
[0d00h01m16s] 2024-04-20T02:15:32 <DBG> [CNN] CNN Type: Digital
[0d00h01m17s] 2024-04-20T02:15:32 <DBG> [CNN] ROI #3 - TfLite
[0d00h01m17s] 2024-04-20T02:15:32 <DBG> [CNN] CNN Type: Digital
[0d00h01m17s] 2024-04-20T02:15:33 <DBG> [CNN] ROI #4 - TfLite
[0d00h01m17s] 2024-04-20T02:15:33 <DBG> [CNN] CNN Type: Digital
[0d00h01m18s] 2024-04-20T02:15:33 <DBG> [CNN] ROI #5 - TfLite
[0d00h01m18s] 2024-04-20T02:15:33 <DBG> [CNN] CNN Type: Digital
[0d00h01m18s] 2024-04-20T02:15:34 <DBG> [PSRAM] Shared memory used for Tensor Arena and model (PSRAM, part of shared memory) is free again
[0d00h01m18s] 2024-04-20T02:15:34 <DBG> [FLOWCTRL] Status: Post-Processing (02:15:34)
[0d00h01m18s] 2024-04-20T02:15:34 <DBG> [CNN] getReadout _analog=0, _extendedResolution=0, prev=-1
[0d00h01m18s] 2024-04-20T02:15:34 <DBG> [POSTPROC] handleAllowNegativeRate for device: main
[0d00h01m18s] 2024-04-20T02:15:34 <INF> [POSTPROC] main: Raw: 245.044, Value: 245.044, Status: no error
[0d00h01m18s] 2024-04-20T02:15:34 <INF> [MAINCTRL] Round #1 completed (61 seconds)
[0d00h01m18s] 2024-04-20T02:15:34 <DBG> [MAINCTRL] CPU Temperature: 53°C
[0d00h01m18s] 2024-04-20T02:15:34 <DBG> [MAINCTRL] WIFI Signal (RSSI): -68dBm
[0d00h01m20s] 2024-04-20T02:15:36 <DBG> [OTA FILE] download_get_handler
[0d00h01m20s] 2024-04-20T02:15:36 <DBG> [MAIN SERVER] info_get_handler
[0d00h03m42s] 2024-04-20T02:17:58 <DBG> [OTA FILE] download_get_handler
[0d00h03m42s] 2024-04-20T02:17:58 <DBG> [OTA FILE] download_get_handler
[0d00h05m17s] 2024-04-20T02:19:33 <DBG> [MAINCTRL] ----------------------------------------------------------------

Expected Behavior

no image corrutption

Screenshots

No response

Additional Context

15.7 has nice image processing features in terms of autoexposure, b/w and for me zoom. It is bad in converting configs from older versions, because model files dissappear? settings change and the wizard needs to be completed before it'll ever take a picture. But once you do each step individually and check the config file and reboot it works out. Not perfect, I think you need unit tests or something.

SybexX commented 5 months ago

try the current rolling, the zoom is better/expanded there. https://github.com/jomjol/AI-on-the-edge-device/actions/runs/8811491971

caco3 commented 1 month ago

Having the ROI partially outside of the image is a know reason for corrupted images. We should try to catch this.

I change it to an enhancement.