Camera Framebuffer cannot be initialized with v13.0.5 and latest rolling

jomjol / AI-on-the-edge-device

Easy to use device for connecting "old" measuring units (water, power, gas, ...) to the digital world

https://jomjol.github.io/AI-on-the-edge-device-docs/

6.12k stars 658 forks source link

Camera Framebuffer cannot be initialized with v13.0.5 and latest rolling #1553

Closed DanielWe2 closed 1 year ago

DanielWe2 commented 2 years ago

The Problem

Hello,

I was using rolling a53188e (from November) without an issue. Today I tried to update to v13.0.5.

But with the new version I am getting "Camera Framebuffer cannot be initialized" (same as #1513).

As that issue is closed I tried the latest rolling release (3387315) but that seems to fail with the same problem.

I reverted back to the old rolling a53188e and everything works fine again.

Version

v13.0.5 and a53188e

Logfile

[       4] 2022-12-12T01:30:48 <INF> [MAIN] ==================== Startup ====================
[       4] 2022-12-12T01:30:48 <INF> [MAIN] =================================================
[       4] 2022-12-12T01:30:48 <INF> [MAIN] Branch: 'HEAD', Revision: e23c89a, Date/Time: 2022-12-06 19:34, Web UI: Release: v13.0.5 (Commit: e23c89a)
[       4] 2022-12-12T01:30:48 <INF> [MAIN] Reset reason: Via esp_restart
[      10] 2022-12-12T01:30:54 <INF> [MAIN] =================================================
[      10] 2022-12-12T01:30:54 <INF> [MAIN] ================== Main Started =================
[      10] 2022-12-12T01:30:54 <INF> [MAIN] =================================================
[      14] 2022-12-12T01:30:58 <ERR> [MAIN] Camera Framebuffer cannot be initialized!
[      16] 2022-12-12T01:31:00 <ERR> [MAIN] Initialization failed. Will restart in 5 minutes!
[     241] 2022-12-12T01:34:44 <INF> [OTA] Update for reboot.
[     256] 2022-12-12T01:35:00 <ERR> [MAIN] Initialization failed. Will restart in 1 minute!
[     316] 2022-12-12T01:36:00 <ERR> [MAIN] Initialization failed. Will restart now!
[     316] 2022-12-12T01:36:00 <INF> [OTA] Reboot triggered by Software (5s).
[     316] 2022-12-12T01:36:00 <WRN> [OTA] Reboot in 5sec
[       4] 2022-12-12T02:35:42 <INF> [MAIN] =================================================
[       4] 2022-12-12T02:35:42 <INF> [MAIN] ==================== Startup ====================
[       4] 2022-12-12T02:35:42 <INF> [MAIN] =================================================
[       4] 2022-12-12T02:35:42 <INF> [MAIN] Branch: 'HEAD', Revision: e23c89a, Date/Time: 2022-12-06 19:34, Web UI: Release: v13.0.5 (Commit: e23c89a)
[       4] 2022-12-12T02:35:42 <INF> [MAIN] Reset reason: Via esp_restart
[       5] 2022-12-12T02:35:43 <INF> [OTA] Update during boot triggered - Update File: /sdcard/firmware/update__continue-boot-on-bad-psram_(3387315).zip
[       5] 2022-12-12T02:35:43 <INF> [OTA] File: /sdcard/firmware/update__continue-boot-on-bad-psram_(3387315).zip Filetype: ZIP
[      59] 2022-12-12T02:36:37 <INF> [OTA] Files unzipped.
[      59] 2022-12-12T02:36:37 <INF> [OTA] Found firmware.bin
[      77] 2022-12-12T02:36:56 <INF> [OTA] Trigger reboot due to firmware update.
[      78] 2022-12-12T02:36:56 <INF> [OTA] Reboot triggered by Software (5s).
[      78] 2022-12-12T02:36:56 <WRN> [OTA] Reboot in 5sec
[      21] 2022-12-12T00:45:07 <INF> [SNTP] Time zone set to CET-1CEST,M3.5.0,M10.5.0/3
[      21] 2022-12-12T00:45:07 <INF> [MQTT] Digitizer interval is 4.8 minutes => setting MQTT LWT timeout to 12.1 minutes.
[      21] 2022-12-12T00:45:07 <INF> [MQTT IF] URI: mqtt://192.168.178.22:1883, clientname: heatingmeter, user: user, password: XXXXXXXX, maintopic: heatingmeter, last-will-topic: heatingmeter/connection, keepAlive: 727
[      21] 2022-12-12T00:45:07 <INF> [MQTT IF] Init
[      21] 2022-12-12T00:45:07 <INF> [MQTT IF] Client started, waiting for established connection...
[      21] 2022-12-12T00:45:07 <DBG> [TFLITE SERVER] ----------------------------------------------------------------
[      21] 2022-12-12T00:45:07 <INF> [TFLITE SERVER] Round #1 started
[      21] 2022-12-12T00:45:07 <DBG> [CTRL] Take Image
[      21] 2022-12-12T00:45:07 <INF> [MQTT IF] Connected to broker
[      21] 2022-12-12T00:45:07 <DBG> [MQTT IF] Published topic: heatingmeter/connection, content: connected (msg_id=7906)
[      21] 2022-12-12T00:45:07 <DBG> [MQTT SERVER] Publishing static MQTT topics...
[      21] 2022-12-12T00:45:07 <DBG> [MQTT IF] Published topic: heatingmeter/MAC, content: C8:F0:9E:A6:41:A4 (msg_id=62537)
[      21] 2022-12-12T00:45:07 <DBG> [MQTT IF] Published topic: heatingmeter/IP, content: 192.168.178.51 (msg_id=54141)
[      21] 2022-12-12T00:45:07 <DBG> [MQTT IF] Published topic: heatingmeter/hostname, content: heatingmeter (msg_id=26203)
[      22] 2022-12-12T00:45:07 <DBG> [MQTT IF] Published topic: heatingmeter/interval, content: 4.8 (msg_id=29595)
[      22] 2022-12-12T00:45:07 <DBG> [MQTT SERVER] Publishing system MQTT topics...
[      22] 2022-12-12T00:45:07 <DBG> [MQTT IF] Published topic: heatingmeter/uptime, content: 10 (msg_id=43234)
[      22] 2022-12-12T00:45:08 <DBG> [MQTT IF] Published topic: heatingmeter/freeMem, content: 3112111 (msg_id=43023)
[      22] 2022-12-12T00:45:08 <DBG> [MQTT IF] Published topic: heatingmeter/wifiRSSI, content: -54 (msg_id=16215)
[      22] 2022-12-12T00:45:08 <DBG> [MQTT IF] Published topic: heatingmeter/CPUtemp, content: 28 (msg_id=28865)
[      22] 2022-12-12T00:45:08 <DBG> [MQTT IF] Published topic: heatingmeter/status, content: Take Image (msg_id=10679)
[      35] 2022-12-12T00:45:21 <ERR> [CAM] is not working anymore (CCamera::CaptureToBasisImage) - most probably caused by a hardware problem (instablility, ...). System will reboot.
[      35] 2022-12-12T00:45:21 <INF> [OTA] Reboot triggered by Software (5s).
[      35] 2022-12-12T00:45:21 <WRN> [OTA] Reboot in 5sec
[       4] 2022-12-12T02:04:54 <INF> [MAIN] =================================================
[       4] 2022-12-12T02:04:54 <INF> [MAIN] ==================== Startup ====================
[       4] 2022-12-12T02:04:54 <INF> [MAIN] =================================================
[       4] 2022-12-12T02:04:54 <INF> [MAIN] Branch: 'continue-boot-on-bad-psram', Revision: 3387315, Date/Time: 2022-12-11 16:59, Web UI: Development-Branch: continue-boot-on-bad-psram (Commit: 3387315)
[       4] 2022-12-12T02:04:54 <INF> [MAIN] Reset reason: Via esp_restart
[      11] 2022-12-12T02:05:01 <INF> [MAIN] =================================================
[      11] 2022-12-12T02:05:01 <INF> [MAIN] ================== Main Started =================
[      11] 2022-12-12T02:05:01 <INF> [MAIN] =================================================
[      11] 2022-12-12T02:05:01 <INF> [MAIN] The device has %0.1f MBytes in the PSRAM
[      15] 2022-12-12T02:05:05 <ERR> [MAIN] Camera Framebuffer cannot be initialized!
[      15] 2022-12-12T02:05:05 <ERR> [HELPER] New System Status: 0x00000004
[      17] 2022-12-12T02:05:07 <WRN> [MAIN] Initialization completed with errors, but trying to continue...
[      20] 2022-12-12T03:05:10 <INF> [SNTP] Time zone set to CET-1CEST,M3.5.0,M10.5.0/3
[      20] 2022-12-12T03:05:10 <INF> [MQTT] Digitizer interval is 4.8 minutes => setting MQTT LWT timeout to 12.1 minutes.
[      20] 2022-12-12T03:05:10 <INF> [MQTT IF] URI: mqtt://192.168.178.22:1883, clientname: heatingmeter, user: user, password: XXXXXXXX, maintopic: heatingmeter, last-will-topic: heatingmeter/connection, keepAlive: 727
[      20] 2022-12-12T03:05:10 <INF> [MQTT IF] Init
[      20] 2022-12-12T03:05:10 <INF> [MQTT IF] Client started, waiting for established connection...
[      20] 2022-12-12T03:05:10 <DBG> [TFLITE SERVER] ----------------------------------------------------------------
[      20] 2022-12-12T03:05:10 <INF> [TFLITE SERVER] Round #1 started
[      20] 2022-12-12T03:05:10 <DBG> [CTRL] Take Image
[      21] 2022-12-12T03:05:12 <INF> [MQTT IF] Connected to broker
[      21] 2022-12-12T03:05:12 <DBG> [MQTT IF] Published topic: heatingmeter/connection, content: connected (msg_id=47603)
[      21] 2022-12-12T03:05:12 <DBG> [MQTT SERVER] Publishing static MQTT topics...
[      21] 2022-12-12T03:05:12 <DBG> [MQTT IF] Published topic: heatingmeter/MAC, content: C8:F0:9E:A6:41:A4 (msg_id=19569)
[      21] 2022-12-12T03:05:12 <DBG> [MQTT IF] Published topic: heatingmeter/IP, content: 192.168.178.51 (msg_id=63810)
[      22] 2022-12-12T03:05:12 <DBG> [MQTT IF] Published topic: heatingmeter/hostname, content: heatingmeter (msg_id=65417)
[      22] 2022-12-12T03:05:12 <DBG> [MQTT IF] Published topic: heatingmeter/interval, content: 4.8 (msg_id=56696)
[      22] 2022-12-12T03:05:12 <DBG> [MQTT SERVER] Publishing system MQTT topics...
[      22] 2022-12-12T03:05:12 <DBG> [MQTT IF] Published topic: heatingmeter/uptime, content: 11 (msg_id=31148)
[      22] 2022-12-12T03:05:12 <DBG> [MQTT IF] Published topic: heatingmeter/freeMem, content: 3113475 (msg_id=59126)
[      22] 2022-12-12T03:05:12 <DBG> [MQTT IF] Published topic: heatingmeter/wifiRSSI, content: -57 (msg_id=41724)
[      22] 2022-12-12T03:05:12 <DBG> [MQTT IF] Published topic: heatingmeter/CPUtemp, content: 31 (msg_id=55247)
[      22] 2022-12-12T03:05:12 <WRN> [MQTT IF] Failed to publish topic '', re-trying...
[      22] 2022-12-12T03:05:12 <ERR> [MQTT IF] Failed to publish topic '', skipping all MQTT publishings in this round!
[      22] 2022-12-12T03:05:12 <DBG> [MQTT IF] Published topic: heatingmeter/status, content: Take Image (msg_id=30497)
[      35] 2022-12-12T03:05:25 <ERR> [CAM] is not working anymore (CCamera::CaptureToBasisImage) - most probably caused by a hardware problem (instablility, ...). System will reboot.
[      35] 2022-12-12T03:05:25 <INF> [OTA] Reboot triggered by Software (5s).
[      35] 2022-12-12T03:05:25 <WRN> [OTA] Reboot in 5sec
[       4] 2022-12-12T02:57:15 <INF> [MAIN] =================================================
[       4] 2022-12-12T02:57:15 <INF> [MAIN] ==================== Startup ====================
[       4] 2022-12-12T02:57:15 <INF> [MAIN] =================================================
[       4] 2022-12-12T02:57:15 <INF> [MAIN] Branch: 'continue-boot-on-bad-psram', Revision: 3387315, Date/Time: 2022-12-11 16:59, Web UI: Development-Branch: continue-boot-on-bad-psram (Commit: 3387315)
[       4] 2022-12-12T02:57:15 <INF> [MAIN] Reset reason: Exception/panic
[      12] 2022-12-12T02:57:23 <INF> [MAIN] =================================================
[      12] 2022-12-12T02:57:23 <INF> [MAIN] ================== Main Started =================
[      12] 2022-12-12T02:57:23 <INF> [MAIN] =================================================
[      12] 2022-12-12T02:57:23 <INF> [MAIN] The device has %0.1f MBytes in the PSRAM
[      16] 2022-12-12T02:57:27 <ERR> [MAIN] Camera Framebuffer cannot be initialized!
[      16] 2022-12-12T02:57:27 <ERR> [HELPER] New System Status: 0x00000004
[      18] 2022-12-12T02:57:29 <WRN> [MAIN] Initialization completed with errors, but trying to continue...
[      18] 2022-12-12T02:57:29 <WRN> [TFLITE SERVER] Restarted due to an Exception/panic! Postponing first round start by 5 minutes to allow for an OTA or to fetch the log!
[      18] 2022-12-12T02:57:29 <WRN> [TFLITE SERVER] Setting logfile level to DEBUG until the next reboot!
[      19] 2022-12-12T02:57:30 <DBG> [MAIN SERVER] info_get_handler
[      20] 2022-12-12T02:57:31 <DBG> [MAIN SERVER] info_get_handler
[      20] 2022-12-12T02:57:31 <DBG> [MAIN SERVER] info_get_handler
[      25] 2022-12-12T02:57:36 <DBG> [OTA FILE] log_get_last_part_handler

Expected Behavior

No response

Screenshots

No response

Additional Context

No response

caco3 commented 2 years ago

Yes, it seems the framebuffer error should not be treated as critical. In the newest release (13.0.7) we continue the operation in such case.

ednax commented 1 year ago

after reverting from 13.0.5 to 12.0.1, I just tried 13.0.7. Instead of rebooting every 5 minutes, it reboots every 5 seconds:

[       8] 2022-12-12T13:12:37 <INF> [MAIN] =================================================
[       8] 2022-12-12T13:12:37 <INF> [MAIN] ================== Main Started =================
[       8] 2022-12-12T13:12:37 <INF> [MAIN] =================================================
[      12] 2022-12-12T13:12:41 <ERR> [MAIN] Camera Framebuffer cannot be initialized!
[      14] 2022-12-12T13:12:43 <INF> [MAIN] Initialization completed successfully!
[      27] 2022-12-12T13:12:56 <ERR> [CAM] Capture Failed (Procedure 'EnableAutoExposure') --> Reboot! Check that your camera module is working and connected properly.
[      38] 2022-12-12T14:13:07 <ERR> [CAM] is not working anymore (CCamera::CaptureToBasisImage) - most probably caused by a hardware problem (instablility, ...). System will reboot.
[      38] 2022-12-12T14:13:07 <WRN> [OTA] Reboot in 5sec
[       4] 2022-12-12T13:24:03 <INF> [MAIN] =================================================
[       4] 2022-12-12T13:24:04 <INF> [MAIN] ==================== Startup ====================
[       4] 2022-12-12T13:24:04 <INF> [MAIN] =================================================
[       4] 2022-12-12T13:24:04 <INF> [MAIN] Branch: 'HEAD', Revision: 304b9e0, Date/Time: 2022-12-11 22:36, Web UI: Release: v13.0.7 (Commit: 304b9e0)
[       4] 2022-12-12T13:24:04 <INF> [MAIN] Reset reason: Via esp_restart
[       9] 2022-12-12T13:24:09 <INF> [MAIN] =================================================
[       9] 2022-12-12T13:24:09 <INF> [MAIN] ================== Main Started =================
[       9] 2022-12-12T13:24:09 <INF> [MAIN] =================================================
[      13] 2022-12-12T13:24:13 <ERR> [MAIN] Camera Framebuffer cannot be initialized!
[      15] 2022-12-12T13:24:15 <INF> [MAIN] Initialization completed successfully!
[      28] 2022-12-12T13:24:27 <ERR> [CAM] Capture Failed (Procedure 'EnableAutoExposure') --> Reboot! Check that your camera module is working and connected properly.
[      39] 2022-12-12T14:24:39 <ERR> [CAM] is not working anymore (CCamera::CaptureToBasisImage) - most probably caused by a hardware problem (instablility, ...). System will reboot.
[      39] 2022-12-12T14:24:39 <WRN> [OTA] Reboot in 5sec
[       4] 2022-12-12T13:20:45 <INF> [MAIN] =================================================
[       4] 2022-12-12T13:20:45 <INF> [MAIN] ==================== Startup ====================
[       4] 2022-12-12T13:20:46 <INF> [MAIN] =================================================
[       4] 2022-12-12T13:20:46 <INF> [MAIN] Branch: 'HEAD', Revision: 304b9e0, Date/Time: 2022-12-11 22:36, Web UI: Release: v13.0.7 (Commit: 304b9e0)
[       4] 2022-12-12T13:20:46 <INF> [MAIN] Reset reason: Via esp_restart
[       8] 2022-12-12T13:20:50 <INF> [MAIN] =================================================
[       8] 2022-12-12T13:20:50 <INF> [MAIN] ================== Main Started =================
[       8] 2022-12-12T13:20:50 <INF> [MAIN] =================================================
[      12] 2022-12-12T13:20:54 <ERR> [MAIN] Camera Framebuffer cannot be initialized!
[      14] 2022-12-12T13:20:56 <INF> [MAIN] Initialization completed successfully!
[      27] 2022-12-12T13:21:08 <ERR> [CAM] Capture Failed (Procedure 'EnableAutoExposure') --> Reboot! Check that your camera module is working and connected properly.
[      38] 2022-12-12T14:21:19 <ERR> [CAM] is not working anymore (CCamera::CaptureToBasisImage) - most probably caused by a hardware problem (instablility, ...). System will reboot.
[      38] 2022-12-12T14:21:19 <WRN> [OTA] Reboot in 5sec
[       4] 2022-12-12T13:24:15 <INF> [MAIN] =================================================
[       4] 2022-12-12T13:24:15 <INF> [MAIN] ==================== Startup ====================
[       4] 2022-12-12T13:24:15 <INF> [MAIN] =================================================
[       4] 2022-12-12T13:24:15 <INF> [MAIN] Branch: 'HEAD', Revision: 304b9e0, Date/Time: 2022-12-11 22:36, Web UI: Release: v13.0.7 (Commit: 304b9e0)
[       4] 2022-12-12T13:24:15 <INF> [MAIN] Reset reason: Via esp_restart
[       8] 2022-12-12T13:24:19 <INF> [MAIN] =================================================
[       8] 2022-12-12T13:24:19 <INF> [MAIN] ================== Main Started =================
[       8] 2022-12-12T13:24:19 <INF> [MAIN] =================================================
[      12] 2022-12-12T13:24:23 <ERR> [MAIN] Camera Framebuffer cannot be initialized!

with 12.0.1 the esp32 has no problem to capture the image. But 13.0.5 and 13.0.7 can't initialize the framebuffer.

marc4s commented 1 year ago

[      10] 2022-12-12T13:10:42 <INF> [MAIN] ================== Main Started =================
[      10] 2022-12-12T13:10:42 <INF> [MAIN] =================================================
[      10] 2022-12-12T13:10:42 <WRN> [MAIN] Web UI version (e23c89a) does not match firmware version (304b9e0) !
[      12] 2022-12-12T13:10:44 <INF> [MAIN] Initialization completed successfully!
[      12] 2022-12-12T12:33:08 <INF> [SNTP] Status OK
[      12] 2022-12-12T12:33:08 <INF> [SNTP] Time is synced with NTP Server pool.ntp.org
[      12] 2022-12-12T13:33:08 <INF> [SNTP] Time zone set to CET-1CEST,M3.5.0,M10.5.0/3
[      12] 2022-12-12T13:33:08 <INF> [MAIN] =================================================
[      12] 2022-12-12T13:33:08 <INF> [MAIN] ================== Main Started =================
[      12] 2022-12-12T13:33:08 <INF> [MAIN] =================================================
[      14] 2022-12-12T13:33:10 <INF> [MAIN] Initialization completed successfully!
[       4] 2022-12-12T12:51:02 <INF> [MAIN] =================================================
[       4] 2022-12-12T12:51:02 <INF> [MAIN] ==================== Startup ====================
[       4] 2022-12-12T12:51:02 <INF> [MAIN] =================================================
[       4] 2022-12-12T12:51:02 <INF> [MAIN] Branch: 'HEAD', Revision: 304b9e0, Date/Time: 2022-12-11 22:36, Web UI: Release: v13.0.7 (Commit: 304b9e0)
[       4] 2022-12-12T12:51:02 <INF> [MAIN] Reset reason: Via esp_restart
[       8] 2022-12-12T12:51:05 <INF> [MAIN] =================================================
[       8] 2022-12-12T12:51:05 <INF> [MAIN] ================== Main Started =================
[       8] 2022-12-12T12:51:05 <INF> [MAIN] =================================================
[      12] 2022-12-12T12:51:09 <ERR> [MAIN] Camera Framebuffer cannot be initialized!
[      14] 2022-12-12T12:51:11 <INF> [MAIN] Initialization completed successfully!
[      33] 2022-12-12T13:51:31 <ERR> [CAM] is not working anymore (CCamera::CaptureToBasisImage) - most probably caused by a hardware problem (instablility, ...). System will reboot.
[       4] 2022-12-12T12:52:01 <INF> [MAIN] =================================================
[       4] 2022-12-12T12:52:01 <INF> [MAIN] ==================== Startup ====================
[       4] 2022-12-12T12:52:01 <INF> [MAIN] =================================================
[       4] 2022-12-12T12:52:01 <INF> [MAIN] Branch: 'HEAD', Revision: 304b9e0, Date/Time: 2022-12-11 22:36, Web UI: Release: v13.0.7 (Commit: 304b9e0)
[       4] 2022-12-12T12:52:01 <INF> [MAIN] Reset reason: Via esp_restart
[       8] 2022-12-12T12:52:05 <INF> [MAIN] =================================================
[       8] 2022-12-12T12:52:05 <INF> [MAIN] ================== Main Started =================
[       8] 2022-12-12T12:52:05 <INF> [MAIN] =================================================
[      12] 2022-12-12T12:52:09 <ERR> [MAIN] Camera Framebuffer cannot be initialized!
[      14] 2022-12-12T12:52:11 <INF> [MAIN] Initialization completed successfully!
[      31] 2022-12-12T13:52:29 <ERR> [CAM] is not working anymore (CCamera::CaptureToBasisImage) - most probably caused by a hardware problem (instablility, ...). System will reboot.
[       4] 2022-12-12T12:52:54 <INF> [MAIN] =================================================
[       4] 2022-12-12T12:52:54 <INF> [MAIN] ==================== Startup ====================
[       4] 2022-12-12T12:52:54 <INF> [MAIN] =================================================
[       4] 2022-12-12T12:52:54 <INF> [MAIN] Branch: 'HEAD', Revision: 304b9e0, Date/Time: 2022-12-11 22:36, Web UI: Release: v13.0.7 (Commit: 304b9e0)
[       4] 2022-12-12T12:52:54 <INF> [MAIN] Reset reason: Via esp_restart
[       8] 2022-12-12T12:52:57 <INF> [MAIN] =================================================
[       8] 2022-12-12T12:52:57 <INF> [MAIN] ================== Main Started =================
[       8] 2022-12-12T12:52:57 <INF> [MAIN] =================================================
[      12] 2022-12-12T12:53:01 <ERR> [MAIN] Camera Framebuffer cannot be initialized!
[      14] 2022-12-12T12:53:03 <INF> [MAIN] Initialization completed successfully!
[      31] 2022-12-12T13:53:21 <ERR> [CAM] is not working anymore (CCamera::CaptureToBasisImage) - most probably caused by a hardware problem (instablility, ...). System will reboot.
[       4] 2022-12-12T12:53:42 <INF> [MAIN] =================================================
[       4] 2022-12-12T12:53:42 <INF> [MAIN] ==================== Startup ====================
[       4] 2022-12-12T12:53:42 <INF> [MAIN] =================================================
[       4] 2022-12-12T12:53:42 <INF> [MAIN] Branch: 'HEAD', Revision: 304b9e0, Date/Time: 2022-12-11 22:36, Web UI: Release: v13.0.7 (Commit: 304b9e0)
[       4] 2022-12-12T12:53:42 <INF> [MAIN] Reset reason: Via esp_restart
[       8] 2022-12-12T12:53:45 <INF> [MAIN] =================================================
[       8] 2022-12-12T12:53:45 <INF> [MAIN] ================== Main Started =================
[       8] 2022-12-12T12:53:45 <INF> [MAIN] =================================================
[      12] 2022-12-12T12:53:49 <ERR> [MAIN] Camera Framebuffer cannot be initialized!
[      14] 2022-12-12T12:53:51 <INF> [MAIN] Initialization completed successfully!
[      32] 2022-12-12T13:54:09 <ERR> [CAM] is not working anymore (CCamera::CaptureToBasisImage) - most probably caused by a hardware problem (instablility, ...). System will reboot.
[       4] 2022-12-12T12:54:28 <INF> [MAIN] =================================================
[       4] 2022-12-12T12:54:28 <INF> [MAIN] ==================== Startup ====================
[       4] 2022-12-12T12:54:28 <INF> [MAIN] =================================================
[       4] 2022-12-12T12:54:28 <INF> [MAIN] Branch: 'HEAD', Revision: 304b9e0, Date/Time: 2022-12-11 22:36, Web UI: Release: v13.0.7 (Commit: 304b9e0)
[       4] 2022-12-12T12:54:28 <INF> [MAIN] Reset reason: Via esp_restart
[       8] 2022-12-12T12:54:31 <INF> [MAIN] =================================================
[       8] 2022-12-12T12:54:31 <INF> [MAIN] ================== Main Started =================
[       8] 2022-12-12T12:54:31 <INF> [MAIN] =================================================
[      12] 2022-12-12T12:54:35 <ERR> [MAIN] Camera Framebuffer cannot be initialized!
[      14] 2022-12-12T12:54:37 <INF> [MAIN] Initialization completed successfully!
[      31] 2022-12-12T13:54:55 <ERR> [CAM] is not working anymore (CCamera::CaptureToBasisImage) - most probably caused by a hardware problem (instablility, ...). System will reboot.
[       4] 2022-12-12T12:55:12 <INF> [MAIN] =================================================
[       4] 2022-12-12T12:55:12 <INF> [MAIN] ==================== Startup ====================
[       4] 2022-12-12T12:55:12 <INF> [MAIN] =================================================
[       4] 2022-12-12T12:55:12 <INF> [MAIN] Branch: 'HEAD', Revision: 304b9e0, Date/Time: 2022-12-11 22:36, Web UI: Release: v13.0.7 (Commit: 304b9e0)
[       4] 2022-12-12T12:55:12 <INF> [MAIN] Reset reason: Via esp_restart
[       8] 2022-12-12T12:55:16 <INF> [MAIN] =================================================
[       8] 2022-12-12T12:55:16 <INF> [MAIN] ================== Main Started =================
[       8] 2022-12-12T12:55:16 <INF> [MAIN] =================================================
[      12] 2022-12-12T12:55:20 <ERR> [MAIN] Camera Framebuffer cannot be initialized!
[      14] 2022-12-12T12:55:22 <INF> [MAIN] Initialization completed successfully!
[      33] 2022-12-12T13:55:41 <ERR> [CAM] is not working anymore (CCamera::CaptureToBasisImage) - most probably caused by a hardware problem (instablility, ...). System will reboot.
[       4] 2022-12-12T12:55:58 <INF> [MAIN] =================================================
[       4] 2022-12-12T12:55:58 <INF> [MAIN] ==================== Startup ====================
[       4] 2022-12-12T12:55:58 <INF> [MAIN] =================================================
[       4] 2022-12-12T12:55:58 <INF> [MAIN] Branch: 'HEAD', Revision: 304b9e0, Date/Time: 2022-12-11 22:36, Web UI: Release: v13.0.7 (Commit: 304b9e0)
[       4] 2022-12-12T12:55:58 <INF> [MAIN] Reset reason: Via esp_restart
[       8] 2022-12-12T12:56:01 <INF> [MAIN] =================================================
[       8] 2022-12-12T12:56:01 <INF> [MAIN] ================== Main Started =================
[       8] 2022-12-12T12:56:01 <INF> [MAIN] =================================================
[      12] 2022-12-12T12:56:05 <ERR> [MAIN] Camera Framebuffer cannot be initialized!
[      14] 2022-12-12T12:56:07 <INF> [MAIN] Initialization completed successfully!
[      32] 2022-12-12T13:56:25 <ERR> [CAM] is not working anymore (CCamera::CaptureToBasisImage) - most probably caused by a hardware problem (instablility, ...). System will reboot.
[       4] 2022-12-12T12:56:42 <INF> [MAIN] =================================================
[       4] 2022-12-12T12:56:42 <INF> [MAIN] ==================== Startup ====================
[       4] 2022-12-12T12:56:42 <INF> [MAIN] =================================================
[       4] 2022-12-12T12:56:42 <INF> [MAIN] Branch: 'HEAD', Revision: 304b9e0, Date/Time: 2022-12-11 22:36, Web UI: Release: v13.0.7 (Commit: 304b9e0)
[       4] 2022-12-12T12:56:42 <INF> [MAIN] Reset reason: Via esp_restart
[       8] 2022-12-12T12:56:46 <INF> [MAIN] =================================================
[       8] 2022-12-12T12:56:46 <INF> [MAIN] ================== Main Started =================
[       8] 2022-12-12T12:56:46 <INF> [MAIN] =================================================
[      12] 2022-12-12T12:56:50 <ERR> [MAIN] Camera Framebuffer cannot be initialized!
[      14] 2022-12-12T12:56:52 <INF> [MAIN] Initialization completed successfully!
[      31] 2022-12-12T13:57:10 <ERR> [CAM] is not working anymore (CCamera::CaptureToBasisImage) - most probably caused by a hardware problem (instablility, ...). System will reboot.
[       4] 2022-12-12T12:57:25 <INF> [MAIN] =================================================
[       4] 2022-12-12T12:57:25 <INF> [MAIN] ==================== Startup ====================
[       4] 2022-12-12T12:57:25 <INF> [MAIN] =================================================
[       4] 2022-12-12T12:57:25 <INF> [MAIN] Branch: 'HEAD', Revision: 304b9e0, Date/Time: 2022-12-11 22:36, Web UI: Release: v13.0.7 (Commit: 304b9e0)
[       4] 2022-12-12T12:57:25 <INF> [MAIN] Reset reason: Via esp_restart
[       8] 2022-12-12T12:57:29 <INF> [MAIN] =================================================
[       8] 2022-12-12T12:57:29 <INF> [MAIN] ================== Main Started =================
[       8] 2022-12-12T12:57:29 <INF> [MAIN] =================================================
[      12] 2022-12-12T12:57:33 <ERR> [MAIN] Camera Framebuffer cannot be initialized!
[      14] 2022-12-12T12:57:35 <INF> [MAIN] Initialization completed successfully!
[      32] 2022-12-12T13:57:53 <ERR> [CAM] is not working anymore (CCamera::CaptureToBasisImage) - most probably caused by a hardware problem (instablility, ...). System will reboot.
[       4] 2022-12-12T12:58:07 <INF> [MAIN] =================================================
[       4] 2022-12-12T12:58:07 <INF> [MAIN] ==================== Startup ====================
[       4] 2022-12-12T12:58:07 <INF> [MAIN] =================================================
[       4] 2022-12-12T12:58:07 <INF> [MAIN] Branch: 'HEAD', Revision: 304b9e0, Date/Time: 2022-12-11 22:36, Web UI: Release: v13.0.7 (Commit: 304b9e0)
[       4] 2022-12-12T12:58:07 <INF> [MAIN] Reset reason: Via esp_restart
[       8] 2022-12-12T12:58:10 <INF> [MAIN] =================================================
[       8] 2022-12-12T12:58:10 <INF> [MAIN] ================== Main Started =================
[       8] 2022-12-12T12:58:10 <INF> [MAIN] =================================================
[      12] 2022-12-12T12:58:15 <ERR> [MAIN] Camera Framebuffer cannot be initialized!
[      14] 2022-12-12T12:58:17 <INF> [MAIN] Initialization completed successfully!
[      31] 2022-12-12T13:58:34 <ERR> [CAM] is not working anymore (CCamera::CaptureToBasisImage) - most probably caused by a hardware problem (instablility, ...). System will reboot.
[       4] 2022-12-12T12:58:48 <INF> [MAIN] =================================================
[       4] 2022-12-12T12:58:48 <INF> [MAIN] ==================== Startup ====================
[       4] 2022-12-12T12:58:48 <INF> [MAIN] =================================================
[       4] 2022-12-12T12:58:48 <INF> [MAIN] Branch: 'HEAD', Revision: 304b9e0, Date/Time: 2022-12-11 22:36, Web UI: Release: v13.0.7 (Commit: 304b9e0)
[       4] 2022-12-12T12:58:48 <INF> [MAIN] Reset reason: Via esp_restart
[       8] 2022-12-12T12:58:52 <INF> [MAIN] =================================================
[       8] 2022-12-12T12:58:52 <INF> [MAIN] ================== Main Started =================
[       8] 2022-12-12T12:58:52 <INF> [MAIN] =================================================
[      12] 2022-12-12T12:58:56 <ERR> [MAIN] Camera Framebuffer cannot be initialized!
[      14] 2022-12-12T12:58:58 <INF> [MAIN] Initialization completed successfully!

13.0.7 still affacted. setup process is FINE!! - camera is working. alignement done. ref image. etc. then reboot -> bootloop

marc4s commented 1 year ago

should #1513 be reopened?

caco3 commented 1 year ago

I am sorry to hear that is does not work anymore. We are trying hard to understand the issue and to solve it.

Could you try to run the following firmware? https://github.com/jomjol/AI-on-the-edge-device/actions/workflows/build.yaml?query=branch%3Arolling It adds more error logging.

And if it fails again, can you please provide the USB log? You can use the Web Console for this: https://jomjol.github.io/AI-on-the-edge-device/index.html

ednax commented 1 year ago

I installed 434e268, the Errorlog is configured as "DEBUG" and shows the following:

[0d00h00m08s] 2022-12-12T17:59:53 <INF> [MAIN] =================================================
[0d00h00m08s] 2022-12-12T17:59:53 <INF> [MAIN] ================== Main Started =================
[0d00h00m08s] 2022-12-12T17:59:53 <INF> [MAIN] =================================================
[0d00h00m08s] 2022-12-12T17:59:53 <INF> [MAIN] The device has 8 MBytes of PSRAM
[0d00h00m12s] 2022-12-12T17:59:57 <ERR> [MAIN] Camera Framebuffer cannot be initialized!
[0d00h00m12s] 2022-12-12T17:59:57 <ERR> [HELPER] New System Status: 0x00000100
[0d00h00m14s] 2022-12-12T17:59:59 <WRN> [MAIN] Initialization completed with errors, but trying to continue...
[0d00h00m27s] 2022-12-12T18:00:12 <ERR> [CAM] Capture Failed (Procedure 'EnableAutoExposure') --> Reboot! Check that your camera module is working and connected properly.
[0d00h00m38s] 2022-12-12T19:00:22 <ERR> [CAM] is not working anymore (CCamera::CaptureToBasisImage) - most probably caused by a hardware problem (instablility, ...). System will reboot.
[0d00h00m38s] 2022-12-12T19:00:22 <WRN> [OTA] Reboot in 5sec
[0d00h00m04s] 2022-12-12T17:56:40 <INF> [MAIN] =================================================
[0d00h00m04s] 2022-12-12T17:56:40 <INF> [MAIN] ==================== Startup ====================
[0d00h00m04s] 2022-12-12T17:56:40 <INF> [MAIN] =================================================
[0d00h00m04s] 2022-12-12T17:56:40 <INF> [MAIN] Branch: 'rolling', Revision: 434e268, Date/Time: 2022-12-12 16:16, Web UI: Development-Branch: rolling (Commit: 434e268)
[0d00h00m04s] 2022-12-12T17:56:40 <INF> [MAIN] Reset reason: Via esp_restart
[0d00h00m09s] 2022-12-12T17:56:45 <INF> [MAIN] =================================================
[0d00h00m09s] 2022-12-12T17:56:45 <INF> [MAIN] ================== Main Started =================
[0d00h00m09s] 2022-12-12T17:56:45 <INF> [MAIN] =================================================
[0d00h00m09s] 2022-12-12T17:56:45 <INF> [MAIN] The device has 8 MBytes of PSRAM
[0d00h00m13s] 2022-12-12T17:56:49 <ERR> [MAIN] Camera Framebuffer cannot be initialized!
[0d00h00m13s] 2022-12-12T17:56:49 <ERR> [HELPER] New System Status: 0x00000100
[0d00h00m15s] 2022-12-12T17:56:51 <WRN> [MAIN] Initialization completed with errors, but trying to continue...
[0d00h00m27s] 2022-12-12T17:57:04 <ERR> [CAM] Capture Failed (Procedure 'EnableAutoExposure') --> Reboot! Check that your camera module is working and connected properly.
[0d00h00m38s] 2022-12-12T18:57:14 <ERR> [CAM] is not working anymore (CCamera::CaptureToBasisImage) - most probably caused by a hardware problem (instablility, ...). System will reboot.
[0d00h00m38s] 2022-12-12T18:57:14 <WRN> [OTA] Reboot in 5sec

I try to get the usb-log as soon as possible, but I have no equipment in the basement where the camera is mounted. So it may take a day or two...

caco3 commented 1 year ago

I think its ok for now without the USB log! With the new rolling we write more data of the startup into the log file.

ednax commented 1 year ago

Ok, but a fortunate circumstance let me grab the device and I was surprised, that it is very easy to get the logs: esp-web-tools-logs.txt

caco3 commented 1 year ago

hmm, but what I do not understand: On your USB log, I see 2 startups, and both where successful: Initialization completed successfully! (It then later crashes, but that's another topic). But on your log file logs, we see Initialization completed with errors.

Do you know why it suddenly succeeded to startup? One explanation could be that you did a power cycle. I also have seen on one of my devices that the frame buffer init successes once after a power cycle (but after a the next reset it failed again).

I made a minor improvement in https://github.com/jomjol/AI-on-the-edge-device/actions/workflows/build.yaml?query=branch%3Afix-fb-init-errors could you try that one (pick the most top one)?

marc4s commented 1 year ago

the reason for the successful starts from my point of view: flashed the release on the ESP32 then the script made the hardreset and the device rebooted. (I flashed the "initial_...zip" and did not expected html changes). thats why i took the device and installed it back on the watermeter. then realized that something is wrong and got back to pc. found the log -> refreshed html refreshed the referenceimage,alignement etc. and then it was back in the bootloop. something like that...

I will try your improvement soon today

jomjol commented 1 year ago

Some background experience: 1) It can happen, that the initialization works, but taking the first image is failing --> reboot 2) This also depends on the explicit esp-idf version and/or the esp32cam version. This could be a topic here (esp-idf v4.4 changed to v5.2). I could not find a rule for this.

marc4s commented 1 year ago

I see all your efforts and appreciate that very much worth. I would like to help more with more debugging infos for example... somehow the bootloop stopped at 14:01 with successful. but images was not taken and no information sent via mqtt

`[ 4] 2022-12-12T14:00:24 [MAIN] ================================================= [ 4] 2022-12-12T14:00:24 [MAIN] ==================== Startup ==================== [ 4] 2022-12-12T14:00:24 [MAIN] ================================================= [ 4] 2022-12-12T14:00:24 [MAIN] Branch: 'HEAD', Revision: 304b9e0, Date/Time: 2022-12-11 22:36, Web UI: Release: v13.0.7 (Commit: 304b9e0) [ 4] 2022-12-12T14:00:24 [MAIN] Reset reason: Via esp_restart [ 8] 2022-12-12T14:00:27 [MAIN] ================================================= [ 8] 2022-12-12T14:00:27 [MAIN] ================== Main Started ================= [ 8] 2022-12-12T14:00:27 [MAIN] ================================================= [ 12] 2022-12-12T14:00:31 [MAIN] Camera Framebuffer cannot be initialized! [ 14] 2022-12-12T14:00:33 [MAIN] Initialization completed successfully! [ 32] 2022-12-12T15:00:51 [CAM] is not working anymore (CCamera::CaptureToBasisImage) - most probably caused by a hardware problem (instablility, ...). System will reboot. [ 4] 2022-12-12T14:01:06 [MAIN] ================================================= [ 4] 2022-12-12T14:01:06 [MAIN] ==================== Startup ==================== [ 4] 2022-12-12T14:01:06 [MAIN] ================================================= [ 4] 2022-12-12T14:01:06 [MAIN] Branch: 'HEAD', Revision: 304b9e0, Date/Time: 2022-12-11 22:36, Web UI: Release: v13.0.7 (Commit: 304b9e0) [ 4] 2022-12-12T14:01:06 [MAIN] Reset reason: Via esp_restart [ 8] 2022-12-12T14:01:09 [MAIN] ================================================= [ 8] 2022-12-12T14:01:09 [MAIN] ================== Main Started ================= [ 8] 2022-12-12T14:01:09 [MAIN] ================================================= [ 12] 2022-12-12T14:01:13 [MAIN] Camera Framebuffer cannot be initialized! [ 14] 2022-12-12T14:01:15 [MAIN] Initialization completed successfully! [ 4] 2022-12-12T17:58:19 [MAIN] ================================================= [ 4] 2022-12-12T17:58:19 [MAIN] ==================== Startup ==================== [ 4] 2022-12-12T17:58:20 [MAIN] ================================================= [ 4] 2022-12-12T17:58:20 [MAIN] Branch: 'HEAD', Revision: 304b9e0, Date/Time: 2022-12-11 22:36, Web UI: Release: v13.0.7 (Commit: 304b9e0) [ 4] 2022-12-12T17:58:20 [MAIN] Reset reason: Via esp_restart [ 5] 2022-12-12T17:58:20 [OTA] Update during boot triggered - Update File: /sdcard/firmware/update_fix-fb-init-errors(b59e16a).zip [ 5] 2022-12-12T17:58:20 [OTA] File: /sdcard/firmware/update_fix-fb-init-errors(b59e16a).zip Filetype: ZIP [ 76] 2022-12-12T17:59:31 [OTA] Files unzipped. [ 76] 2022-12-12T17:59:31 [OTA] Found firmware.bin [ 91] 2022-12-12T17:59:47 [OTA] Trigger reboot due to firmware update. [ 91] 2022-12-12T17:59:47 [OTA] Reboot triggered by Software (5s). [ 91] 2022-12-12T17:59:47 [OTA] Reboot in 5sec [0d00h00m04s] 2022-12-12T17:48:15 [MAIN] ================================================= [0d00h00m04s] 2022-12-12T17:48:15 [MAIN] ==================== Startup ==================== [0d00h00m04s] 2022-12-12T17:48:15 [MAIN] ================================================= [0d00h00m04s] 2022-12-12T17:48:15 [MAIN] Branch: 'fix-fb-init-errors', Revision: b59e16a+, Date/Time: 2022-12-12 17:26, Web UI: Development-Branch: fix-fb-init-errors (Commit: b59e16a+) [0d00h00m04s] 2022-12-12T17:48:15 [MAIN] Reset reason: Via esp_restart [0d00h00m08s] 2022-12-12T17:48:19 [MAIN] ================================================= [0d00h00m08s] 2022-12-12T17:48:19 [MAIN] ================== Main Started ================= [0d00h00m08s] 2022-12-12T17:48:19 [MAIN] ================================================= [0d00h00m08s] 2022-12-12T17:48:19 [MAIN] The device has 8 MBytes of PSRAM [0d00h00m12s] 2022-12-12T17:48:23 [MAIN] Camera Framebuffer cannot be initialized! [0d00h00m12s] 2022-12-12T17:48:23 [HELPER] New System Status: 0x00000100 [0d00h00m14s] 2022-12-12T17:48:25 [MAIN] Initialization completed with errors, but trying to continue... [0d00h00m43s] 2022-12-12T18:48:54 [CAM] is not working anymore (CCamera::CaptureToBasisImage) - most probably caused by a hardware problem (instablility, ...). System will reboot. [0d00h00m04s] 2022-12-12T17:48:02 [MAIN] ================================================= [0d00h00m04s] 2022-12-12T17:48:02 [MAIN] ==================== Startup ==================== [0d00h00m04s] 2022-12-12T17:48:02 [MAIN] ================================================= [0d00h00m04s] 2022-12-12T17:48:02 [MAIN] Branch: 'fix-fb-init-errors', Revision: b59e16a+, Date/Time: 2022-12-12 17:26, Web UI: Development-Branch: fix-fb-init-errors (Commit: b59e16a+) [0d00h00m04s] 2022-12-12T17:48:02 [MAIN] Reset reason: Via esp_restart [0d00h00m08s] 2022-12-12T17:48:06 [MAIN] ================================================= [0d00h00m08s] 2022-12-12T17:48:06 [MAIN] ================== Main Started ================= [0d00h00m08s] 2022-12-12T17:48:06 [MAIN] ================================================= [0d00h00m08s] 2022-12-12T17:48:06 [MAIN] The device has 8 MBytes of PSRAM [0d00h00m12s] 2022-12-12T17:48:10 [MAIN] Camera Framebuffer cannot be initialized! [0d00h00m12s] 2022-12-12T17:48:10 [HELPER] New System Status: 0x00000100 [0d00h00m14s] 2022-12-12T17:48:12 [MAIN] Initialization completed with errors, but trying to continue...

ednax commented 1 year ago

Do you know why it suddenly succeeded to startup?

hm, is there a difference between a reboot, and power off/on ?

ednax commented 1 year ago

I made a minor improvement in https://github.com/jomjol/AI-on-the-edge-device/actions/workflows/build.yaml?query=branch%3Afix-fb-init-errors could you try that one (pick the most top one)? here is the log directly after installing 8794b8a esp-web-tools-logs.txt

it seems to me, that after flashing and rebooting, it keeps in a bootloop

booting
Camera Framebuffer cannot be initialized!
Capture Failed
CAM: is not working anymore -> System will reboot

If I pull the plug and disconnect the device from the power supply, it makes at least two pictuires and then it reboots.

[0d00h00m18s] 2022-12-12T18:11:50 <ERR> [MAIN] Camera Framebuffer cannot be initialized!
[0d00h00m18s] 2022-12-12T18:11:50 <ERR> [HELPER] New System Status: 0x00000100
[0d00h00m20s] 2022-12-12T18:11:52 <WRN> [MAIN] Initialization completed with errors, but trying to continue...
[0d00h00m35s] 2022-12-12T19:12:07 <ERR> [CAM] is not working anymore (CCamera::CaptureToBasisImage) - most probably caused by a hardware problem (instablility, ...). System will reboot.
[0d00h00m35s] 2022-12-12T19:12:07 <WRN> [OTA] Reboot in 5sec

caco3 commented 1 year ago

I could not find a rule for this.

I will build a test version with 4.4

caco3 commented 1 year ago

Can you try with https://github.com/jomjol/AI-on-the-edge-device/actions/workflows/build.yaml?query=branch%3Afix-fb-init-errors-IDF4.4 ?

I changed IDF back to 4.4.0 and disabled the PSRAM size check as those functions are missing there.

marc4s commented 1 year ago

[0d00h00m12s] 2022-12-12T19:48:08 <ERR> [HELPER] New System Status: 0x00000100
[0d00h00m14s] 2022-12-12T19:48:10 <WRN> [MAIN] Initialization completed with errors, but trying to continue...
[0d00h00m43s] 2022-12-12T20:48:39 <ERR> [CAM] is not working anymore (CCamera::CaptureToBasisImage) - most probably caused by a hardware problem (instablility, ...). System will reboot.
[0d00h00m04s] 2022-12-12T19:48:36 <INF> [MAIN] =================================================
[0d00h00m04s] 2022-12-12T19:48:36 <INF> [MAIN] ==================== Startup ====================
[0d00h00m04s] 2022-12-12T19:48:36 <INF> [MAIN] =================================================
[0d00h00m04s] 2022-12-12T19:48:36 <INF> [MAIN] Branch: 'fix-fb-init-errors', Revision: b59e16a+, Date/Time: 2022-12-12 17:26, Web UI: Development-Branch: fix-fb-init-errors (Commit: b59e16a+)
[0d00h00m04s] 2022-12-12T19:48:36 <INF> [MAIN] Reset reason: Via esp_restart
[0d00h00m05s] 2022-12-12T19:48:37 <INF> [OTA] Update during boot triggered - Update File: /sdcard/firmware/update__fix-fb-init-errors-IDF4.4_(d7724b7).zip
[0d00h00m05s] 2022-12-12T19:48:37 <INF> [OTA] File: /sdcard/firmware/update__fix-fb-init-errors-IDF4.4_(d7724b7).zip Filetype: ZIP
[0d00h01m16s] 2022-12-12T19:49:48 <INF> [OTA] Files unzipped.
[0d00h01m16s] 2022-12-12T19:49:48 <INF> [OTA] Found firmware.bin
[0d00h01m35s] 2022-12-12T19:50:07 <INF> [OTA] Trigger reboot due to firmware update.
[0d00h01m35s] 2022-12-12T19:50:07 <INF> [OTA] Reboot triggered by Software (5s).
[0d00h01m35s] 2022-12-12T19:50:07 <WRN> [OTA] Reboot in 5sec
[0d00h00m04s] 2022-12-12T19:51:01 <INF> [MAIN] =================================================
[0d00h00m04s] 2022-12-12T19:51:01 <INF> [MAIN] ==================== Startup ====================
[0d00h00m04s] 2022-12-12T19:51:01 <INF> [MAIN] =================================================
[0d00h00m04s] 2022-12-12T19:51:01 <INF> [MAIN] Branch: 'fix-fb-init-errors-IDF4.4', Revision: d7724b7, Date/Time: 2022-12-12 19:05, Web UI: Development-Branch: fix-fb-init-errors-IDF4.4 (Commit: d7724b7)
[0d00h00m04s] 2022-12-12T19:51:01 <INF> [MAIN] Reset reason: Via esp_restart
[0d00h00m16s] 2022-12-12T20:04:29 <INF> [SNTP] Status OK
[0d00h00m16s] 2022-12-12T20:04:29 <INF> [SNTP] Time is synced with NTP Server pool.ntp.org
[0d00h00m16s] 2022-12-12T21:04:29 <INF> [SNTP] Time zone set to CET-1CEST,M3.5.0,M10.5.0/3
[0d00h00m16s] 2022-12-12T21:04:29 <INF> [MAIN] =================================================
[0d00h00m16s] 2022-12-12T21:04:29 <INF> [MAIN] ================== Main Started =================
[0d00h00m16s] 2022-12-12T21:04:29 <INF> [MAIN] =================================================
[0d00h00m18s] 2022-12-12T21:04:31 <INF> [MAIN] Initialization completed successfully!
[0d00h00m13s] 2022-12-12T20:06:54 <INF> [SNTP] Status OK
[0d00h00m13s] 2022-12-12T20:06:54 <INF> [SNTP] Time is synced with NTP Server pool.ntp.org
[0d00h00m13s] 2022-12-12T21:06:54 <INF> [SNTP] Time zone set to CET-1CEST,M3.5.0,M10.5.0/3
[0d00h00m13s] 2022-12-12T21:06:54 <INF> [MAIN] =================================================
[0d00h00m13s] 2022-12-12T21:06:54 <INF> [MAIN] ================== Main Started =================
[0d00h00m13s] 2022-12-12T21:06:54 <INF> [MAIN] =================================================
[0d00h00m15s] 2022-12-12T21:06:56 <INF> [MAIN] Initialization completed successfully!

IDF4.4 build looks good. first boot... should I will start the alignment process ... and check if it is fine also after it...result will follow

marc4s commented 1 year ago

sorry. after alignement same like the others before... flashed-> fine -> aligned image,refs...->reboot->bootloop. same with the IDF build...

[0d00h00m04s] 2022-12-12T19:48:36 <INF> [MAIN] =================================================
[0d00h00m04s] 2022-12-12T19:48:36 <INF> [MAIN] ==================== Startup ====================
[0d00h00m04s] 2022-12-12T19:48:36 <INF> [MAIN] =================================================
[0d00h00m04s] 2022-12-12T19:48:36 <INF> [MAIN] Branch: 'fix-fb-init-errors', Revision: b59e16a+, Date/Time: 2022-12-12 17:26, Web UI: Development-Branch: fix-fb-init-errors (Commit: b59e16a+)
[0d00h00m04s] 2022-12-12T19:48:36 <INF> [MAIN] Reset reason: Via esp_restart
[0d00h00m05s] 2022-12-12T19:48:37 <INF> [OTA] Update during boot triggered - Update File: /sdcard/firmware/update__fix-fb-init-errors-IDF4.4_(d7724b7).zip
[0d00h00m05s] 2022-12-12T19:48:37 <INF> [OTA] File: /sdcard/firmware/update__fix-fb-init-errors-IDF4.4_(d7724b7).zip Filetype: ZIP
[0d00h01m16s] 2022-12-12T19:49:48 <INF> [OTA] Files unzipped.
[0d00h01m16s] 2022-12-12T19:49:48 <INF> [OTA] Found firmware.bin
[0d00h01m35s] 2022-12-12T19:50:07 <INF> [OTA] Trigger reboot due to firmware update.
[0d00h01m35s] 2022-12-12T19:50:07 <INF> [OTA] Reboot triggered by Software (5s).
[0d00h01m35s] 2022-12-12T19:50:07 <WRN> [OTA] Reboot in 5sec
[0d00h00m04s] 2022-12-12T19:51:01 <INF> [MAIN] =================================================
[0d00h00m04s] 2022-12-12T19:51:01 <INF> [MAIN] ==================== Startup ====================
[0d00h00m04s] 2022-12-12T19:51:01 <INF> [MAIN] =================================================[0d00h00m04s] 2022-12-12T19:51:01 <INF> [MAIN] Branch: 'fix-fb-init-errors-IDF4.4', Revision: d7724b7, Date/Time: 2022-12-12 19:05, Web UI: Development-Branch: fix-fb-init-errors-IDF4.4 (Commit: d7724b7)
[0d00h00m04s] 2022-12-12T19:51:01 <INF> [MAIN] Reset reason: Via esp_restart
[0d00h00m16s] 2022-12-12T20:04:29 <INF> [SNTP] Status OK
[0d00h00m16s] 2022-12-12T20:04:29 <INF> [SNTP] Time is synced with NTP Server pool.ntp.org
[0d00h00m16s] 2022-12-12T21:04:29 <INF> [SNTP] Time zone set to CET-1CEST,M3.5.0,M10.5.0/3
[0d00h00m16s] 2022-12-12T21:04:29 <INF> [MAIN] =================================================
[0d00h00m16s] 2022-12-12T21:04:29 <INF> [MAIN] ================== Main Started =================
[0d00h00m16s] 2022-12-12T21:04:29 <INF> [MAIN] =================================================
[0d00h00m18s] 2022-12-12T21:04:31 <INF> [MAIN] Initialization completed successfully!
[0d00h00m13s] 2022-12-12T20:06:54 <INF> [SNTP] Status OK
[0d00h00m13s] 2022-12-12T20:06:54 <INF> [SNTP] Time is synced with NTP Server pool.ntp.org
[0d00h00m13s] 2022-12-12T21:06:54 <INF> [SNTP] Time zone set to CET-1CEST,M3.5.0,M10.5.0/3
[0d00h00m13s] 2022-12-12T21:06:54 <INF> [MAIN] =================================================
[0d00h00m13s] 2022-12-12T21:06:54 <INF> [MAIN] ================== Main Started =================
[0d00h00m13s] 2022-12-12T21:06:54 <INF> [MAIN] =================================================
[0d00h00m15s] 2022-12-12T21:06:56 <INF> [MAIN] Initialization completed successfully!
[0d00h00m04s] 2022-12-12T20:17:26 <INF> [MAIN] =================================================
[0d00h00m04s] 2022-12-12T20:17:26 <INF> [MAIN] ==================== Startup ====================
[0d00h00m04s] 2022-12-12T20:17:26 <INF> [MAIN] =================================================
[0d00h00m04s] 2022-12-12T20:17:26 <INF> [MAIN] Branch: 'fix-fb-init-errors-IDF4.4', Revision: d7724b7, Date/Time: 2022-12-12 19:05, Web UI: Development-Branch: fix-fb-init-errors-IDF4.4 (Commit: d7724b7)
[0d00h00m04s] 2022-12-12T20:17:26 <INF> [MAIN] Reset reason: Via esp_restart
[0d00h00m07s] 2022-12-12T20:17:29 <INF> [MAIN] =================================================
[0d00h00m07s] 2022-12-12T20:17:29 <INF> [MAIN] ================== Main Started =================
[0d00h00m07s] 2022-12-12T20:17:29 <INF> [MAIN] =================================================
[0d00h00m11s] 2022-12-12T20:17:33 <WRN> [MAIN] Camera Framebuffer init failed, retrying..
[0d00h00m17s] 2022-12-12T20:17:39 <ERR> [MAIN] Camera Framebuffer cannot be initialized!
[0d00h00m17s] 2022-12-12T20:17:39 <ERR> [HELPER] New System Status: 0x00000100
[0d00h00m19s] 2022-12-12T20:17:41 <WRN> [MAIN] Initialization completed with errors, but trying to continue...
[0d00h00m47s] 2022-12-12T21:18:10 <ERR> [CAM] is not working anymore (CCamera::CaptureToBasisImage) - most probably caused by a hardware problem (instablility, ...). System will reboot.
[0d00h00m04s] 2022-12-12T20:18:27 <INF> [MAIN] =================================================
[0d00h00m04s] 2022-12-12T20:18:27 <INF> [MAIN] ==================== Startup ====================
[0d00h00m04s] 2022-12-12T20:18:27 <INF> [MAIN] =================================================
[0d00h00m04s] 2022-12-12T20:18:27 <INF> [MAIN] Branch: 'fix-fb-init-errors-IDF4.4', Revision: d7724b7, Date/Time: 2022-12-12 19:05, Web UI: Development-Branch: fix-fb-init-errors-IDF4.4 (Commit: d7724b7)
[0d00h00m04s] 2022-12-12T20:18:27 <INF> [MAIN] Reset reason: Via esp_restart
[0d00h00m07s] 2022-12-12T20:18:30 <INF> [MAIN] =================================================
[0d00h00m07s] 2022-12-12T20:18:30 <INF> [MAIN] ================== Main Started =================
[0d00h00m07s] 2022-12-12T20:18:30 <INF> [MAIN] =================================================
[0d00h00m11s] 2022-12-12T20:18:34 <WRN> [MAIN] Camera Framebuffer init failed, retrying..
[0d00h00m17s] 2022-12-12T20:18:40 <ERR> [MAIN] Camera Framebuffer cannot be initialized!
[0d00h00m17s] 2022-12-12T20:18:40 <ERR> [HELPER] New System Status: 0x00000100
[0d00h00m19s] 2022-12-12T20:18:42 <WRN> [MAIN] Initialization completed with errors, but trying to continue...
[0d00h00m36s] 2022-12-12T21:18:59 <ERR> [CAM] is not working anymore (CCamera::CaptureToBasisImage) - most probably caused by a hardware problem (instablility, ...). System will reboot.
[0d00h00m04s] 2022-12-12T20:19:17 <INF> [MAIN] =================================================
[0d00h00m04s] 2022-12-12T20:19:17 <INF> [MAIN] ==================== Startup ====================
[0d00h00m04s] 2022-12-12T20:19:17 <INF> [MAIN] =================================================
[0d00h00m04s] 2022-12-12T20:19:17 <INF> [MAIN] Branch: 'fix-fb-init-errors-IDF4.4', Revision: d7724b7, Date/Time: 2022-12-12 19:05, Web UI: Development-Branch: fix-fb-init-errors-IDF4.4 (Commit: d7724b7)
[0d00h00m04s] 2022-12-12T20:19:17 <INF> [MAIN] Reset reason: Via esp_restart
[0d00h00m07s] 2022-12-12T20:19:20 <INF> [MAIN] =================================================
[0d00h00m07s] 2022-12-12T20:19:20 <INF> [MAIN] ================== Main Started =================
[0d00h00m07s] 2022-12-12T20:19:20 <INF> [MAIN] =================================================
[0d00h00m11s] 2022-12-12T20:19:24 <WRN> [MAIN] Camera Framebuffer init failed, retrying..
[0d00h00m17s] 2022-12-12T20:19:30 <ERR> [MAIN] Camera Framebuffer cannot be initialized!
[0d00h00m17s] 2022-12-12T20:19:30 <ERR> [HELPER] New System Status: 0x00000100
[0d00h00m19s] 2022-12-12T20:19:32 <WRN> [MAIN] Initialization completed with errors, but trying to continue...
[0d00h00m47s] 2022-12-12T21:19:59 <ERR> [CAM] is not working anymore (CCamera::CaptureToBasisImage) - most probably caused by a hardware problem (instablility, ...). System will reboot.
[0d00h00m04s] 2022-12-12T20:20:16 <INF> [MAIN] =================================================
[0d00h00m04s] 2022-12-12T20:20:16 <INF> [MAIN] ==================== Startup ====================
[0d00h00m04s] 2022-12-12T20:20:16 <INF> [MAIN] =================================================
[0d00h00m04s] 2022-12-12T20:20:16 <INF> [MAIN] Branch: 'fix-fb-init-errors-IDF4.4', Revision: d7724b7, Date/Time: 2022-12-12 19:05, Web UI: Development-Branch: fix-fb-init-errors-IDF4.4 (Commit: d7724b7)
[0d00h00m04s] 2022-12-12T20:20:16 <INF> [MAIN] Reset reason: Via esp_restart
[0d00h00m07s] 2022-12-12T20:20:19 <INF> [MAIN] =================================================
[0d00h00m07s] 2022-12-12T20:20:19 <INF> [MAIN] ================== Main Started =================
[0d00h00m07s] 2022-12-12T20:20:19 <INF> [MAIN] =================================================
[0d00h00m11s] 2022-12-12T20:20:23 <WRN> [MAIN] Camera Framebuffer init failed, retrying..
[0d00h00m17s] 2022-12-12T20:20:29 <ERR> [MAIN] Camera Framebuffer cannot be initialized!
[0d00h00m17s] 2022-12-12T20:20:29 <ERR> [HELPER] New System Status: 0x00000100
[0d00h00m19s] 2022-12-12T20:20:31 <WRN> [MAIN] Initialization completed with errors, but trying to continue...
[0d00h00m47s] 2022-12-12T21:20:59 <ERR> [CAM] is not working anymore (CCamera::CaptureToBasisImage) - most probably caused by a hardware problem (instablility, ...). System will reboot.
[0d00h00m04s] 2022-12-12T20:21:32 <INF> [MAIN] =================================================
[0d00h00m04s] 2022-12-12T20:21:32 <INF> [MAIN] ==================== Startup ====================
[0d00h00m04s] 2022-12-12T20:21:32 <INF> [MAIN] =================================================
[0d00h00m04s] 2022-12-12T20:21:32 <INF> [MAIN] Branch: 'fix-fb-init-errors-IDF4.4', Revision: d7724b7, Date/Time: 2022-12-12 19:05, Web UI: Development-Branch: fix-fb-init-errors-IDF4.4 (Commit: d7724b7)
[0d00h00m04s] 2022-12-12T20:21:32 <INF> [MAIN] Reset reason: Via esp_restart
[0d00h00m07s] 2022-12-12T20:21:35 <INF> [MAIN] =================================================
[0d00h00m07s] 2022-12-12T20:21:35 <INF> [MAIN] ================== Main Started =================
[0d00h00m07s] 2022-12-12T20:21:35 <INF> [MAIN] =================================================
[0d00h00m11s] 2022-12-12T20:21:39 <WRN> [MAIN] Camera Framebuffer init failed, retrying..
[0d00h00m17s] 2022-12-12T20:21:45 <ERR> [MAIN] Camera Framebuffer cannot be initialized!
[0d00h00m17s] 2022-12-12T20:21:45 <ERR> [HELPER] New System Status: 0x00000100
[0d00h00m19s] 2022-12-12T20:21:47 <WRN> [MAIN] Initialization completed with errors, but trying to continue...
[0d00h00m47s] 2022-12-12T21:22:14 <ERR> [CAM] is not working anymore (CCamera::CaptureToBasisImage) - most probably caused by a hardware problem (instablility, ...). System will reboot.
[0d00h00m04s] 2022-12-12T20:22:28 <INF> [MAIN] =================================================
[0d00h00m04s] 2022-12-12T20:22:28 <INF> [MAIN] ==================== Startup ====================
[0d00h00m04s] 2022-12-12T20:22:28 <INF> [MAIN] =================================================
[0d00h00m04s] 2022-12-12T20:22:28 <INF> [MAIN] Branch: 'fix-fb-init-errors-IDF4.4', Revision: d7724b7, Date/Time: 2022-12-12 19:05, Web UI: Development-Branch: fix-fb-init-errors-IDF4.4 (Commit: d7724b7)
[0d00h00m04s] 2022-12-12T20:22:28 <INF> [MAIN] Reset reason: Via esp_restart
[0d00h00m07s] 2022-12-12T20:22:31 <INF> [MAIN] =================================================
[0d00h00m07s] 2022-12-12T20:22:31 <INF> [MAIN] ================== Main Started =================
[0d00h00m07s] 2022-12-12T20:22:31 <INF> [MAIN] =================================================
[0d00h00m11s] 2022-12-12T20:22:35 <WRN> [MAIN] Camera Framebuffer init failed, retrying..
[0d00h00m17s] 2022-12-12T20:22:41 <ERR> [MAIN] Camera Framebuffer cannot be initialized!
[0d00h00m17s] 2022-12-12T20:22:41 <ERR> [HELPER] New System Status: 0x00000100
[0d00h00m19s] 2022-12-12T20:22:43 <WRN> [MAIN] Initialization completed with errors, but trying to continue...
[0d00h00m47s] 2022-12-12T21:23:11 <ERR> [CAM] is not working anymore (CCamera::CaptureToBasisImage) - most probably caused by a hardware problem (instablility, ...). System will reboot.
[0d00h00m04s] 2022-12-12T20:23:23 <INF> [MAIN] =================================================
[0d00h00m04s] 2022-12-12T20:23:23 <INF> [MAIN] ==================== Startup ====================
[0d00h00m04s] 2022-12-12T20:23:23 <INF> [MAIN] =================================================
[0d00h00m04s] 2022-12-12T20:23:23 <INF> [MAIN] Branch: 'fix-fb-init-errors-IDF4.4', Revision: d7724b7, Date/Time: 2022-12-12 19:05, Web UI: Development-Branch: fix-fb-init-errors-IDF4.4 (Commit: d7724b7)
[0d00h00m04s] 2022-12-12T20:23:24 <INF> [MAIN] Reset reason: Via esp_restart
[0d00h00m07s] 2022-12-12T20:23:27 <INF> [MAIN] =================================================
[0d00h00m07s] 2022-12-12T20:23:27 <INF> [MAIN] ================== Main Started =================
[0d00h00m07s] 2022-12-12T20:23:27 <INF> [MAIN] =================================================
[0d00h00m11s] 2022-12-12T20:23:31 <WRN> [MAIN] Camera Framebuffer init failed, retrying..
[0d00h00m17s] 2022-12-12T20:23:37 <ERR> [MAIN] Camera Framebuffer cannot be initialized!
[0d00h00m17s] 2022-12-12T20:23:37 <ERR> [HELPER] New System Status: 0x00000100
[0d00h00m19s] 2022-12-12T20:23:39 <WRN> [MAIN] Initialization completed with errors, but trying to continue...
[0d00h00m47s] 2022-12-12T21:24:06 <ERR> [CAM] is not working anymore (CCamera::CaptureToBasisImage) - most probably caused by a hardware problem (instablility, ...). System will reboot.
[0d00h00m04s] 2022-12-12T20:24:21 <INF> [MAIN] =================================================
[0d00h00m04s] 2022-12-12T20:24:21 <INF> [MAIN] ==================== Startup ====================
[0d00h00m04s] 2022-12-12T20:24:21 <INF> [MAIN] =================================================
[0d00h00m04s] 2022-12-12T20:24:21 <INF> [MAIN] Branch: 'fix-fb-init-errors-IDF4.4', Revision: d7724b7, Date/Time: 2022-12-12 19:05, Web UI: Development-Branch: fix-fb-init-errors-IDF4.4 (Commit: d7724b7)
[0d00h00m04s] 2022-12-12T20:24:21 <INF> [MAIN] Reset reason: Via esp_restart
[0d00h00m07s] 2022-12-12T20:24:24 <INF> [MAIN] =================================================
[0d00h00m07s] 2022-12-12T20:24:24 <INF> [MAIN] ================== Main Started =================
[0d00h00m07s] 2022-12-12T20:24:24 <INF> [MAIN] =================================================
[0d00h00m11s] 2022-12-12T20:24:28 <WRN> [MAIN] Camera Framebuffer init failed, retrying..
[0d00h00m17s] 2022-12-12T20:24:34 <ERR> [MAIN] Camera Framebuffer cannot be initialized!
[0d00h00m17s] 2022-12-12T20:24:34 <ERR> [HELPER] New System Status: 0x00000100
[0d00h00m19s] 2022-12-12T20:24:36 <WRN> [MAIN] Initialization completed with errors, but trying to continue...
[0d00h00m47s] 2022-12-12T21:25:04 <ERR> [CAM] is not working anymore (CCamera::CaptureToBasisImage) - most probably caused by a hardware problem (instablility, ...). System will reboot.

marc4s commented 1 year ago

cold-boot seems to "solve" the issue temporary. after coldboot everything is fine until next hotreboot. I tried a coldboot twice which both was successful and with multiple hotreboots between which all was unsuccessful

caco3 commented 1 year ago

As an explanation: On a cold boot resp. a power cycle, the power gets interrupted. On a warm boot reps. restart only the CPU gets restarted, but all chips (PSRAM, Camera, ...) keep power and continue working with their settings.

I thought I also saw this once on a device, but can not reproduce it atm. And that makes it very difficult to trace your issue down!

To sum it up:

12.0.1 was ok?
13.0.x was all bad?
Current rolling with IDF 4.4.0 was bad?

Could you provide success/failure for the individual versions of 13.0.x? See below the link to the auto build versions:

Build of 13.0.1
Build of 13.0.2
Build of 13.0.3
Build of 13.0.4
Build of 13.0.5 (bad if I remember correctly)

This would help us to nail down where we broke something.

BTW, you only need to flash the firmware.zip file!

ednax commented 1 year ago

firmware_v13.0.1(575d59d) -> OTA-Update (without cold-boot) -> works just fine, no error in the log-file (DEBUG-Mode)

firmware_v13.0.2(ecf1e78) -> Camera Framebuffer cannot be initialzed. Will restart in 5 minutes!

so 13.0.2 seems to be the culprit.

ednax commented 1 year ago

just went back from 13.0.2 to 13.0.1 and now I get the same error :( hmm...

I did a cold-boot, and the camera worked again. Then a warm-boot, and now I get the same error. So maybe the error exists within 13.0.1. But why did it work 15 minutes ago? I don't know.

If there is anything more I can do to help find the error, I'd be happy to help.

marc4s commented 1 year ago

12.0.1 was ok? -> yes
13.0.x was all bad? -> I started with 13.0.5 -> but yes it seems to be
Current rolling with IDF 4.4.0 was bad? -> still this issue. yes

marc4s commented 1 year ago

ednax highlighted a good point which was already noticed here but not so clear: If the firmware got updated and warm-rebooted into the new firmware. it was all the time working. but after alignement and then reboot it happend. thats why I first pointed the wrong config-file and I did not noticed that a cold-boot "fixed" it.

btw: I am still on IDF4.4 build and the "keep aspect between ROIs" works for digital ROIs but not for analog ROIs anymore... but this is not critical ;-)

ednax commented 1 year ago

but that is not always the case. When I did update from 12.0.1 to 13.0.5 it did not work. Neither after the Update with a warm-boot, nor after the cold-boot.

btw: what is IDF4.4?

marc4s commented 1 year ago

but that is not always the case. When I did update from 12.0.1 to 13.0.5 it did not work. Neither after the Update with a warm-boot, nor after the cold-boot.

are you sure? this is what exactly worked for me. Updat3ed from 12.0.1 to 13.0.5 -> warmboot-> everything is fine-> reboot stopped it....

btw: what is IDF4.4?

a special build mentioned above: https://github.com/jomjol/AI-on-the-edge-device/actions/workflows/build.yaml?query=branch%3Afix-fb-init-errors-IDF4.4 to understand which change in 13.x caused our issue

marc4s commented 1 year ago

would an option "reload config" may help here to activate changed ROI/ref-image -Data to avoid an reboot. just for analyze and as improvement maybe (but I expect that is far to complicate as if it would be easy it would be already there I think...)

marc4s commented 1 year ago

and if its related to special types of ESP32 it may help to understand which hardware is affected... my ESP32 are this https://de.aliexpress.com/item/1005001900359624.html?spm=a2g0o.order_list.order_list_main.112.42f05c5f4tRsUX&gatewayAdapt=glo2deu manufactor as far as I understand is DM (diymore ) Cam is OV2640 and I am using this ESP32-CAM-MB Board with Powersupply and for easy flashing

caco3 commented 1 year ago

If there is anything more I can do to help find the error, I'd be happy to help.

Since it is not clear if 13.0.1 is ok or not, it would help to just retry a few times so you clearly can say which version is the last working one.

Make sure to test a few times with warm boot (restart) as this seems to be an issue.

"keep aspect between ROIs" works for digital ROIs but not for analog ROIs anymore

I guess that is just an issue on the Web UI (Functionality is implemented in Javascript). If it still is an issue with 13.0.7, please create a separate issue for it.

btw: what is IDF4.4?

ESP-IDF is the framework Espressif provides. It contains the camera driver, OS and much more, see https://github.com/espressif/esp-idf. Correctly speaking, it is not the version of that ESP-IDF, but more the version of the packaging by the PlatformIO project, see https://github.com/platformio/platform-espressif32/releases

would an option "reload config" may help

As you pointed out, it would be quite complex to implement this. It just makes it much simpler to enforce a reboot.

manufactor as far as I understand is DM (diymore )

I am actually not sure how this all works. Those boards all look very similar, so it might be that DIY also just is a re-seller. None of the boards I have seen have any manufacturer informations

ednax commented 1 year ago

Since it is not clear if 13.0.1 is ok or not, it would help to just retry a few times so you clearly can say which version is the last working one.

I just installed 13.0.1 and tried several times a warm boot -> never got an error. Cold-boot: also no errors...

I did the cold and warm boot several times, and let it take at minimum two pictures until the next boot. I could not recreate the error - so 13.0.1 seems to be fine. Maybe something other went wrong, the last time.

caco3 commented 1 year ago

Your testplan sounds ok!

The problem is, there seems to be no essential change between 13.0.1 and 13.0.2! See https://github.com/jomjol/AI-on-the-edge-device/compare/v13.0.1...v13.0.2 -> main.cpp So if we broke something in there, I really do not see what it could be!

Can you do the same tests also with 13.0.2, and if it is ok there also with the newer versions?

ednax commented 1 year ago

ok, I problably need a few days (I never know, when I find the time). oh, one thing: I remember that one setting is different from before. At the moment I have FixedExposure=false - I'm pretty sure, that last time this setting was true. Bevore updating to 13.0.2 I will try again with FixedExposure=true.

caco3 commented 1 year ago

FixedExposure does not have an impact on the ~~PSRAM~~ Framebuffer issue!

ednax commented 1 year ago

ok, I'm not sure what happened. The camera worked a few hours, then I changed something at the config (I activated analog LogImage and LogfileRetentionInDays) and did a reboot - now I get the error again. A warm boot does not work.

2022-12-14T13:11:03 <DBG> [CNN] ZeigerEvalHybridNeu - KEIN Analoger Vorgänger, kein Ziffernwechsel, da Vorkomma weit genug weg = 0 zahl: 9.997297 zahl_vorgaenger = 4.151498 eval_vorgaenger = 4 DigitalUnschaerfe = 0.200000
2022-12-14T13:11:03 <DBG> [CNN] getReadout#ZeigerEvalHybridNeu()= 0
2022-12-14T13:11:03 <DBG> [CNN] getReadout#result= 045158
2022-12-14T13:11:03 <DBG> [CNN] ZeigerEvalHybridNeu - KEIN Analoger Vorgänger, Nulldurchgang hat stattgefunden = 0 zahl: 0.002038 zahl_vorgaenger = 9.997297 eval_vorgaenger = 0 DigitalUnschaerfe = 0.200000
2022-12-14T13:11:03 <DBG> [CNN] getReadout#ZeigerEvalHybridNeu()= 0
2022-12-14T13:11:03 <DBG> [CNN] getReadout#result= 0045158
2022-12-14T13:11:03 <INF> [TFLITE] task_autodoFlow - round #20 done
2022-12-14T13:11:03 <INF> [TFLITE] CPU Temperature: 53.3
2022-12-14T13:11:11 <DBG> [OTA FILE] delete_post_handler
2022-12-14T13:11:11 <DBG> [OTA FILE] download_get_handler
2022-12-14T13:11:11 <DBG> [OTA FILE] upload_post_handler
2022-12-14T13:11:12 <DBG> [OTA FILE] download_get_handler
2022-12-14T13:11:17 <DBG> [OTA FILE] download_get_handler
2022-12-14T13:11:17 <DBG> [OTA FILE] download_get_handler
2022-12-14T13:12:04 <DBG> [OTA FILE] delete_post_handler
2022-12-14T13:12:04 <DBG> [OTA FILE] download_get_handler
2022-12-14T13:12:04 <DBG> [OTA FILE] upload_post_handler
2022-12-14T13:12:04 <DBG> [OTA FILE] download_get_handler
2022-12-14T13:12:11 <DBG> [OTA FILE] download_get_handler
2022-12-14T13:12:11 <DBG> [HEAP] handler_editflow - Start Heap: 3082870 Min Free: 740514 larg. Block: 1998848 SPI Heap: 3060475 NOT_SPI Heap: 22395 Internal Heap: 22395 Internal Min Heap free: 2551
2022-12-14T13:12:22 <DBG> [OTA] handler_reboot
2022-12-14T13:12:22 <INF> [OTA] Reboot triggered by Software (5s).
2022-12-14T13:12:22 <WRN> [OTA] Reboot in 5sec
2022-12-14T12:12:57 <INF> [MAIN] =============================================================================================
2022-12-14T12:12:57 <INF> [MAIN] =================================== Main Started ============================================
2022-12-14T12:12:57 <INF> [MAIN] =============================================================================================
2022-12-14T12:12:57 <INF> [MAIN] Branch: 'HEAD', Revision: 575d59d, Date/Time: 2022-11-28 20:04, Web UI: Release: v13.0.1 (Commit: 575d59d)
2022-12-14T12:12:57 <INF> [MAIN] Reset reason: Via esp_restart
2022-12-14T12:12:57 <WRN> [MAIN] Web UI version does not match firmware version!
2022-12-14T12:13:01 <ERR> [MAIN] Camera Framebuffer cannot be initialzed. Will restart in 5 minutes!
2022-12-14T12:13:03 <ERR> [MAIN] Initialization failed. Will restart in 5 minutes!
2022-12-14T12:17:03 <ERR> [MAIN] Initialization failed. Will restart in 1 minute!
2022-12-14T12:18:04 <ERR> [MAIN] Initialization failed. Will restart now!
2022-12-14T12:18:04 <INF> [OTA] Reboot triggered by Software (5s).
2022-12-14T12:18:04 <WRN> [OTA] Reboot in 5sec
2022-12-14T12:17:22 <INF> [MAIN]

caco3 commented 1 year ago

if it just happens once, you simply should ignore it. It is ok if it fails once in a while.

Keep in mind that the hardware is very low cost and the firmware is not able to handling all possible issues.

marc4s commented 1 year ago

I cannot tell that 13.0.1 is working. it has still the same issue as all the time before. But because I had downgraded with the firmware.bin - then I was able to setup/align everything and after that warm-reboot it mentioned that the webui is wrong (still the old one).. -> then uploaded html_13.0.1 -> reboot and still the webui warning:

2022-12-14T13:46:01 <INF> [MAIN] =============================================================================================
2022-12-14T13:46:01 <INF> [MAIN] =================================== Main Started ============================================
2022-12-14T13:46:01 <INF> [MAIN] =============================================================================================
2022-12-14T13:46:01 <INF> [MAIN] Branch: 'HEAD', Revision: 575d59d, Date/Time: 2022-11-28 20:04, Web UI: Development-Branch: fix-fb-init-errors-IDF4.4 (Commit: d7724b7)
2022-12-14T13:46:01 <INF> [MAIN] Reset reason: Power-on event
2022-12-14T13:46:01 <WRN> [MAIN] Web UI version does not match firmware version!
2022-12-14T13:46:03 <INF> [MAIN] Initialization completed successfully!
2022-12-14T13:46:07 <INF> [FLOW MQTT] Digitizer interval is 5.0 minutes => setting MQTT LWT timeout to 12.5 minutes.
2022-12-14T13:46:07 <INF> [MQTT INTERFACE] URI: mqtt://192.168.178.110:1883, clientname: wasser, user: mqttuser, password: XXXXXXXX, maintopic: wasserzaehler, last-will-topic: wasserzaehler/connection, keepAlive: 750
2022-12-14T13:46:07 <INF> [MQTT INTERFACE] Init
2022-12-14T13:46:07 <INF> [MQTT INTERFACE] Init successful
2022-12-14T13:46:07 <INF> [MQTT INTERFACE] Connected
2022-12-14T13:46:07 <INF> [MQTT SERVER] MQTT - Sending Homeassistant Discovery Topics (Meter Type: water, Value Unit: m³ , Rate Unit: m³/h)...
2022-12-14T13:46:08 <INF> [FLOW MQTT] Digitizer interval is 5.0 minutes => setting MQTT LWT timeout to 12.5 minutes.
2022-12-14T13:46:08 <INF> [MQTT SERVER] Publishing static MQTT topics...
2022-12-14T13:46:08 <INF> [MQTT INTERFACE] Init
2022-12-14T13:46:08 <WRN> [MQTT INTERFACE] Could not start client (ret=-1), retrying...
2022-12-14T13:46:08 <INF> [MQTT INTERFACE] Init successful
2022-12-14T13:46:09 <ERR> [MQTT INTERFACE] Could not start client (ret=-1)!
2022-12-14T13:12:19 <INF> [MAIN] =============================================================================================
2022-12-14T13:12:19 <INF> [MAIN] =================================== Main Started ============================================
2022-12-14T13:12:19 <INF> [MAIN] =============================================================================================
2022-12-14T13:12:19 <INF> [MAIN] Branch: 'HEAD', Revision: 575d59d, Date/Time: 2022-11-28 20:04, Web UI: Development-Branch: fix-fb-init-errors-IDF4.4 (Commit: d7724b7)
2022-12-14T13:12:19 <INF> [MAIN] Reset reason: Via esp_restart
2022-12-14T13:12:19 <WRN> [MAIN] Web UI version does not match firmware version!
2022-12-14T13:12:23 <ERR> [MAIN] Camera Framebuffer cannot be initialzed. Will restart in 5 minutes!
2022-12-14T13:12:25 <ERR> [MAIN] Initialization failed. Will restart in 5 minutes!
2022-12-14T13:16:25 <ERR> [MAIN] Initialization failed. Will restart in 1 minute!
2022-12-14T13:17:25 <ERR> [MAIN] Initialization failed. Will restart now!
2022-12-14T13:17:25 <INF> [OTA] Reboot triggered by Software (5s).
2022-12-14T13:17:25 <WRN> [OTA] Reboot in 5sec
2022-12-14T13:17:52 <INF> [MAIN] =============================================================================================
2022-12-14T13:17:52 <INF> [MAIN] =================================== Main Started ============================================
2022-12-14T13:17:52 <INF> [MAIN] =============================================================================================
2022-12-14T13:17:52 <INF> [MAIN] Branch: 'HEAD', Revision: 575d59d, Date/Time: 2022-11-28 20:04, Web UI: Development-Branch: fix-fb-init-errors-IDF4.4 (Commit: d7724b7)
2022-12-14T13:17:52 <INF> [MAIN] Reset reason: Via esp_restart
2022-12-14T13:17:52 <WRN> [MAIN] Web UI version does not match firmware version!
2022-12-14T13:17:56 <ERR> [MAIN] Camera Framebuffer cannot be initialzed. Will restart in 5 minutes!
2022-12-14T13:17:58 <ERR> [MAIN] Initialization failed. Will restart in 5 minutes!
2022-12-14T13:21:58 <ERR> [MAIN] Initialization failed. Will restart in 1 minute!
2022-12-14T13:22:58 <ERR> [MAIN] Initialization failed. Will restart now!
2022-12-14T13:22:58 <INF> [OTA] Reboot triggered by Software (5s).
2022-12-14T13:22:58 <WRN> [OTA] Reboot in 5sec
2022-12-14T13:23:22 <INF> [MAIN] =============================================================================================
2022-12-14T13:23:22 <INF> [MAIN] =================================== Main Started ============================================
2022-12-14T13:23:22 <INF> [MAIN] =============================================================================================
2022-12-14T13:23:22 <INF> [MAIN] Branch: 'HEAD', Revision: 575d59d, Date/Time: 2022-11-28 20:04, Web UI: Development-Branch: fix-fb-init-errors-IDF4.4 (Commit: d7724b7)
2022-12-14T13:23:22 <INF> [MAIN] Reset reason: Via esp_restart
2022-12-14T13:23:22 <WRN> [MAIN] Web UI version does not match firmware version!
2022-12-14T13:23:26 <ERR> [MAIN] Camera Framebuffer cannot be initialzed. Will restart in 5 minutes!
2022-12-14T13:23:28 <ERR> [MAIN] Initialization failed. Will restart in 5 minutes!
2022-12-14T13:25:13 <INF> [OTA] Update for reboot.
2022-12-14T13:25:15 <INF> [OTA] Reboot triggered by Software (5s).
2022-12-14T13:25:15 <WRN> [OTA] Reboot in 5sec
2022-12-14T13:25:31 <INF> [OTA] Update during boot triggered - Update File: /sdcard/firmware/html__v13.0.1__(575d59d).zip
2022-12-14T13:25:31 <INF> [OTA] File: /sdcard/firmware/html__v13.0.1__(575d59d).zip Filetype: ZIP
2022-12-14T13:25:45 <INF> [OTA] Files unzipped.
2022-12-14T13:25:45 <INF> [OTA] Trigger reboot due to firmware update.
2022-12-14T13:25:46 <INF> [OTA] Reboot triggered by Software (5s).
2022-12-14T13:25:46 <WRN> [OTA] Reboot in 5sec
2022-12-14T13:26:03 <INF> [MAIN] =============================================================================================
2022-12-14T13:26:04 <INF> [MAIN] =================================== Main Started ============================================
2022-12-14T13:26:04 <INF> [MAIN] =============================================================================================
2022-12-14T13:26:04 <INF> [MAIN] Branch: 'HEAD', Revision: 575d59d, Date/Time: 2022-11-28 20:04, Web UI: Release: v13.0.1 (Commit: 575d59d)
2022-12-14T13:26:04 <INF> [MAIN] Reset reason: Via esp_restart
2022-12-14T13:26:04 <WRN> [MAIN] Web UI version does not match firmware version!
2022-12-14T13:26:08 <ERR> [MAIN] Camera Framebuffer cannot be initialzed. Will restart in 5 minutes!
2022-12-14T13:26:10 <ERR> [MAIN] Initialization failed. Will restart in 5 minutes!

which looks kind strange for me I think I need to do a complete refresh... what do you think?

btw. not too long time ago I also played with the fixedexposure but I somehow it was not working so I manually switched back... may it be related to this somehow... is there a cache?

ednax commented 1 year ago

if it just happens once, you simply should ignore it. It is ok if it fails once in a while.

no, it stopped to work from this point on entirely. After a cold-boot it worked again.

With 12.0.1 it was very stable and I never noticed a failure.

caco3 commented 1 year ago

Like state before, fixedexposure does not have an impact to the framebuffer error! So changing it most likely does not help or make it worse.

Regarding the refresh, there are 3 components:

firmware
web UI
config.ini

Point 1 is clear, there might be a change which does not work well with your hardware. Point 2 has no impact. The warning about the version mismatch can be ignored for your analysis Point 3 could have impact, but again I don't see a reason how it should.

As you can see in https://github.com/jomjol/AI-on-the-edge-device/compare/v12.0.1...v13.0.1#diff-2773902c8e6d0ea3798e839063d7b39aa9b94a2e27c405ae00cadf0ee66ca2de, there is not much change in the initialization between 12.0.1 and 13.0.1. The biggest change might be the upgrade of the IDF from 4.4.0 to 5.2.0.

To try to nail it further down, I created a firmware based on 13.0.7 but with the IDF and main.cpp from 12.0.1. Please try how it works: https://github.com/jomjol/AI-on-the-edge-device/actions/workflows/build.yaml?query=branch%3Av13.0.7-with-IDF-and-main.cpp-from-v12.0.1 Note: You will have to disable MQTT as the old IDF crashes with the recent changes to the MQTT functionality!

And please understand that this is one of the last tries I am able to do blindly. You two seem not to be able to find a common last version which works, so it is impossible to nail it down. There are many other users which are fine with the latest release 13.0.7! This points to hardware issues on your side, sry!

To go further, I would need malfunctioning hardware. But even then I don't really have the time to go deeply into it!

marc4s commented 1 year ago

I absolutly understand you. As the hardware is fine from (my point of view) because I have multiple devices of the same type and I already exchanged same randomly...

result for the last release:

2022-12-15T06:22:04 <INF> [OTA] Update during boot triggered - Update File: /sdcard/firmware/update__v13.0.7-with-IDF-and-main.cpp-from-v12.0.1_(b993d25).zip
2022-12-15T06:22:04 <INF> [OTA] File: /sdcard/firmware/update__v13.0.7-with-IDF-and-main.cpp-from-v12.0.1_(b993d25).zip Filetype: ZIP
2022-12-15T06:23:12 <INF> [OTA] Files unzipped.
2022-12-15T06:23:12 <INF> [OTA] Found firmware.bin
2022-12-15T06:23:29 <INF> [OTA] Trigger reboot due to firmware update.
2022-12-15T06:23:29 <INF> [OTA] Reboot triggered by Software (5s).
2022-12-15T06:23:29 <WRN> [OTA] Reboot in 5sec
[       7] 2022-12-15T06:23:40 <INF> [main] =============================================================================================
[       7] 2022-12-15T06:23:40 <INF> [main] =================================== Main Started ============================================
[       7] 2022-12-15T06:23:40 <INF> [main] =============================================================================================
[       7] 2022-12-15T06:23:40 <INF> [main] Branch: 'v13.0.7-with-IDF-and-main.cpp-from-v12.0.1', Tag: '', Revision: b993d25, Date/Time: 2022-12-14 21:09
[      11] 2022-12-15T06:23:44 <INF> [main] Camera cannot be initialzed. System will reboot.
[      11] 2022-12-15T06:23:44 <INF> [OTA] Reboot triggered by Software (5s).
[      11] 2022-12-15T06:23:44 <WRN> [OTA] Reboot in 5sec
[       7] 2022-12-15T06:24:03 <INF> [main] =============================================================================================
[       7] 2022-12-15T06:24:03 <INF> [main] =================================== Main Started ============================================
[       7] 2022-12-15T06:24:03 <INF> [main] =============================================================================================
[       7] 2022-12-15T06:24:03 <INF> [main] Branch: 'v13.0.7-with-IDF-and-main.cpp-from-v12.0.1', Tag: '', Revision: b993d25, Date/Time: 2022-12-14 21:09
[      11] 2022-12-15T06:24:07 <INF> [main] Camera cannot be initialzed. System will reboot.
[      11] 2022-12-15T06:24:07 <INF> [OTA] Reboot triggered by Software (5s).
[      11] 2022-12-15T06:24:07 <WRN> [OTA] Reboot in 5sec

thanks for your effort . I need to stay at 12.0.1 - thats ok - even I really like the new 13.x features.

caco3 commented 1 year ago

If you want, you can send me one of your bad devices and can have a look on it. I am located in Switzerland, just send me an email at caco3 [at ]ruinelli.ch

ednax commented 1 year ago

I just installed the "firmware__v13.0.7-with-IDF-and-main.cpp-from-v12.0.1":

I get an MQTT error although I have deactivated MQTT in the config (I just ignore it)
And I get the "Version of the Web Interface does not match the Firmware Version" Error althoug I uploaded the matching html-package (I also ignore it).
I did several reboots, and it seems to work. But I will use this firmware a few days, do a bit of testing and then I will report here.

caco3 commented 1 year ago

@ednax Do you have an update?

If it is working now it either depends on the IDF version or the initialization in main.cpp.

I made a new version where only main.cpp got reverted to 12.0.1: https://github.com/jomjol/AI-on-the-edge-device/actions/workflows/build.yaml?query=branch%3Av13.0.7-with-main.cpp-from-v12.0.1

And I tried to create a table, please update it accordingly:

firmware	main.cpp	IDF	Outcome	Build
13.0.7	13.0.7	13.0.7	Unstable	See release
13.0.7	12.0.1	12.0.1	Stable	v13.0.7-with-IDF-and-main.cpp-from-v12.0.1 https://github.com/jomjol/AI-on-the-edge-device/actions/workflows/build.yaml?query=branch%3Av13.0.7-with-IDF-and-main.cpp-from-v12.0.1
13.0.7	12.0.1	13.0.7	?	v13.0.7-with-main.cpp-from-v12.0.1 https://github.com/jomjol/AI-on-the-edge-device/actions/workflows/build.yaml?query=branch%3Av13.0.7-with-main.cpp-from-v12.0.1
13.0.7	13.0.7	12.0.1	?	fix-fb-init-errors-IDF4.4 https://github.com/jomjol/AI-on-the-edge-device/actions/workflows/build.yaml?query=branch%3Afix-fb-init-errors-IDF4.4

ednax commented 1 year ago

v13.0.7-with-IDF-and-main.cpp-from-v12.0.1 seems to be very stable. I just installed v13.0.7-with-main.cpp-from-v12.0.1. The next few days I'm maybe not able to test, but every few days I'm at home and will keep you updated. Next time I will try fix-fb-init-errors-IDF4.4 - it might take some days...

ednax commented 1 year ago

ok, "v13.0.7-with-main.cpp-from-v12.0.1" does work, but sometimes it crashes after a reboot:

[ 42716] 2022-12-18T12:19:14 <INF> [OTA] Reboot triggered by Software (5s).
[ 42716] 2022-12-18T12:19:14 <WRN> [OTA] Reboot in 5sec
[ 8] 2022-12-18T11:14:18 <INF> [main] =============================================================================================
[ 8] 2022-12-18T11:14:18 <INF> [main] =================================== Main Started ============================================
[ 8] 2022-12-18T11:14:18 <INF> [main] =============================================================================================
[ 8] 2022-12-18T11:14:18 <INF> [main] Branch: 'v13.0.7-with-main.cpp-from-v12.0.1', Tag: '', Revision: ba48f4d+, Date/Time: 2022-12-17 20:49
[ 18] 2022-12-18T12:14:29 <INF> [SNTP] Time zone set to CET-1CEST,M3.5.0,M10.5.0/3
[ 18] 2022-12-18T12:14:29 <DBG> [TFLITE SERVER] ----------------------------------------------------------------
[ 19] 2022-12-18T12:14:29 <INF> [TFLITE SERVER] Round #1 started
[ 19] 2022-12-18T12:14:29 <ERR> [MQTT IF] Init with no Client_ID () or Last Will Topic (). Abort Init!
[ 19] 2022-12-18T12:14:29 <ERR> [MQTT IF] Init failed, skipping all MQTT publishings in this round!
[ 33] 2022-12-18T12:14:43 <DBG> [CNN] doFlow nach Alignment
[ 8] 2022-12-18T11:29:16 <INF> [main] =============================================================================================
[ 8] 2022-12-18T11:29:16 <INF> [main] =================================== Main Started ============================================
[ 8] 2022-12-18T11:29:16 <INF> [main] =============================================================================================
[ 8] 2022-12-18T11:29:16 <INF> [main] Branch: 'v13.0.7-with-main.cpp-from-v12.0.1', Tag: '', Revision: ba48f4d+, Date/Time: 2022-12-17 20:49
[ 10] 2022-12-18T11:29:18 <WRN> [TFLITE SERVER] Restarted due to an Exception/panic! Postponing first round start by 5 minutes to allow for an OTA or to fetch the log!
[ 10] 2022-12-18T11:29:18 <WRN> [TFLITE SERVER] Setting logfile level to DEBUG until the next reboot!
[ 20] 2022-12-18T11:29:27 <DBG> [OTA FILE] log_get_last_part_handler

I'm not sure, but the time in the logfile seems to be a bit off, after the reboot. That ist probably normal,until the time gets refreshed?

But this is not the "Camera Framebuffer cannot be initialzed" error. After a reboot, it works again. I'm testing a few days more, with this version.

caco3 commented 1 year ago

yes, the time is inaccurate resp. wrong until NTP got synced. The crashes you see are something else, maybe already fixed in rolling.

I'm testing a few days more, with this version.

Thanks! If your experience is also right for others it would mean that the update of IDF causes those new issues. We then should think if we should revert to 4.4. Personally I would not want to do this as we would miss quite some enhancements and stabilizations on other places.

Lets see what @marc4s finds out.

caco3 commented 1 year ago

Updated table:

firmware	main.cpp	IDF	Outcome @ednax	Build
13.0.7	13.0.7	13.0.7	Unstable	See release
13.0.7	12.0.1	12.0.1	Stable	v13.0.7-with-IDF-and-main.cpp-from-v12.0.1 https://github.com/jomjol/AI-on-the-edge-device/actions/workflows/build.yaml?query=branch%3Av13.0.7-with-IDF-and-main.cpp-from-v12.0.1
13.0.7	12.0.1	13.0.7	mostly stable (non-related issues)	v13.0.7-with-main.cpp-from-v12.0.1 https://github.com/jomjol/AI-on-the-edge-device/actions/workflows/build.yaml?query=branch%3Av13.0.7-with-main.cpp-from-v12.0.1
13.0.7	13.0.7	12.0.1	?	fix-fb-init-errors-IDF4.4 https://github.com/jomjol/AI-on-the-edge-device/actions/workflows/build.yaml?query=branch%3Afix-fb-init-errors-IDF4.4

@marc4s I am not sure about your outcomes.

@ both, please copy the table and update yourself, it is hard for me to keep track.

marc4s commented 1 year ago

since i would have to enter "unstable" in every place except 12.0.1 in the table, i had the following observation over the weekend and broke the whole thing down to a different problem:

i have several identical devices that were stable with 12.0.1 and were all absolutely unstable/unusable after the update to 13.0.5/7 and the Debug builds.
I have now grabbed another device and reflashed it AND started with a new SD card. Funnily enough, 13.0.7 was now absolutely stable.

Now I've played around some more and I'm sure I can narrow down the config.ini as the cause. So I am back at my original-theory that something is broken there which I already mentioned in the old bugreport. but I do not understand whats may causing this issue config.zip

so hopefully this is really the case what I found here. but I switched between both files now 3 times and I am sure that on my side this is causing it.

diff /media/sdcard/config /media/sdcard/config_unstable 

diff --color /media/sdcard/config.ini /media/sdcard/config_unstable/config.ini
2c2
< ;LogImageLocation = /log/source
---
> LogImageLocation = /log/source
4c4
< ;LogfileRetentionInDays = 15
---
> LogfileRetentionInDays = 15
6,9c6,9
< Contrast = 2
< Saturation = 2
< LEDIntensity = 100
< ImageQuality = 12
---
> Contrast = 0
> Saturation = 0
> LEDIntensity = 70
> ImageQuality = 0
14c14
< InitialRotate = 94
---
> InitialRotate = 175
20,21c20,21
< /config/ref0.jpg 179 206
< /config/ref1.jpg 477 315
---
> /config/ref0.jpg 212 235
> /config/ref1.jpg 383 190
24c24
< Model = /config/dig-cont_0600_s3.tflite
---
> Model = /config/dig-cont_0570_s3.tflite
28,31c28,31
< main.dig0 319 87 24 43 0
< main.dig1 358 87 24 43 false
< main.dig2 397 87 24 43 false
< main.dig3 436 87 24 43 false
---
> main.dig1 233 185 21 39 false
> main.dig2 269 185 22 40 false
> main.dig3 306 185 23 42 false
> main.dig4 343 185 23 42 0
34c34
< Model = /config/ana-class100_0130_s1_q.tflite
---
> Model = /config/ana-cont_11.3.0_s2.tflite
36,40c36,40
< LogfileRetentionInDays = 5
< main.ana1 472 172 75 75 false
< main.ana2 428 253 75 75 false
< main.ana3 331 271 75 75 false
< main.ana4 237 239 75 75 false
---
> ;LogfileRetentionInDays = 3
> main.ana1 381 269 62 62 false
> main.ana2 335 349 67 67 false
> main.ana3 246 373 86 86 false
> main.ana4 144 333 88 88 false
57,58c57,58
< MainTopic = wasserzaehlertest
< ClientID = wassertest
---
> MainTopic = wasserzaehler
> ClientID = wasser
62,63c62,63
< HomeassistantDiscovery = true
< MeterType = water_m3
---
> ;HomeassistantDiscovery = true
> ;MeterType = other
85c85
< Intervall = 5
---
> Intervall = 4.85
92,93c92,93
< ;Logfile = 
< ;LogfileRetentionInDays = 3
---
> Logfile = 2
> LogfileRetentionInDays = 3
Nur in /media/sdcard/config_unstable: dig-class100-0130_s2_q.tflite.
Nur in /media/sdcard/config: dig-class100-0140_s2_q.tflite.
Nur in /media/sdcard/config_unstable: dig-class11_1430_s2.tflite.
Nur in /media/sdcard/config: dig-class11_1500_s2.tflite.
Nur in /media/sdcard/config_unstable: dig-cont_0570_s3.tflite.
Nur in /media/sdcard/config: dig-cont_0600_s3.tflite.
Binärdateien /media/sdcard/config/ref0.jpg und /media/sdcard/config_unstable/ref0.jpg sind verschieden.
Binärdateien /media/sdcard/config/ref0_org.jpg und /media/sdcard/config_unstable/ref0_org.jpg sind verschieden.
Binärdateien /media/sdcard/config/ref1.jpg und /media/sdcard/config_unstable/ref1.jpg sind verschieden.
Binärdateien /media/sdcard/config/ref1_org.jpg und /media/sdcard/config_unstable/ref1_org.jpg sind verschieden.
Binärdateien /media/sdcard/config/reference.jpg und /media/sdcard/config_unstable/reference.jpg sind verschieden.

since I simply backed up my original sd card with 12.0.1 over the now functioning 13.0.7, I hope that other things do not cause the problem, but maybe this will still give me the right clue as to the direction to take.

Slider0007 commented 1 year ago

@marc4s: In your "unstable" config the image quality is set to 0 compared to 12 in the "stable" and "default" version. This could cause instability due to limited RAM.

Compare the remark in config screen (expert mode has to activated):

jomjol commented 1 year ago

@marc4s : this is a know bug / problem and has been mentioned in the changelog at that point: https://github.com/jomjol/AI-on-the-edge-device/blob/rolling/Changelog.md#1030---2022-01-29

Truely - hard to find if you are not aware of this.

ednax commented 1 year ago

Hi,

Merry Christmas everyone :)

It took me some time to do further testing, because I was sic in bed. But now I'm a bit better and did try out the "fix-fb-init-errors-IDF4.4" Version. I can't get this version to work.

firmware	main.cpp	IDF	Outcome @ednax	Build
13.0.7	13.0.7	13.0.7	Unstable	See release
13.0.7	12.0.1	12.0.1	Stable	v13.0.7-with-IDF-and-main.cpp-from-v12.0.1 https://github.com/jomjol/AI-on-the-edge-device/actions/workflows/build.yaml?query=branch%3Av13.0.7-with-IDF-and-main.cpp-from-v12.0.1
13.0.7	12.0.1	13.0.7	mostly stable (non-related issues)	v13.0.7-with-main.cpp-from-v12.0.1 https://github.com/jomjol/AI-on-the-edge-device/actions/workflows/build.yaml?query=branch%3Av13.0.7-with-main.cpp-from-v12.0.1
13.0.7	13.0.7	12.0.1	does not work at all - camera framebuffer error	fix-fb-init-errors-IDF4.4 https://github.com/jomjol/AI-on-the-edge-device/actions/workflows/build.yaml?query=branch%3Afix-fb-init-errors-IDF4.4
13.0.8	13.0.8	?	same as fix-fb-init-errors-IDF4.4	https://github.com/jomjol/AI-on-the-edge-device/releases/tag/v13.0.8

btw: ImageQuality is set to 13.