PSRAM does not work in Person Detection example with XIAO ESP32S3 Sense and no PSRAM-related configuration option available in menuconfig (TFMIC-9)

[chrisy810]

My board is XIAO ESP32S3 Sense, which has 8MB of PSRAM. When testing out the person detection example, after setting up the camera pins, I was able to successfully run it with camera frame buffers allocated to DRAM. If I allocate to PSRAM, I will receive the error below when initializing. I also was not able to find PSRAM related settings in menuconfig despite being on the latest ESP-IDF version, while these options are available in a new project. My development environment is VSCode on Windows 11. Same issue persists when switching to a different Windows 11 machine with the same setup and source code, and when switching to a different XIAO ESP32S3 Sense instance. PSRAM also functions properly for examples in Arduino IDE that require PSRAM.

Build:Mar 27 2021
rst:0x15 (USB_UART_CHIP_RESET),boot:0x8 (SPI_FAST_FLASH_BOOT)
Saved PC:0x40378b12
0x40378b12: esp_cpu_wait_for_intr at C:/Users/czy01/esp/esp-idf/components/esp_hw_support/cpu.c:121

SPIWP:0xee
mode:DIO, clock div:1
load:0x3fce3818,len:0x18dc
load:0x403c9700,len:0x4
load:0x403c9704,len:0xda4
load:0x403cc700,len:0x3084
entry 0x403c9934
I (26) boot: ESP-IDF v5.1.2 2nd stage bootloader
I (27) boot: compile time Nov 18 2023 01:13:10
I (27) boot: Multicore bootloader
I (30) boot: chip revision: v0.2
I (34) qio_mode: Enabling default flash chip QIO
I (39) boot.esp32s3: Boot SPI Speed : 80MHz
I (44) boot.esp32s3: SPI Mode       : QIO
I (48) boot.esp32s3: SPI Flash Size : 8MB
I (53) boot: Enabling RNG early entropy source...
I (58) boot: Partition Table:
I (62) boot: ## Label            Usage          Type ST Offset   Length
I (69) boot:  0 nvs              WiFi data        01 02 00009000 00006000
I (77) boot:  1 phy_init         RF data          01 01 0000f000 00001000
I (84) boot:  2 factory          factory app      00 00 00010000 00180000
I (92) boot: End of partition table
I (96) esp_image: segment 0: paddr=00010020 vaddr=3c070020 size=6afb0h (438192) map
I (171) esp_image: segment 1: paddr=0007afd8 vaddr=3fc92600 size=03994h ( 14740) load
I (174) esp_image: segment 2: paddr=0007e974 vaddr=40374000 size=016a4h (  5796) load
I (178) esp_image: segment 3: paddr=00080020 vaddr=42000020 size=62718h (403224) map
I (246) esp_image: segment 4: paddr=000e2740 vaddr=403756a4 size=0ce80h ( 52864) load
I (263) boot: Loaded app from partition at offset 0x10000
I (263) boot: Disabling RNG early entropy source...
I (274) cpu_start: Multicore app
I (275) cpu_start: Pro cpu up.
I (275) cpu_start: Starting app cpu, entry point is 0x4037545c
0x4037545c: call_start_cpu1 at C:/Users/czy01/esp/esp-idf/components/esp_system/port/cpu_start.c:157

I (0) cpu_start: App cpu up.
I (293) cpu_start: Pro cpu start user code
I (293) cpu_start: cpu freq: 240000000 Hz
I (293) cpu_start: Application information:
I (296) cpu_start: Project name:     person_detection
I (301) cpu_start: App version:      v1.2.0-16-g46d0cc9-dirty
I (308) cpu_start: Compile time:     Nov 18 2023 01:12:50
I (314) cpu_start: ELF file SHA256:  2f048b14dd9e9bc6...
I (320) cpu_start: ESP-IDF:          v5.1.2
I (325) cpu_start: Min chip rev:     v0.0
I (329) cpu_start: Max chip rev:     v0.99
I (334) cpu_start: Chip rev:         v0.2
I (339) heap_init: Initializing. RAM available for dynamic allocation:
I (346) heap_init: At 3FC98088 len 00051688 (325 KiB): DRAM
I (352) heap_init: At 3FCE9710 len 00005724 (21 KiB): STACK/DRAM
I (359) heap_init: At 600FE010 len 00001FD8 (7 KiB): RTCRAM
I (366) spi_flash: detected chip: gd
I (370) spi_flash: flash io: qio
I (375) sleep: Configure to isolate all GPIO pins in sleep state
I (380) sleep: Enable automatic switching of GPIO sleep configuration
I (388) app_start: Starting scheduler on CPU0
I (393) app_start: Starting scheduler on CPU1
I (393) main_task: Started on CPU0
I (403) main_task: Calling app_main()
I (423) s3 ll_cam: DMA Channel=4
I (423) cam_hal: cam init ok
I (423) sccb: pin_sda 40 pin_scl 39
I (423) sccb: sccb_i2c_port=1
I (433) camera: Detected camera at address=0x30
I (433) camera: Detected OV2640 camera
I (433) camera: Camera PID=0x26 VER=0x42 MIDL=0x7f MIDH=0xa2
I (513) s3 ll_cam: node_size: 3072, nodes_per_line: 1, lines_per_node: 16
I (513) s3 ll_cam: dma_half_buffer_min:  3072, dma_half_buffer:  9216, lines_per_half_buffer: 48, dma_buffer_size: 27648
I (523) cam_hal: buffer_size: 27648, half_buffer_size: 9216, node_buffer_size: 3072, node_cnt: 9, total_cnt: 2
I (533) cam_hal: Allocating 9216 Byte frame buffer in PSRAM
E (543) cam_hal: cam_dma_config(300): frame buffer malloc failed
E (543) cam_hal: cam_config(384): cam_dma_config failed
E (553) camera: Camera config failed with error 0xffffffff
E (553) app_camera: Camera init failed with error 0xffffffff
Camera init failed

InitCamera failed

E (563) app_camera: Camera capture failed
Image capture failed.
person score:25%, no person score 75%
Total execution time: 60380 microseconds```

[chrisy810]

After tens of hours of testing, I discovered a fix (along with more issues, which I have solved).

By chance, I discovered that by enabling ESP32 S3 EYE BSP option in menuconfig (instead of the camera pin out of my specific board as an option I added manually), PSRAM would function normally and the system would crash at initializing the camera. The latter part is expected, but the former part gave me the intuition to later discover the fix.

The fix is to install the generic BSP dependency: espressif/esp_bsp_generic. I then ran into an issue where when I try to add any dependency through command line, I would get this:

Executing action: add-dependency
ERROR: Environment variable "TFLITE_USE_BSP_S3_EYE" is not set

I have no idea why this environment variable is related to adding dependencies, nor do I think I changed something I shouldn't to cause this to happen. I got around this problem by adding the dependency directly to the idf_component.yml file, which allowed the project to correctly build with this generic BSP dependency.

After which, everything worked out smoothly. PSRAM was successfully initialized and frame buffers were allocated to the PSRAM. I don't have enough technical experience to find out the root cause of this issue, heck, I can't even figure out how the sdkconfig file is generated; I could have just manually added the PSRAM configuration options if I figured out how sdkconfig file worked. That said, this workaround does solve the problem, so I will leave it to the author of this repository to figure out.

[igrr]

For the lack of psram option, we probably have to add esp_psram component over here:

https://github.com/espressif/esp-tflite-micro/blob/46d0cc95f6b5cd6e69b57bf7bd2f85483357c8b6/examples/person_detection/CMakeLists.txt#L2

/cc @tore-espressif regarding the add-dependency issue.

[tore-espressif]

@chrisy810 Thank you for the detailed report! I could reproduce all the issues you mentioned. We recently refactored the examples to support more boards (and especially boards with cameras and displays), but unfortunately these regressions happened

1. PSRAM must be included in the list of dependencies, as @igrr pointed out. I'll post a fix soon
2. You're right, that there shouldn't be any error when you try to add more dependencies with idf.py add-dependency, I'll follow this up internally with the idf-component-manager team

EDIT: The second issue will be fixed in upcoming release of idf-component-manager

[vikramdattu]

@chrisy810 the issues should have been fixed with this change already merged to master: https://github.com/espressif/esp-tflite-micro/commit/85fdecd262413f18e0a9c11909710bcc8c034283

[dskw]

When will this change be published with a new release? ~Or can I test this via vscode esp-idf against master? (could not find any info how to set this up)~

[dskw]

Ok, so I found the correct pin config for the camera on Seeed Studio XIAO ESP32S3 Sense: https://github.com/Seeed-Studio/SSCMA-Micro/blob/dev/porting/espressif/boards/seeed_xiao_esp32s3/board.h

But HREF and Y9 are outside the range of the kconfig settings for these pins.

After adjusting the ranges, settings the custom cam default pins to the ones from the link I do not get the PSRAM error any longer, but instant cpu1 crashes:

Boot/Exception log: https://pastebin.com/uWxh9wxf

[chrisy810]

Here is how I configured XIAO ESP32S3 for the person detection example:

1. Add the following to app_camera_esp.h (follow the existing pattern in the file):

#elif CONFIG_CAMERA_MODULE_XIAO_ESP32S3
#define CAMERA_MODULE_NAME "XIAO_ESP32S3"
#define CAMERA_PIN_PWDN   -1
#define CAMERA_PIN_RESET  -1
#define CAMERA_PIN_XCLK   10
#define CAMERA_PIN_SIOD   40
#define CAMERA_PIN_SIOC   39

#define CAMERA_PIN_D7     48
#define CAMERA_PIN_D6     11
#define CAMERA_PIN_D5     12
#define CAMERA_PIN_D4     14
#define CAMERA_PIN_D3     16
#define CAMERA_PIN_D2     18
#define CAMERA_PIN_D1     17
#define CAMERA_PIN_D0     15
#define CAMERA_PIN_VSYNC  38
#define CAMERA_PIN_HREF   47
#define CAMERA_PIN_PCLK   13

2. Add the following to Kconfig.projbuild under menu "Camera Configuration" (follow the existing pattern):

config CAMERA_MODULE_XIAO_ESP32S3
        bool "XIAO-ESP32S3 by SEEED Studio"

3. Install the generic BSP package. The issue I previously faced and described in this GitHub Issue should have been fixed. Read the previous comments to learn about the problem if you face any issue installing it.

4. Go to menuconfig and select CAMERA_MODULE_XIAO_ESP32S3 as the camera module under Application Configuration

5. Enable PSRAM in menuconfig if you plan on using PSRAM. This is done with the following line in app_camera_esp.c:

config.fb_location =   CAMERA_FB_IN_PSRAM; // For DRAM, use "CAMERA_FB_IN_PSRAM"

Please let me know if you face more issues. I recently completed a TinyML project using XIAO ESP32S3. I should be fairly familiar with the person detection example in this repo, which I used as a framework to build my project upon.

[dskw]

I need to reinstall esp-idf without VSCode. This managed environment is only causing issues for me. After several tries to get it to use the new Kconfig entries I managed to rebuilt it with your suggestions. And it finally works now :)

So the main points are indeed the added BSP dependency: espressif/esp_bsp_generic Adding the config table to app_camera_esp.h and config entry to Kconfig.projbuild Then I had to clean the project, build first, THEN do menuconfig and was finally able to select the new entry.

Thanks @chrisy810 !

Oh and merry christmas!

[NavodPeiris]

you can allocate memory for your TensorArena on PSRAM. change this block on .ino file:

if (tensor_arena == NULL) {
    //allocate memory for TensorArena on PSRAM
    tensor_arena = (uint8_t *) ps_malloc(kTensorArenaSize);
  }