Support for chip from espressif family with Zephyr

[OgnjenX]

Is it possible to use this module on an ESP chip (I'm trying to use it on ESP32)? I'm able to build 'micro_ros_zephyr_module' using 'west build -b disco_l475_iot1' but I'm having errors caused by 'app.overlay' content when building with 'west build -p auto -b esp32_devkitc_wroom':

-- west build: generating a build systemwroom
Loading Zephyr default modules (Zephyr base).
-- Application: /home/ognjen/zephyrproject/applications/micRobbit-firmware/app/modules/micro_ros_zephyr_module
-- CMake version: 3.22.1
-- Found Python3: /usr/bin/python3 (found suitable version "3.10.12", minimum required is "3.8") found components: Interpreter 
-- Cache files will be written to: /home/ognjen/.cache/zephyr
-- Zephyr version: 3.5.99 (/home/ognjen/zephyrproject/zephyr)
-- Found west (found suitable version "1.2.0", minimum required is "0.14.0")
-- Board: esp32_devkitc_wroom
-- ZEPHYR_TOOLCHAIN_VARIANT not set, trying to locate Zephyr SDK
-- Found host-tools: zephyr 0.16.3 (/opt/zephyr-sdk-0.16.3)
-- Found toolchain: zephyr 0.16.3 (/opt/zephyr-sdk-0.16.3)
-- Found Dtc: /opt/zephyr-sdk-0.16.3/sysroots/x86_64-pokysdk-linux/usr/bin/dtc (found suitable version "1.6.0", minimum required is "1.4.6") 
-- Found BOARD.dts: /home/ognjen/zephyrproject/zephyr/boards/xtensa/esp32_devkitc_wroom/esp32_devkitc_wroom.dts
-- Found devicetree overlay: /home/ognjen/zephyrproject/applications/micRobbit-firmware/app/modules/micro_ros_zephyr_module/app.overlay
devicetree error: /home/ognjen/zephyrproject/applications/micRobbit-firmware/app/modules/micro_ros_zephyr_module/app.overlay:2 (column 14): parse error: undefined node label 'usbotg_fs'
-- In: /home/ognjen/zephyrproject/applications/micRobbit-firmware/app/modules/micro_ros_zephyr_module/build/zephyr, command: /usr/bin/python3;/home/ognjen/zephyrproject/zephyr/scripts/dts/gen_defines.py;--dts;/home/ognjen/zephyrproject/applications/micRobbit-firmware/app/modules/micro_ros_zephyr_module/build/zephyr/zephyr.dts.pre;--dtc-flags;'';--bindings-dirs;/home/ognjen/zephyrproject/zephyr/dts/bindings;--header-out;/home/ognjen/zephyrproject/applications/micRobbit-firmware/app/modules/micro_ros_zephyr_module/build/zephyr/include/generated/devicetree_generated.h.new;--dts-out;/home/ognjen/zephyrproject/applications/micRobbit-firmware/app/modules/micro_ros_zephyr_module/build/zephyr/zephyr.dts.new;--edt-pickle-out;/home/ognjen/zephyrproject/applications/micRobbit-firmware/app/modules/micro_ros_zephyr_module/build/zephyr/edt.pickle;--vendor-prefixes;/home/ognjen/zephyrproject/zephyr/dts/bindings/vendor-prefixes.txt
CMake Error at /home/ognjen/zephyrproject/zephyr/cmake/modules/dts.cmake:268 (message):
  gen_defines.py failed with return code: 1
Call Stack (most recent call first):
  /home/ognjen/zephyrproject/zephyr/cmake/modules/zephyr_default.cmake:129 (include)
  /home/ognjen/zephyrproject/zephyr/share/zephyr-package/cmake/ZephyrConfig.cmake:66 (include)
  /home/ognjen/zephyrproject/zephyr/share/zephyr-package/cmake/ZephyrConfig.cmake:92 (include_boilerplate)
  CMakeLists.txt:7 (find_package)

-- Configuring incomplete, errors occurred!
FATAL ERROR: command exited with status 1: /usr/bin/cmake -DWEST_PYTHON=/usr/bin/python3 -B/home/ognjen/zephyrproject/applications/micRobbit-firmware/app/modules/micro_ros_zephyr_module/build -GNinja -DBOARD=esp32_devkitc_wroom -S/home/ognjen/zephyrproject/applications/micRobbit-firmware/app/modules/micro_ros_zephyr_module

I found in this issue statement that this module should be usable on any board supported by Zephyr.

https://github.com/micro-ROS/micro_ros_zephyr_module/issues/60#issuecomment-1141161320

Thanks in advance.

[pablogs9]

Does it works if you revert this changes: https://github.com/micro-ROS/micro_ros_zephyr_module/pull/125 ?

[OgnjenX]

Thanks for the fast response @pablogs9. If I revert the whole commit, I have problems with undefined references to ring buffer related functions... But if I just remove the 'app.overlay' build works. I managed to build my project with the microROS library by just linking 'libmicroros' (https://github.com/micro-ROS/micro_ros_zephyr_module/tree/iron/modules/libmicroros), so I'm not sure if that is the way to go? (and if yes, then building 'micro_ros_zephyr_module' as a whole is probably less relevant?)

[pablogs9]

This repo is an example of a micro-ROS application for disco_l475_iot1, in that sense, you will need to tune the main folder to fit your platform and "reuse" the module (https://github.com/micro-ROS/micro_ros_zephyr_module/tree/iron/modules/libmicroros).

Let me know if you have any questions.

[OgnjenX]

I linked libmicroros as it is done in the example in this repo (basically I just copied 'libmicroros' module and put it in the 'modules' subdirectory for now). The strange thing is that when I enable CPP support (even with minimal parts of the std library, I have dram0_0_seg overflows (it is also happening if I try to compile for other boards, RAM and flash are being overflowed...). I wonder if that is a known limitation, or if I have something misconfigured. Thank you in advance.

[pablogs9]

I do not know how big is your flash. CPP std lib tends to be big and micro-ROS is also a big library (depending on the types and features you use). Normally is a good idea to build with -Os and with -ffuntions -fsections to allow the linker to optimize out not used code sections.

[OgnjenX]

It has a 4MiB flash size. I set the CONFIG_SIZE_OPTIMIZATIONS flag to 'y' (which should result in setting -the Os flag to the compiler) but did not resolve the issue. I suspect the flash size of the chip is a problem because I used the same chip with esp-idf and https://github.com/micro-ROS/micro_ros_espidf_component (Zephyr and FreeRTOS probably have different footprint sizes, but I guess it should not be that much in difference). I'm not sure how to provide -ffuntions -fsections flags to compiler in Zephyr's build system (I tried 'zephyr_library_compile_options' and 'zephyr_compile_options' but it reports that those flags are not recognized). Thanks.

[towen]

I managed to build for the esp32s3_devkitm board. The main thing I needed to do, was to move app.overlay to boards/disco_l475_iot1.overlay (it contains stuff for enabling USB console on ST devices, so it is specific to that board) However, in order to enable UDP transport over WiFi, I did have to modify the library code slightly. In modules/libmicroros/microros_transports/udp/microros_transports.h, I had to do

-#include <posix/sys/socket.h>
-#include <posix/poll.h>
+#include <zephyr/posix/sys/socket.h>
+#include <zephyr/posix/poll.h>

[towen]

Unfortunately though, rclc_support_init causes a fatal exception. I have added a printk at the beginning of main to check the code is running, and unless I return out of main before the line RCCHECK(rclc_support_init(&support, 0, NULL, &allocator)); then I get the following I'm not sure how to debug further on esp32.

ESP-ROM:esp32s3-20210327
Build:Mar 27 2021
rst:0x1 (POWERON),boot:0x8 (SPI_FAST_FLASH_BOOT)
SPIWP:0xee
mode:DIO, clock div:2
load:0x3fcd0108,len:0x164c
load:0x403b6000,len:0xc68
load:0x403ba000,len:0x30e8
SHA-256 comparison failed:
Calculated: 25f55e651297fc7d396b76b2757e3b0496cfbbb37d61e1de9c8819c17d98b5a6
Expected: ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
Attempting to boot anyway...
entry 0x403b61f4
I (48) boot: ESP-IDF eb7e415dc1 2nd stage bootloader
I (48) boot: compile time 16:20:25
I (48) boot: chip revision: 0
I (50) boot.esp32s3: Boot SPI Speed : 40MHz
I (55) boot.esp32s3: SPI Mode       : DIO
I (60) boot.esp32s3: SPI Flash Size : 16MB
I (65) boot: Enabling RNG early entropy source...
I (70) boot: Partition Table:
I (74) boot: ## Label            Usage          Type ST Offset   Length
I (81) boot:  0 nvs              WiFi data        01 02 00009000 00006000
I (88) boot:  1 phy_init         RF data          01 01 0000f000 00001000
I (96) boot:  2 factory          factory app      00 00 00010000 00100000
I (103) boot: End of partition table
I (108) esp_image: segment 0: paddr=00010020 vaddr=00000020 size=0001ch (    28) 
I (116) esp_image: segment 1: paddr=00010044 vaddr=3fca07a8 size=0133ch (  4924) load
I (126) esp_image: segment 2: paddr=00011388 vaddr=3fca1ae4 size=0024ch (   588) load
I (133) esp_image: segment 3: paddr=000115dc vaddr=40374000 size=0e248h ( 57928) load
I (158) esp_image: segment 4: paddr=0001f82c vaddr=00000000 size=0080ch (  2060) 
I (159) esp_image: segment 5: paddr=00020040 vaddr=3c000040 size=03f5ch ( 16220) map
I (168) esp_image: segment 6: paddr=00023fa4 vaddr=00000000 size=0c054h ( 49236) 
I (184) esp_image: segment 7: paddr=00030000 vaddr=42030000 size=10fe8h ( 69608) map
I (207) boot: Loaded app from partition at offset 0x10000
I (208) boot: Disabling RNG early entropy source...

*** Booting Zephyr OS build zephyr-v3.5.0-2657-gbf652b20a9de ***
Starting ROS node ... 
[00:00:02.245,000] <err> os:  ** FATAL EXCEPTION
[00:00:02.245,000] <err> os:  ** CPU 0 EXCCAUSE 28 (load prohibited)
[00:00:02.245,000] <err> os:  **  PC 0x40050c30 VADDR (nil)
[00:00:02.245,000] <err> os:  **  PS 0x60220
[00:00:02.245,000] <err> os:  **    (INTLEVEL:0 EXCM: 0 UM:1 RING:0 WOE:1 OWB:2 CALLINC:2)
[00:00:02.245,000] <err> os:  **  A0 0x800552aa  SP 0x3fc9ef50  A2 (nil)  A3 0x3b9aca00
[00:00:02.245,000] <err> os:  **  A4 0x77359400  A5 (nil)  A6 (nil)  A7 (nil)
[00:00:02.245,000] <err> os:  **  A8 (nil)  A9 0x3fc9ef10 A10 0xea20860 A11 (nil)
[00:00:02.245,000] <err> os:  ** A12 0x9c40 A13 0x5edc A14 0x3fca0ddc A15 0xff000000
[00:00:02.245,000] <err> os:  ** LBEG 0x40056f5c LEND 0x40056f72 LCOUNT (nil)
[00:00:02.245,000] <err> os:  ** SAR 0x12
[00:00:02.245,000] <err> os: >>> ZEPHYR FATAL ERROR 0: CPU exception on CPU 0
[00:00:02.245,000] <err> os: Current thread: 0x3fc964f8 (main)
[00:00:02.363,000] <err> os: Halting system

[OgnjenX]

I have similar issue during runtime:

*** Booting Zephyr OS build zephyr-v3.5.0-2350-gf9c4ae6cf6da ***
[00:00:01.579,000] <err> os:  ** FATAL EXCEPTION
[00:00:01.579,000] <err> os:  ** CPU 0 EXCCAUSE 28 (load prohibited)
[00:00:01.579,000] <err> os:  **  PC 0x4000be94 VADDR (nil)
[00:00:01.579,000] <err> os:  **  PS 0x60520
[00:00:01.579,000] <err> os:  **    (INTLEVEL:0 EXCM: 0 UM:1 RING:0 WOE:1 OWB:5 CALLINC:2)
[00:00:01.579,000] <err> os:  **  A0 0x8000100a  SP 0x3ffec130  A2 (nil)  A3 0x3b9aca00
[00:00:01.579,000] <err> os:  **  A4 0x3b9aca00  A5 (nil)  A6 (nil)  A7 (nil)
[00:00:01.579,000] <err> os:  **  A8 (nil)  A9 0x3ffec0f0 A10 0x228bfe80 A11 (nil)
[00:00:01.579,000] <err> os:  ** A12 (nil) A13 (nil) A14 0x3ffb07c8 A15 (nil)
[00:00:01.579,000] <err> os:  ** LBEG 0x4000c2e0 LEND 0x4000c2f6 LCOUNT (nil)
[00:00:01.579,000] <err> os:  ** SAR 0x12

Backtrace:0x4000be91:0x3ffec130 0x40001007:0x3ffec150 0x400dfb80:0x3ffec170 0x400dad44:0x3ffec1b0 0x400de6cd:0x3ffec270 0x400d07f1:0x3ffec2a0 0x40081873:0x3ffec3b0 0x400d1197:0x3ffec3d0 

0x400dfb80: rmw_init_options_init at /home/ognjen/zephyrproject/applications/micro_ros_zephyr_module/modules/libmicroros/micro_ros_src/src/rmw-microxrcedds/rmw_microxrcedds_c/src/rmw_init.c:112

0x400dad44: rcl_init_options_init at /home/ognjen/zephyrproject/applications/micro_ros_zephyr_module/modules/libmicroros/micro_ros_src/src/rcl/rcl/src/rcl/init_options.c:72

0x400de6cd: rclc_support_init at /home/ognjen/zephyrproject/applications/micro_ros_zephyr_module/modules/libmicroros/micro_ros_src/src/rclc/rclc/src/rclc/init.c:39 (discriminator 2)

0x400d07f1: main at /home/ognjen/zephyrproject/applications/micro_ros_zephyr_module/src/main.c:57

0x40081873: bg_thread_main at /home/ognjen/zephyrproject/zephyr/kernel/init.c:371

0x400d1197: z_thread_entry at /home/ognjen/zephyrproject/zephyr/lib/os/thread_entry.c:48

[00:00:01.580,000] <err> os: >>> ZEPHYR FATAL ERROR 0: CPU exception on CPU 0
[00:00:01.580,000] <err> os: Current thread: 0x3ffb66e0 (unknown)
[00:00:01.683,000] <err> os: Halting system

That is happening in my app (using UDP as microros transport), as well as in your example.

[OgnjenX]

@towen I believe we are experiencing the same issue... For some reason it breaks when rcl_support_init is called (concretely it breaks on line srand(uxr_nanos()); - I wondered if newlibc in espressif toolchain is missing). Any help would be appreciated. Thanks.

[towen]

Interesting. I have occasionally had "multiple definition of 'random()'" errors, but been unable to pin them down. Is CONFIG_TEST_RANDOM_GENERATOR set? Also of note, if CONFIG_MICROROS=y then NET_L2_WIFI_SHELL crashes on any command e.g. wifi scan or wifi status but no idea atm if this is related.

[OgnjenX]

CONFIG_TEST_RANDOM_GENERATOR has not resolved the issue. I also tried adding full libc support (CONFIG_REQUIRES_FULL_LIBC) but also did not resolve the issue.

[towen]

This seems like a Zephyr bug (if I try to use rand/srand in samples/hello_world it still crashes) - perhaps it is to do with recent work on the random subsystem -maybe they neglected the POSIX API when they did this work. I will try to see if it is a regression or whether it has never worked, before I file the issue with Zephyr

However, I think if Zephyr is using a proper random number generator (e.g. from hardware) then calling srand() makes no sense.

We could perhaps remove all calls to srand and replace rand() with sys_rand32_get() which does work, and should produce better random numbers.

There's also sys_rand_get() which can produce faster random numbers for large blocks of memory, so we can get rid of a loop.

[OgnjenX]

I tried replacing srand and rand, but there is an issue with all libc functions. I'm just wondering if that is happening on other targets or if that is a problem specific to espressif's toolchain in Zephyr...

[OgnjenX]

Replacing POSIX functions with Zephyr specifics for random number generation would require modifying 'rmw-microxrcedds' source code (which, I think, is not an ideal option - I think it is using POSIX APIs for easier portability on different platforms). Assuming that the project example contained in this repo works on the 'disco_l475_iot' board (which I cannot check because I do not have one), that would mean that there is a problem with espressif toolchain in Zephyr (it could probably mean something else, but I do not have better explanation currently in mind).

[sylvioalves]

@OgnjenX, while I don't have a proper fix for this issue yet, please, would you try commenting this line?
https://github.com/zephyrproject-rtos/hal_espressif/blob/zephyr/zephyr/esp32/CMakeLists.txt#L77

Let me know if it solves it for now.

[OgnjenX]

@sylvioalves That solves the issue with newlib functions used in the app, so I guess it is a reasonable workaround until the issue is fixed. Thank you very much!

[OgnjenX]

There is still the issue with rclc_support_init, it always returns RMW_RET_ERROR (in my application, as well as in a sample app from this repo) @pablogs9

Any help would be appreciated.