esp32: make bootloader is not working for building from the source code

acassis commented 6 months ago

While trying to build the bootloader from source code I'm getting this error:

-- Building ESP-IDF components for target esp32
CMake Error at /nuttxspace/nuttx/arch/xtensa/src/esp32/bootloader/esp-nuttx-bootloader/esp-idf/tools/cmake/build.cmake:542 (message):
  /usr/bin/python: Error while finding module specification for
  'idf_component_manager.prepare_components' (ModuleNotFoundError: No module
  named 'idf_component_manager')

These are the steps

Configure the board:

$ ./tools/configure.sh esp32-devkitc:nsh

Enable CONFIG_ESP32_BOOTLOADER_BUILD_FROM_SOURCE :

$ make menuconfig

System Type  --->
    Bootloader and Image Configuration  --->
    [*] Build binaries from source

Then try to build the bootloader:

$ make bootloader
Cloning into '/nuttxspace/nuttx/arch/xtensa/src/chip/bootloader/esp-nuttx-bootloader'...
remote: Enumerating objects: 210, done.
remote: Counting objects: 100% (132/132), done.
remote: Compressing objects: 100% (66/66), done.
remote: Total 210 (delta 63), reused 76 (delta 46), pack-reused 78
Receiving objects: 100% (210/210), 32.23 KiB | 6.45 MiB/s, done.
Resolving deltas: 100% (111/111), done.
Submodule 'esp-idf' (https://github.com/espressif/esp-idf.git) registered for path 'esp-idf'
Cloning into '/nuttxspace/nuttx/arch/xtensa/src/esp32/bootloader/esp-nuttx-bootloader/esp-idf'...
Submodule path 'esp-idf': checked out '1559b6309fe207add518828246a05f80582ee59d'
Creating Bootloader configuration
Building Bootloader binaries
CMake Warning:
  Ignoring empty string ("") provided on the command line.

-- Found Git: /usr/bin/git (found version "2.34.1") 
-- Initialising new submodule components/bootloader/subproject/components/micro-ecc/micro-ecc...
Submodule 'components/bootloader/subproject/components/micro-ecc/micro-ecc' (https://github.com/kmackay/micro-ecc.git) registered for path 'components/bootloader/subproject/components/micro-ecc/micro-ecc'
Cloning into '/nuttxspace/nuttx/arch/xtensa/src/esp32/bootloader/esp-nuttx-bootloader/esp-idf/components/bootloader/subproject/components/micro-ecc/micro-ecc'...
Submodule path 'components/bootloader/subproject/components/micro-ecc/micro-ecc': checked out 'd037ec89546fad14b5c4d5456c2e23a71e554966'
-- Initialising new submodule components/bt/controller/lib_esp32...
Submodule 'components/bt/controller/lib_esp32' (https://github.com/espressif/esp32-bt-lib.git) registered for path 'components/bt/controller/lib_esp32'
Cloning into '/nuttxspace/nuttx/arch/xtensa/src/esp32/bootloader/esp-nuttx-bootloader/esp-idf/components/bt/controller/lib_esp32'...
Submodule path 'components/bt/controller/lib_esp32': checked out 'b867411caf88b485b2fa8b6d272692b6535a267a'
-- Initialising new submodule components/bt/controller/lib_esp32c2/esp32c2-bt-lib...
Submodule 'components/bt/controller/lib_esp32c2/esp32c2-bt-lib' (https://github.com/espressif/esp32c2-bt-lib.git) registered for path 'components/bt/controller/lib_esp32c2/esp32c2-bt-lib'
Cloning into '/nuttxspace/nuttx/arch/xtensa/src/esp32/bootloader/esp-nuttx-bootloader/esp-idf/components/bt/controller/lib_esp32c2/esp32c2-bt-lib'...
Submodule path 'components/bt/controller/lib_esp32c2/esp32c2-bt-lib': checked out '10a3ea96e083dd0a7a18089407361fd52f0cbc47'
-- Initialising new submodule components/bt/controller/lib_esp32c3_family...
Submodule 'components/bt/controller/lib_esp32c3_family' (https://github.com/espressif/esp32c3-bt-lib.git) registered for path 'components/bt/controller/lib_esp32c3_family'
Cloning into '/nuttxspace/nuttx/arch/xtensa/src/esp32/bootloader/esp-nuttx-bootloader/esp-idf/components/bt/controller/lib_esp32c3_family'...
Submodule path 'components/bt/controller/lib_esp32c3_family': checked out 'bba9af9259e0999ef246426d31a793fe0a3ff4db'
-- Initialising new submodule components/bt/controller/lib_esp32c6/esp32c6-bt-lib...
Submodule 'components/bt/controller/lib_esp32c6/esp32c6-bt-lib' (https://github.com/espressif/esp32c6-bt-lib.git) registered for path 'components/bt/controller/lib_esp32c6/esp32c6-bt-lib'
Cloning into '/nuttxspace/nuttx/arch/xtensa/src/esp32/bootloader/esp-nuttx-bootloader/esp-idf/components/bt/controller/lib_esp32c6/esp32c6-bt-lib'...
Submodule path 'components/bt/controller/lib_esp32c6/esp32c6-bt-lib': checked out '658a412455d652be3b07a602a988b25d97b00e03'
-- Initialising new submodule components/bt/controller/lib_esp32h2/esp32h2-bt-lib...
Submodule 'components/bt/controller/lib_esp32h2/esp32h2-bt-lib' (https://github.com/espressif/esp32h2-bt-lib.git) registered for path 'components/bt/controller/lib_esp32h2/esp32h2-bt-lib'
Cloning into '/nuttxspace/nuttx/arch/xtensa/src/esp32/bootloader/esp-nuttx-bootloader/esp-idf/components/bt/controller/lib_esp32h2/esp32h2-bt-lib'...
Submodule path 'components/bt/controller/lib_esp32h2/esp32h2-bt-lib': checked out '048df1c5932ee225cf93ae2def8946cee7ab6399'
-- Initialising new submodule components/bt/host/nimble/nimble...
Submodule 'components/bt/host/nimble/nimble' (https://github.com/espressif/esp-nimble.git) registered for path 'components/bt/host/nimble/nimble'
Cloning into '/nuttxspace/nuttx/arch/xtensa/src/esp32/bootloader/esp-nuttx-bootloader/esp-idf/components/bt/host/nimble/nimble'...
Submodule path 'components/bt/host/nimble/nimble': checked out '2c3ca76948c71c32fce2a25698c58323d83b3c19'
-- Initialising new submodule components/cmock/CMock...
Submodule 'components/cmock/CMock' (https://github.com/ThrowTheSwitch/CMock.git) registered for path 'components/cmock/CMock'
Cloning into '/nuttxspace/nuttx/arch/xtensa/src/esp32/bootloader/esp-nuttx-bootloader/esp-idf/components/cmock/CMock'...
Submodule path 'components/cmock/CMock': checked out 'eeecc49ce8af123cf8ad40efdb9673e37b56230f'
Submodule 'vendor/c_exception' (https://github.com/throwtheswitch/cexception.git) registered for path 'components/cmock/CMock/vendor/c_exception'
Submodule 'vendor/unity' (https://github.com/throwtheswitch/unity.git) registered for path 'components/cmock/CMock/vendor/unity'
Cloning into '/nuttxspace/nuttx/arch/xtensa/src/esp32/bootloader/esp-nuttx-bootloader/esp-idf/components/cmock/CMock/vendor/c_exception'...
Cloning into '/nuttxspace/nuttx/arch/xtensa/src/esp32/bootloader/esp-nuttx-bootloader/esp-idf/components/cmock/CMock/vendor/unity'...
Submodule path 'components/cmock/CMock/vendor/c_exception': checked out '71b47be7c950f1bf5f7e5303779fa99a16224bb6'
Submodule path 'components/cmock/CMock/vendor/unity': checked out 'cf949f45ca6d172a177b00da21310607b97bc7a7'
-- Initialising new submodule components/esp_coex/lib...
Submodule 'components/esp_coex/lib' (https://github.com/espressif/esp-coex-lib.git) registered for path 'components/esp_coex/lib'
Cloning into '/nuttxspace/nuttx/arch/xtensa/src/esp32/bootloader/esp-nuttx-bootloader/esp-idf/components/esp_coex/lib'...
Submodule path 'components/esp_coex/lib': checked out 'cc8eae05caf9b67ddbde40b057e32748c197b95b'
-- Initialising new submodule components/esp_phy/lib...
Submodule 'components/esp_phy/lib' (https://github.com/espressif/esp-phy-lib.git) registered for path 'components/esp_phy/lib'
Cloning into '/nuttxspace/nuttx/arch/xtensa/src/esp32/bootloader/esp-nuttx-bootloader/esp-idf/components/esp_phy/lib'...
Submodule path 'components/esp_phy/lib': checked out '1b8e12d3e0e8b7bcd87c115f09ec0f385700579a'
-- Initialising new submodule components/esp_wifi/lib...
Submodule 'components/esp_wifi/lib' (https://github.com/espressif/esp32-wifi-lib.git) registered for path 'components/esp_wifi/lib'
Cloning into '/nuttxspace/nuttx/arch/xtensa/src/esp32/bootloader/esp-nuttx-bootloader/esp-idf/components/esp_wifi/lib'...
Submodule path 'components/esp_wifi/lib': checked out 'a821d6aef8e9ba26b1003a0fa3276774a2113433'
-- Initialising new submodule components/heap/tlsf...
Submodule 'components/heap/tlsf' (https://github.com/espressif/tlsf.git) registered for path 'components/heap/tlsf'
Cloning into '/nuttxspace/nuttx/arch/xtensa/src/esp32/bootloader/esp-nuttx-bootloader/esp-idf/components/heap/tlsf'...
Submodule path 'components/heap/tlsf': checked out '8c9cd0517adf99e363812e9a295dfe3898fdd345'
-- Initialising new submodule components/ieee802154/lib...
Submodule 'components/ieee802154/lib' (https://github.com/espressif/esp-ieee802154-lib.git) registered for path 'components/ieee802154/lib'
Cloning into '/nuttxspace/nuttx/arch/xtensa/src/esp32/bootloader/esp-nuttx-bootloader/esp-idf/components/ieee802154/lib'...
Submodule path 'components/ieee802154/lib': checked out '56f4a209e8b88e32505172b7632380b8f260e789'
-- Initialising new submodule components/json/cJSON...
Submodule 'components/json/cJSON' (https://github.com/DaveGamble/cJSON.git) registered for path 'components/json/cJSON'
Cloning into '/nuttxspace/nuttx/arch/xtensa/src/esp32/bootloader/esp-nuttx-bootloader/esp-idf/components/json/cJSON'...
Submodule path 'components/json/cJSON': checked out 'd348621ca93571343a56862df7de4ff3bc9b5667'
-- Initialising new submodule components/lwip/lwip...
Submodule 'components/lwip/lwip' (https://github.com/espressif/esp-lwip.git) registered for path 'components/lwip/lwip'
Cloning into '/nuttxspace/nuttx/arch/xtensa/src/esp32/bootloader/esp-nuttx-bootloader/esp-idf/components/lwip/lwip'...
Submodule path 'components/lwip/lwip': checked out '10197b212a95c49c733fb18ffed56cafb0d196d4'
-- Initialising new submodule components/mbedtls/mbedtls...
Submodule 'components/mbedtls/mbedtls' (https://github.com/espressif/mbedtls.git) registered for path 'components/mbedtls/mbedtls'
Cloning into '/nuttxspace/nuttx/arch/xtensa/src/esp32/bootloader/esp-nuttx-bootloader/esp-idf/components/mbedtls/mbedtls'...
Submodule path 'components/mbedtls/mbedtls': checked out 'e39975969d548572a39875ad29abd88b23285f94'
-- Initialising new submodule components/mqtt/esp-mqtt...
Submodule 'components/mqtt/esp-mqtt' (https://github.com/espressif/esp-mqtt.git) registered for path 'components/mqtt/esp-mqtt'
Cloning into '/nuttxspace/nuttx/arch/xtensa/src/esp32/bootloader/esp-nuttx-bootloader/esp-idf/components/mqtt/esp-mqtt'...
Submodule path 'components/mqtt/esp-mqtt': checked out 'dffabb067fb3c39f486033d2e47eb4b1416f0c82'
-- Initialising new submodule components/openthread/lib...
Submodule 'components/openthread/lib' (https://github.com/espressif/esp-thread-lib.git) registered for path 'components/openthread/lib'
Cloning into '/nuttxspace/nuttx/arch/xtensa/src/esp32/bootloader/esp-nuttx-bootloader/esp-idf/components/openthread/lib'...
Submodule path 'components/openthread/lib': checked out '2e5a3f236f8cd7c753fd8b8f775e48e8be5f08d6'
-- Initialising new submodule components/openthread/openthread...
Submodule 'components/openthread/openthread' (https://github.com/espressif/openthread.git) registered for path 'components/openthread/openthread'
Cloning into '/nuttxspace/nuttx/arch/xtensa/src/esp32/bootloader/esp-nuttx-bootloader/esp-idf/components/openthread/openthread'...
Submodule path 'components/openthread/openthread': checked out '091f68ed706ce7a4831802408cdd0b0b4f309e3b'
-- Initialising new submodule components/protobuf-c/protobuf-c...
Submodule 'components/protobuf-c/protobuf-c' (https://github.com/protobuf-c/protobuf-c.git) registered for path 'components/protobuf-c/protobuf-c'
Cloning into '/nuttxspace/nuttx/arch/xtensa/src/esp32/bootloader/esp-nuttx-bootloader/esp-idf/components/protobuf-c/protobuf-c'...
Submodule path 'components/protobuf-c/protobuf-c': checked out 'abc67a11c6db271bedbb9f58be85d6f4e2ea8389'
-- Initialising new submodule components/spiffs/spiffs...
Submodule 'components/spiffs/spiffs' (https://github.com/pellepl/spiffs.git) registered for path 'components/spiffs/spiffs'
Cloning into '/nuttxspace/nuttx/arch/xtensa/src/esp32/bootloader/esp-nuttx-bootloader/esp-idf/components/spiffs/spiffs'...
Submodule path 'components/spiffs/spiffs': checked out '0dbb3f71c5f6fae3747a9d935372773762baf852'
-- Initialising new submodule components/unity/unity...
Submodule 'components/unity/unity' (https://github.com/ThrowTheSwitch/Unity.git) registered for path 'components/unity/unity'
Cloning into '/nuttxspace/nuttx/arch/xtensa/src/esp32/bootloader/esp-nuttx-bootloader/esp-idf/components/unity/unity'...
Submodule path 'components/unity/unity': checked out '7d2bf62b7e6afaf38153041a9d53c21aeeca9a25'
-- The C compiler identification is GNU 12.2.0
-- The CXX compiler identification is GNU 12.2.0
-- The ASM compiler identification is GNU
-- Found assembler: /usr/local/bin/xtensa-esp32-elf-gcc
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/local/bin/xtensa-esp32-elf-gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/local/bin/xtensa-esp32-elf-g++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Building ESP-IDF components for target esp32
CMake Error at /nuttxspace/nuttx/arch/xtensa/src/esp32/bootloader/esp-nuttx-bootloader/esp-idf/tools/cmake/build.cmake:542 (message):
  /usr/bin/python: Error while finding module specification for
  'idf_component_manager.prepare_components' (ModuleNotFoundError: No module
  named 'idf_component_manager')

Call Stack (most recent call first):
  /nuttxspace/nuttx/arch/xtensa/src/esp32/bootloader/esp-nuttx-bootloader/esp-idf/tools/cmake/project.cmake:453 (idf_build_process)
  CMakeLists.txt:58 (project)

-- Configuring incomplete, errors occurred!
See also "/nuttxspace/nuttx/arch/xtensa/src/esp32/bootloader/esp-nuttx-bootloader/.build-esp32/CMakeFiles/CMakeOutput.log".
make[1]: *** [chip/Bootloader.mk:178: bootloader] Error 1
make: *** [tools/Unix.mk:600: bootloader] Error 2

acassis commented 6 months ago

@tmedicci any idea?

tmedicci commented 6 months ago

Hi @acassis !

I tested it locally (and our internal CI also tests it daily). I could not find any problems. I suspect this is somehow related to the outdated kconfig-frontends, but I was not even able to build it locally (it requires very old libraries). Can you check with kconfiglib?

acassis commented 6 months ago

Thank you @tmedicci actually this issue was reported by @casaroli and I just tried to reproduced the issue. I think he is used kconfiglib, but if this is some issue caused by Kconfig, it should be modified to work with kconfig-frontends and kconfiglib.

tmedicci commented 5 months ago

@acassis and @casaroli , the bootloader built from the sources uses a wrapper around an IDF-provided repository (the esp-nuttx-bootloader). Kconfiglib is a dependency of ESP-IDF, so it must be installed (not by NuttX, but for esp-nuttx-bootloader). This is why we are removing support for the IDF's bootloader.

NuttX adopted kconfiglib later and it uses it by default. This required users that still want to use kconfig-frontends to uninstall kconfiglib (or, equivalently, uninstall for their python environment): this is why this problem has risen.

I'm closing this issue because IDF's bootloader is about to be deprecated. I recommend keeping kconfiglib installed to enable building the bootloader from the sources. There are a list of possible workarounds to be used (if kconfig-frontends is required) until we completely deprecate the use of the IDF's bootloader:

Install kconfiglib in a python environment and activate/deactivate it before/after running make bootloader;
Modify NuttX's build system to enable selecting a preferred tool (instead of using kconfiglib by default, when available).

acassis commented 5 months ago

Thank you very much for this explanation @tmedicci !

Actually kconfiglib is not the default option for NuttX, it is optional: https://nuttx.apache.org/docs/latest/quickstart/install.html

I think both kconfig-frontends and kconfiglib work correctly in the NuttX building system, I wasn't aware of this limitation/constraints of ESP-IDF. Thank you again for above explanation.

tmedicci commented 5 months ago

You're welcome, Alan!

Actually kconfiglib is not the default option for NuttX, it is optional: https://nuttx.apache.org/docs/latest/quickstart/install.html

I meant that, if available, kconfiglib will be used instead of kconfig-frontends: https://github.com/apache/nuttx/blob/cfa544357eb3081e49f54e5fe3eef8392f26d9ab/tools/Unix.mk#L639

apache / nuttx

esp32: make bootloader is not working for building from the source code #12335

These are the steps