Can not override ADF components (AUD-1172)

[chombium]

Hi,

I'm working on an ADF application for a while and now I wanted to update my ADF copy to the latest master (commit fae539c3035b2c041f49c5b01cdc4c99038595b0) which includes ESP-IDF v3.3. After updating my ADF copy I'm experiencing a problem when overriding ADF components. With the latest master I'm not able to build my app. In the output from idf.py build in the Component paths: section I see that the components are taken from the $ADF_PATH/components and not from <PROJECT_PATH>/components.

Development environment:

• Arch Linux
• Xtensa esp32 toolchain v5.2.0 and v8.2.0
• ESP-ADF master hash: fae539c3035b2c041f49c5b01cdc4c99038595b0

Steps to reproduce the problem:

1. git clone --recursive https://github.com/espressif/esp-adf.git

2. setting the envvars:

   export ADF_PATH=/<path-to-esp-adf>
   export IDF_PATH=$ADF_PATH/esp-idf
   export PATH=$IDF_PATH/tools:$PATH
   export PATH=/<path-to-xtensa-esp32-elf-5.2.0>/bin:$PATH

3. Prepare a demo project: copied IDF hello-world and ADF play_mp3 examples to a new folder and initialized it as a git repo. The demo project can be found here.

   $IDF_PATH/examples/getting-started/hello-world
   $ADF_PATH/examples/getting-started/play_mp3

4. build the apps with idf.py build to check if everything works

5. add dummy components which override the standard components: wifi_service for ADF and wifi_provisioning for IDF and I've tried to rebuild the projects:

The expected result was that both the projects build successfully and the ADF/IDF components are overridden by the components in the project.

The actual result is:

• the IDF hello_world project was build with the IDF wifi_provisioning component overridden with the one from the project;
• for the ADF play_mp3 project, the wifi_service component was not overriden with the one from the project. I also checked Component paths: list from the idf.py build output and I saw that the path was $ADF_PATH/components/wifi_service.

6. I continued testing with older commits to find which commit breaks the component overrides and I found out that problem starts with the commit with hash e9c4faff248cd300207ad938a93d67d880484986.

commit e9c4faff248cd300207ad938a93d67d880484986
Author: maojianxin <maojianxin@espressif.com>
Date:   Fri Dec 13 13:52:43 2019 +0800

    Change COMPONENT_NAME to COMPONENT_TARGET for support IDF v3.3.1

Can someone please help me to solve the problem?

Best Regards, Jovan

[jason-mao]

@chombium Thanks for your report.

[chombium]

I've done few tests with the commits in the repo and it seems that the problematic commit was a force push and messed something.

I did one more test: I've checked out the previous commit, created a diff, applied the changes and tried to build my app. The build was successful and everything workd as expected.

in my local esp-adf repo:

$ git checkout 30403e26485e00c42bb3d8a96064ebcb0090edb8
$ git diff --full-index --binary 30403e26485e00c42bb3d8a96064ebcb0090edb8 e9c4faff248cd300207ad938a93d67d880484986 | git apply

than in my project repo I ran:

idf.py build

and everything worked as expected.

It seems that there are no problems with the changes introduced with the commit e9c4faff248cd300207ad938a93d67d880484986.

Now I'm searching for a clean way to revert this commit, apply the changes introduced by it and add the rest of the commits up to the master HEAD on top of it.

I'm opened to ideas on how to do this.

Best Regards, Jovan

[embelgit]

path-to-xtensa-esp32-elf-5.2.0>/bin:$PATH where do i find this??

[embelgit]

now i am not able to build the project I have some problems "Warn about uninitialized values. CMake Warning (dev) at CMakeLists.txt:5 (include): uninitialized variable 'ADF_PATH' This warning is for project developers. Use -Wno-dev to suppress it.

CMake Error at CMakeLists.txt:5 (include): include could not find load file: /CMakeLists.txt

-- Found Git: C:/Program Files/Git/bin/git.exe (found version "2.25.1.windows.1") -- Unexpected file in components directory: C:/Users/Embedded/Desktop/esp/esp-idf/components/.gitkeep -- ccache will be used for faster recompilation -- The C compiler identification is GNU 8.2.0 -- The CXX compiler identification is GNU 8.2.0 -- The ASM compiler identification is GNU -- Found assembler: C:/Users/Embedded/.espressif/tools/xtensa-esp32-elf/esp-2019r2-8.2.0/xtensa-esp32-elf/bin/xtensa-esp32-elf-gcc.exe -- Check for working C compiler: C:/Users/Embedded/.espressif/tools/xtensa-esp32-elf/esp-2019r2-8.2.0/xtensa-esp32-elf/bin/xtensa-esp32-elf-gcc.exe -- Check for working C compiler: C:/Users/Embedded/.espressif/tools/xtensa-esp32-elf/esp-2019r2-8.2.0/xtensa-esp32-elf/bin/xtensa-esp32-elf-gcc.exe -- works -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Detecting C compile features -- Detecting C compile features - done -- Check for working CXX compiler: C:/Users/Embedded/.espressif/tools/xtensa-esp32-elf/esp-2019r2-8.2.0/xtensa-esp32-elf/bin/xtensa-esp32-elf-g++.exe -- Check for working CXX compiler: C:/Users/Embedded/.espressif/tools/xtensa-esp32-elf/esp-2019r2-8.2.0/xtensa-esp32-elf/bin/xtensa-esp32-elf-g++.exe -- works -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Detecting CXX compile features -- Detecting CXX compile features - done -- Building ESP-IDF components for target esp32 warning: the default selection SR_RUN_WM6_CORE1 (undefined) of (defined at C:/Users/Embedded/Desktop/esp/esp-idf/components/esp-sr/Kconfig.projbuild:83) is not contained in the choice warning: the default selection CONFIG_SR_MN1_MODEL_QUANT (undefined) of (defined at C:/Users/Embedded/Desktop/esp/esp-idf/components/esp-sr/Kconfig.projbuild:100) is not contained in the choice -- App "play_mp3" version: v2.0-beta2-21-g30403e2-dirty -- Found PythonInterp: C:/Users/Embedded/.espressif/python_env/idf4.2_py2.7_env/Scripts/python.exe (found version "2.7.12") -- Could NOT find Perl (missing: PERL_EXECUTABLE) -- Adding linker script C:/Users/Embedded/Desktop/esp/esp-adf/examples/get-started/play_mp3/build/esp-idf/esp32/esp32_out.ld -- Adding linker script C:/Users/Embedded/Desktop/esp/esp-idf/components/esp32/ld/esp32.project.ld.in -- Adding linker script C:/Users/Embedded/Desktop/esp/esp-idf/components/esp32/ld/esp32.peripherals.ld -- Adding linker script C:/Users/Embedded/Desktop/esp/esp-idf/components/esp_rom/esp32/ld/esp32.rom.newlib-funcs-time.ld -- Adding linker script C:/Users/Embedded/Desktop/esp/esp-idf/components/esp_rom/esp32/ld/esp32.rom.ld -- Adding linker script C:/Users/Embedded/Desktop/esp/esp-idf/components/esp_rom/esp32/ld/esp32.rom.libgcc.ld -- Adding linker script C:/Users/Embedded/Desktop/esp/esp-idf/components/esp_rom/esp32/ld/esp32.rom.newlib-data.ld -- Adding linker script C:/Users/Embedded/Desktop/esp/esp-idf/components/esp_rom/esp32/ld/esp32.rom.syscalls.ld -- Adding linker script C:/Users/Embedded/Desktop/esp/esp-idf/components/esp_rom/esp32/ld/esp32.rom.newlib-funcs.ld -- Current board name is CONFIG_ESP_LYRAT_V4_3_BOARD -- Components: adf_utils app_trace app_update asio audio_board audio_hal audio_pipeline audio_sal audio_stream bluetooth_service bootloader bootloader_support bt cbor clouds coap console cxx display_service driver dueros_service efuse esp-adf-libs esp-sr esp-tls esp32 esp_actions esp_adc_cal esp_common esp_dispatcher esp_eth esp_event esp_gdbstub esp_http_client esp_http_server esp_https_ota esp_https_server esp_local_ctrl esp_netif esp_peripherals esp_ringbuf esp_rom esp_serial_slave_link esp_timer esp_websocket_client esp_wifi espcoredump esptool_py expat fatfs freemodbus freertos heap idf_test input_key_service jsmn json libsodium log lwip main mbedtls mdns mqtt newlib nghttp nvs_flash openssl partition_table perfmon playlist protobuf-c protocomm pthread sdmmc soc spi_flash spiffs tcp_transport tcpip_adapter ulp unity vfs wear_levelling wifi_provisioning wifi_service wpa_supplicant xtensa -- Component paths: C:/Users/Embedded/Desktop/esp/esp-idf/components/adf_utils C:/Users/Embedded/Desktop/esp/esp-idf/components/app_trace C:/Users/Embedded/Desktop/esp/esp-idf/components/app_update C:/Users/Embedded/Desktop/esp/esp-idf/components/asio C:/Users/Embedded/Desktop/esp/esp-idf/components/audio_board C:/Users/Embedded/Desktop/esp/esp-idf/components/audio_hal C:/Users/Embedded/Desktop/esp/esp-idf/components/audio_pipeline C:/Users/Embedded/Desktop/esp/esp-idf/components/audio_sal C:/Users/Embedded/Desktop/esp/esp-idf/components/audio_stream C:/Users/Embedded/Desktop/esp/esp-idf/components/bluetooth_service C:/Users/Embedded/Desktop/esp/esp-idf/components/bootloader C:/Users/Embedded/Desktop/esp/esp-idf/components/bootloader_support C:/Users/Embedded/Desktop/esp/esp-idf/components/bt C:/Users/Embedded/Desktop/esp/esp-idf/components/cbor C:/Users/Embedded/Desktop/esp/esp-idf/components/clouds C:/Users/Embedded/Desktop/esp/esp-idf/components/coap C:/Users/Embedded/Desktop/esp/esp-idf/components/console C:/Users/Embedded/Desktop/esp/esp-idf/components/cxx C:/Users/Embedded/Desktop/esp/esp-idf/components/display_service C:/Users/Embedded/Desktop/esp/esp-idf/components/driver C:/Users/Embedded/Desktop/esp/esp-idf/components/dueros_service C:/Users/Embedded/Desktop/esp/esp-idf/components/efuse C:/Users/Embedded/Desktop/esp/esp-idf/components/esp-adf-libs C:/Users/Embedded/Desktop/esp/esp-idf/components/esp-sr C:/Users/Embedded/Desktop/esp/esp-idf/components/esp-tls C:/Users/Embedded/Desktop/esp/esp-idf/components/esp32 C:/Users/Embedded/Desktop/esp/esp-idf/components/esp_actions C:/Users/Embedded/Desktop/esp/esp-idf/components/esp_adc_cal C:/Users/Embedded/Desktop/esp/esp-idf/components/esp_common C:/Users/Embedded/Desktop/esp/esp-idf/components/esp_dispatcher C:/Users/Embedded/Desktop/esp/esp-idf/components/esp_eth C:/Users/Embedded/Desktop/esp/esp-idf/components/esp_event C:/Users/Embedded/Desktop/esp/esp-idf/components/esp_gdbstub C:/Users/Embedded/Desktop/esp/esp-idf/components/esp_http_client C:/Users/Embedded/Desktop/esp/esp-idf/components/esp_http_server C:/Users/Embedded/Desktop/esp/esp-idf/components/esp_https_ota C:/Users/Embedded/Desktop/esp/esp-idf/components/esp_https_server C:/Users/Embedded/Desktop/esp/esp-idf/components/esp_local_ctrl C:/Users/Embedded/Desktop/esp/esp-idf/components/esp_netif C:/Users/Embedded/Desktop/esp/esp-idf/components/esp_peripherals C:/Users/Embedded/Desktop/esp/esp-idf/components/esp_ringbuf C:/Users/Embedded/Desktop/esp/esp-idf/components/esp_rom C:/Users/Embedded/Desktop/esp/esp-idf/components/esp_serial_slave_link C:/Users/Embedded/Desktop/esp/esp-idf/components/esp_timer C:/Users/Embedded/Desktop/esp/esp-idf/components/esp_websocket_client C:/Users/Embedded/Desktop/esp/esp-idf/components/esp_wifi C:/Users/Embedded/Desktop/esp/esp-idf/components/espcoredump C:/Users/Embedded/Desktop/esp/esp-idf/components/esptool_py C:/Users/Embedded/Desktop/esp/esp-idf/components/expat C:/Users/Embedded/Desktop/esp/esp-idf/components/fatfs C:/Users/Embedded/Desktop/esp/esp-idf/components/freemodbus C:/Users/Embedded/Desktop/esp/esp-idf/components/freertos C:/Users/Embedded/Desktop/esp/esp-idf/components/heap C:/Users/Embedded/Desktop/esp/esp-idf/components/idf_test C:/Users/Embedded/Desktop/esp/esp-idf/components/input_key_service C:/Users/Embedded/Desktop/esp/esp-idf/components/jsmn C:/Users/Embedded/Desktop/esp/esp-idf/components/json C:/Users/Embedded/Desktop/esp/esp-idf/components/libsodium C:/Users/Embedded/Desktop/esp/esp-idf/components/log C:/Users/Embedded/Desktop/esp/esp-idf/components/lwip C:/Users/Embedded/Desktop/esp/esp-adf/examples/get-started/play_mp3/main C:/Users/Embedded/Desktop/esp/esp-idf/components/mbedtls C:/Users/Embedded/Desktop/esp/esp-idf/components/mdns C:/Users/Embedded/Desktop/esp/esp-idf/components/mqtt C:/Users/Embedded/Desktop/esp/esp-idf/components/newlib C:/Users/Embedded/Desktop/esp/esp-idf/components/nghttp C:/Users/Embedded/Desktop/esp/esp-idf/components/nvs_flash C:/Users/Embedded/Desktop/esp/esp-idf/components/openssl C:/Users/Embedded/Desktop/esp/esp-idf/components/partition_table C:/Users/Embedded/Desktop/esp/esp-idf/components/perfmon C:/Users/Embedded/Desktop/esp/esp-idf/components/playlist C:/Users/Embedded/Desktop/esp/esp-idf/components/protobuf-c C:/Users/Embedded/Desktop/esp/esp-idf/components/protocomm C:/Users/Embedded/Desktop/esp/esp-idf/components/pthread C:/Users/Embedded/Desktop/esp/esp-idf/components/sdmmc C:/Users/Embedded/Desktop/esp/esp-idf/components/soc C:/Users/Embedded/Desktop/esp/esp-idf/components/spi_flash C:/Users/Embedded/Desktop/esp/esp-idf/components/spiffs C:/Users/Embedded/Desktop/esp/esp-idf/components/tcp_transport C:/Users/Embedded/Desktop/esp/esp-idf/components/tcpip_adapter C:/Users/Embedded/Desktop/esp/esp-idf/components/ulp C:/Users/Embedded/Desktop/esp/esp-idf/components/unity C:/Users/Embedded/Desktop/esp/esp-idf/components/vfs C:/Users/Embedded/Desktop/esp/esp-idf/components/wear_levelling C:/Users/Embedded/Desktop/esp/esp-idf/components/wifi_provisioning C:/Users/Embedded/Desktop/esp/esp-idf/components/wifi_service C:/Users/Embedded/Desktop/esp/esp-idf/components/wpa_supplicant C:/Users/Embedded/Desktop/esp/esp-idf/components/xtensa -- Configuring incomplete, errors occurred! See also "C:/Users/Embedded/Desktop/esp/esp-adf/examples/get-started/play_mp3/build/CMakeFiles/CMakeOutput.log". cmake failed with exit code 1 "

[chombium]

Hi @embelgit,

thanks for taking a look into this. Feel free to ping me if you need more help.

Here are some <path-to-xtensa-esp32-elf-5.2.0> is the path to the folder where you've installed the Xtensa toolchain on your machine. You also have to export the ADF_PATH and IDF_PATH variables. ADF_PATH has to point to the path where you've cloned ESP ADF and IDF_PATH to the ESP IDF folder inside the ADF_PATH, $ADF_PATH/esp-idf.

BTW. If you are trying to compile my demo project, you are getting the same error as I do: If you check the component paths the wifi_provisioning component is read from your C:/Users/Embedded/Desktop/esp/esp-idf/components/wifi_provisioning folder, so from the ESP_IDF and not from the components/wifi_provisioning provided in the project

[jason-mao]

@chombium Could you help to test with follow modifications? IDF is v3.3.

[chombium]

@jason-mao sure, I will be glad to help. At the moment I don't have time, but I will test this tonight (I'm in GMT+1 time zone) and I will update you.

[chombium]

@jason-mao: Could you please send me a link to ADF github repo where the file is located?

[jason-mao]

https://github.com/espressif/esp-idf/blob/f3c3605fc2aa3cb729e065893fdfcdd97a9e6db2/tools/cmake/project.cmake#L42, @chombium Here it is.

[chombium]

Hi @jason-mao, hi @embelgit,

I've tested the suggested changes and I can confirm that they fix the problem :)

Here is a detailed description of what I've done:

1. cloned a fresh copy of the ESP ADF master branch (commit hash d4c36b4ba8d20352205ebaab0e8ddf7d99351044)
2. I'm using Arch Linux and my environment is:

   $ echo $ADF_PATH
   /opt/development/sdk/esp/esp-adf
   $ echo $IDF_PATH 
   /opt/development/sdk/esp/esp-adf/esp-idf
   $ echo $PATH
   /opt/development/sdk/esp/tools/tools/xtensa-esp32-elf/xtensa-esp32-elf-5.2.0/bin:/opt/development/sdk/esp/esp-adf/esp-idf/tools: <rest of the PATH>

3. cd to the ADF project_folder:

   cd /opt/development/projects/esp/component-override/play_mp3

4. tried to compile the project without the changes:

   idf.py build

   in the output from the build I saw that the component path was not overridden:

   -- Component names: ...
   -- Component paths: 
   ...
   /opt/development/sdk/esp/esp-adf/components/wifi_service;
   ...

5. I've applied the suggested changes to the $IDF_PATH/tools/cmake/project.cmake file. The diff is:

   
   diff --git a/tools/cmake/project.cmake b/tools/cmake/project.cmake
   index 13478facc..e9d426b75 100644
   --- a/tools/cmake/project.cmake
   +++ b/tools/cmake/project.cmake
   @@ -30,6 +30,12 @@ macro(project name)
        endforeach()
    endif()

• if(NOT MAIN_SRCS)
• list(APPEND IDF_EXTRA_COMPONENT_DIRS "${CMAKE_SOURCE_DIR}/main")
• endif()
• list(APPEND IDF_EXTRA_COMPONENT_DIRS "${CMAKE_SOURCE_DIR}/components")

• if(EXTRA_COMPONENT_DIRS) spaces2list(EXTRA_COMPONENT_DIRS)

@@ -39,12 +45,6 @@ macro(project name) endforeach() endif()

• list(APPEND IDF_EXTRA_COMPONENT_DIRS "${CMAKE_SOURCE_DIR}/components")
• if(NOT MAIN_SRCS)
• list(APPEND IDF_EXTRA_COMPONENT_DIRS "${CMAKE_SOURCE_DIR}/main")
• endif()

• if(COMPONENTS) set(IDF_COMPONENTS "${COMPONENTS}") endif()

  6. compiled my ADF test `play_mp3` project once again and in the output from `idf.py build` I saw that the component path gets properly overridden:

  Component names: ... -- Component paths: ... /opt/development/projects/esp/component-override/play_mp3/components/wifi_service; ...

  7. as a final test to check if the IDF overrides work properly after the changes, I've compiled my test IDF project `hello_world`.
  ```bash
  $ cd /opt/development/projects/esp/component-override/hello_world
  $ idf.py build

  In the output I've seen that the overrides work properly for IDF projects as well:

  -- Component names: ...
  -- Component paths: 
  ...
  /opt/development/projects/esp/component-override/hello_world/components/wifi_provisioning;
  ...

Great job @jason-mao and @embelgit. Thanks for the help. I guess you can merge the changes to the ESP IDF.

Best Regards, Jovan

[jason-mao]

@chombium Thank you. It is going to merge.