Refactor cmake build system not to require idf environment to be set build-time (IDFGH-4459)

[DCNick3]

Is your feature request related to a problem? Please describe.

I came to this when was trying to use esp-idf as a git-submodule, to ease later project usage (no need to set up additional toolchain with specific versions and stuff).

My cmake file runs install.sh script to install the toolchain and required python modules, then loads the needed environment at configure-time and includes the esp-idf project.cmake. It configures fine, but flash target includes bootloader ExternalProject, which requires ESP_IDF and friends to be set, which fails at build time.

Describe the solution you'd like

As a quick test I added this line to bootloader external project call:

    CONFIGURE_COMMAND "${CMAKE_COMMAND}" -E env "PATH=$ENV{PATH}" "IDF_PATH=$ENV{IDF_PATH}"
        "IDF_PYTHON_ENV_PATH=$ENV{IDF_PYTHON_ENV_PATH}" "OPENOCD_SCRIPTS=$ENV{OPENOCD_SCRIPTS}"
        ${CMAKE_COMMAND} "${CMAKE_CURRENT_LIST_DIR}/subproject"

It effectively passes all environment variables exported by idf.py export to inner cmake project. After this make flash runs fine, but maybe some other target may fail with this.

Ultimately I want to lift the requirement of esp-idf environment being required during build time. It can be achieved to explicitly passing needed values to build-time commands.

Describe alternatives you've considered

Didn't come up with any

Additional context

cmake script

``` function(setup_idf_tools IDF_PATH) set(IDF_PATH "${IDF_PATH}" PARENT_SCOPE) set(ENV{IDF_PATH} "${IDF_PATH}") set(ENV{IDF_TOOLS_PATH} "${CMAKE_CURRENT_BINARY_DIR}/esp-idf-tools/") execute_process(COMMAND "$ENV{IDF_PATH}/install.sh" RESULT_VARIABLE ret) if(NOT ret EQUAL "0") message( FATAL_ERROR "Bad exit status") endif() execute_process(COMMAND "$ENV{IDF_PATH}/tools/idf_tools.py" export --format key-value RESULT_VARIABLE ret OUTPUT_VARIABLE idf_env) if(NOT ret EQUAL "0") message( FATAL_ERROR "Bad exit status") endif() STRING(REGEX REPLACE ";" "\\\\;" idf_env "${idf_env}") STRING(REGEX REPLACE "\n" ";" idf_env "${idf_env}") foreach(NameAndValue ${idf_env}) # Strip leading spaces string(REGEX REPLACE "^[ ]+" "" NameAndValue ${NameAndValue}) # Find variable name string(REGEX MATCH "^[^=]+" Name ${NameAndValue}) # Find the value string(REPLACE "${Name}=" "" Value ${NameAndValue}) # replace the $PATH string(REPLACE "$PATH" "$ENV{PATH}" Value ${Value}) # Set the variable message("setting ${Name} = ${Value}") set(ENV{${Name}} "${Value}") endforeach() message("PATH = $ENV{PATH}") endfunction() ```

[seymar]

+1 I'm all for the idea of using esp-idf as a submodule. Would be nice if there was real support for this without needing to hack things together.

[galah92]

I also want this.

[projectgus]

Hi @DCNick3,

Thanks for explaining your requirement here.

Just to check I understand what's happening here:

• When you run CMake, no environment variables specific to ESP-IDF are already set.
• Your top-level CMake file is adding the environment variables for IDF_PATH, etc. to the CMake ENV variable where they're used by the top-level project & app build.
• When the ESP-IDF build system recursively builds the bootloader via ExternalProject, it passes the system environment variables which don't include any changes made to ENV by the parent CMake process.
• As a result, the bootloader build process fails unless you add a step in the ExternalProject to push all the necessary variables into the subproject's CMake cache of ENV

Is that right?

I think the workaround you've found should work fine, my main concern is future-proofing it so it doesn't break if some other environment variable is added in the future (although I don't think we'll do this if we can help it). Perhaps there's a way to push all of the parent's ENV into the ExternalProject...

[DCNick3]

Is that right?

Yes, this is what happens.

Also, similar thing happens with kconfig target, in tools/cmake/kconfig.cmake. Kconfig expects IDF_ENV environment variable to be set. Maybe there are more such places, but I didn't encounter any environment-related failures in my use-cases.

Perhaps there's a way to push all of the parent's ENV into the ExternalProject...

Good point. I think running "${CMAKE_COMMAND}" "-E" "environment" at configure time should print them, so it would be quite straightforward to pass all of them to either custom target or ExternalProject