search

espressif / esp-idf

Espressif IoT Development Framework. Official development framework for Espressif SoCs.
Apache License 2.0
13.7k stars 7.29k forks source link

Error in convert_to_cmake.py (IDFGH-3180) #5189

Closed ammaree closed 4 years ago

ammaree commented 4 years ago

Environment

Problem Description

Trying to convert a GNU Makefile to CMake format using convert_to_cmake.py

Expected Behavior

Completion with a converted CMakeLists.txt file

Actual Behavior

After running for approximately 2 minutes abnormal termination with error codes as below.

Steps to reproduce

ammaree@giga MINGW32 /c/Dropbox/devs/ws/z-appl/irmacs
# ../../z-sdk/esp-idf/tools/cmake/convert_to_cmake.py .

Debug Logs

Converting ....
Traceback (most recent call last):
  File "../../z-sdk/esp-idf/tools/cmake/convert_to_cmake.py", line 205, in <module>
    main()
  File "../../z-sdk/esp-idf/tools/cmake/convert_to_cmake.py", line 201, in main
    convert_project(args.project)
  File "../../z-sdk/esp-idf/tools/cmake/convert_to_cmake.py", line 127, in convert_project
    converted_components += convert_component(project_path, p)
  File "../../z-sdk/esp-idf/tools/cmake/convert_to_cmake.py", line 159, in convert_component
    cmakelists_path = os.path.join(component_path, "CMakeLists.txt")
  File "C:/msys32/mingw32/lib/python3.8/ntpath.py", line 137, in join
    genericpath._check_arg_types('join', path, *paths)
  File "C:/msys32/mingw32/lib/python3.8/genericpath.py", line 155, in _check_arg_types
    raise TypeError("Can't mix strings and bytes in path components") from None
TypeError: Can't mix strings and bytes in path components

Other items if possible

Makefile.txt

ammaree commented 4 years ago

Update: Have made sure all project and component directories are clean and contain no CMakeLists.txt files. Also checked project makefile to make sure absolute minimal, removed superfluous "debug" type statements/commands. All private component makefiles reduced to a single comment line ie default behaviour. So effectively all CMakeLists.txt files, except for the main project, should be absolute basic defaults.

I then ran 'convert_to_cmake.py --debug . >stdout.txt' againand checked the result. Essentially the first sign of a problem remains right at the end with the same error message. It looks like somewhere in the process a single character 'b' is appended to the component path and this causes an error on the first (component) directory to be converted.

dobairoland commented 4 years ago

Hi @ammaree. Thanks for reporting this issue.

Can you please try that the following fix is enough to fix your issue?

Please try to add .decode('utf-8') to the line 125 of tools/cmake/convert_to_cmake.py which now looks like:

            p = subprocess.check_output(cmd).strip()

so it would become:

            p = subprocess.check_output(cmd).strip().decode('utf-8')
ammaree commented 4 years ago

Thanks, that fixed the problem and the process completed. I am now stuck trying to resolve the issue around includes and components and cannot make any progress. The build process stops on the first include file of the main source not being able to locate the included file. Below is a significantly simplified project structure with a couple of different examples that occur many times.

+\z-sdk\esp-idf\
|
+\z-appl\irmacs\CMakeLists.txt
               \....
               \main\application.c
               \build\......
+\z-components\rtos-support\FreeRTOS_Support.c
              |            \FreeRTOS_Support.h
              \devices\ds2482\ds2482.c
              |              \ds2482.h
              |              \ds1990x.c
              |              \ds1990x.h
              |              \ds18x20.c
              |              \ds18x20.h
              \actuators\actuators.c
                        \actuators.h

The questions I have to get this to work are hopefully simple ones:

  1. I cannot find any detailed documentation explaining the ESP IDF implementation especially deviations and/or alternatives to CMAKe standard as per the CMake documentation () and Tutorial (). Where can I go to learn?

  2. My project required between 20-25 components depending on the hardware platform being built. Where is the best place to specify/define the 18 common and 2-7 selective components?

  3. REQUIRES AND PRIV_REQUIRES when used with idf_component_register() makes available header files (external to the component) either to the component/module or to both the module and its public header file(s). But it seems the same can be achieved with set(EXTRA_CFLAGS -I \dir\subdir). What is the best mechanism to use and why?

If you have a look at the makefile included in the original post it might shed more light on my question. Andre

ammaree commented 4 years ago

@dobairoland Apologies, forgot to tag you in previous comment

dobairoland commented 4 years ago

Thanks, that fixed the problem and the process completed.

Thank you @ammaree for the feedback. The fix will close this issue when it gets pushed to Github.

I cannot find any detailed documentation explaining the ESP IDF implementation especially deviations and/or alternatives to CMAKe standard as per the CMake documentation () and Tutorial (). Where can I go to learn?

This is the best place to start: https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-guides/build-system.html The best place to ask your follow-up questions is the forum: https://www.esp32.com/viewforum.php?f=23

My project required between 20-25 components depending on the hardware platform being built. Where is the best place to specify/define the 18 common and 2-7 selective components?

I think EXTRA_COMPONENT_DIRS is what you are looking for. An example is shown here: https://github.com/espressif/esp-idf/blob/master/examples/system/unit_test/test/CMakeLists.txt#L6. You can add a directory with common components to EXTRA_COMPONENT_DIRS and some other based on the given hardware platform. If you search for IDF_TARGET in our CMake files then you might find a lot of examples when we include components based on that variable.

REQUIRES AND PRIV_REQUIRES when used with idf_component_register() makes available header files (external to the component) either to the component/module or to both the module and its public header file(s). But it seems the same can be achieved with set(EXTRA_CFLAGS -I \dir\subdir). What is the best mechanism to use and why?

REQUIRES and PRIV_REQUIRES are higher level options which are recommended to use because they are further processed inside our cmake files.

ammaree commented 4 years ago

@dobairoland

Thanks for the help so far but I just can not move 1 step forward. I have tried 3 different methods for specifying the location of the very first include file in my application but exact same error all 3 methods. If I can get this 1 fixed I will (hopefully) be able to resolve the rest.

cmake_minimum_required(VERSION 3.5)
include($ENV{IDF_PATH}/tools/cmake/project.cmake)

####################################################################################################

set(MAKE_FILE_SEL 6)

if (MAKE_FILE_SEL MATCHES 2)                    # AC0X Specification token
  set(PROJECT_NAME 09d2b3e3-b034-4753-b1fe-13976a47bac4)
elseif (MAKE_FILE_SEL MATCHES 4)                # EM1P2 Specification token
  set(PROJECT_NAME c77a5342-06ae-49eb-99e8-bf0d84da7829)
elseif (MAKE_FILE_SEL MATCHES 6)                # ESP-WROVER-KIT-V4 Specification token
  set(PROJECT_NAME fc06ff52-9182-4256-8065-655330a57fe9)
elseif (MAKE_FILE_SEL MATCHES 7)                # DoIT DevKit V1 Specification token
  set(PROJECT_NAME 80292bbd-3f32-48f7-affb-cc87db2ea0f2)
elseif (MAKE_FILE_SEL MATCHES 8)                # M5 Stack Fire Specification token
  set(PROJECT_NAME 5df0681b-df82-4682-851e-e289c7866dce)
elseif (MAKE_FILE_SEL MATCHES 10)               # AC00 upstairs
  set(PROJECT_NAME 240ac402d278)
elseif (MAKE_FILE_SEL MATCHES 11)               # WM-EM1P2 (2x1P) upstairs
  set(PROJECT_NAME 807d3ac56254)
elseif (MAKE_FILE_SEL MATCHES 12)               # WM-EM3P2 (2x3P) upstairs
  set(PROJECT_NAME 30aea432d390)
elseif (MAKE_FILE_SEL MATCHES 13)               # KSS-AC00 Shaun
  set(PROJECT_NAME 240ac4036758)
elseif (MAKE_FILE_SEL MATCHES 14)               # WM-EM1P2 Shaun
  set(PROJECT_NAME 30aea430ac14)
elseif (MAKE_FILE_SEL MATCHES 15)               # AC00 DemoKit
  set(PROJECT_NAME 240ac4035c8c)
else()
  set(PROJECT_NAME XXXXXXXXXXXX)
endif()
project(${PROJECT_NAME})                        # Actual PROJECT name specification

####################################################################################################

set(P_H             esp32)                      # HAL directory selector
set(P_W             C:/Dropbox/devs/ws)         # WorkSpace top level directory
set(P_I             $ENV{IDF_PATH})
idf_build_get_property(P_A  PROJECT_DIR)                # Current application/project directory
set(P_B             ${P_A}/build)
set(P_C             ${P_W}/z-components)
set(P_S             ${P_W}/z-support)

set(EXTRA_COMPONENT_DIRS    ${P_C}/rtos-support)
set(COMPONENT_REQUIRES      ${P_C}/rtos-support)
set(EXTRA_CFLAGS        -I ${P_C}/rtos-support)

message("######################################################")
message("Project Name    : " ${PROJECT_NAME})
message("Platform HAL    : " ${P_H})
message("Path ESPIDF root: " ${P_I})
message("Path Application: " ${P_A})
message("Path Appl Build : " ${P_B})
message("Path Work Space : " ${P_W})
message("Path Components : " ${P_C})
message("Path Ext Support: " ${P_S})
message("COMPONENT_REQUIRES  : " ${COMPONENT_REQUIRES})
message("EXTRA_COMPONENT_DIRS: " ${EXTRA_COMPONENT_DIRS})
message("EXTRA_CFLAGS        : " ${EXTRA_CFLAGS})
message("######################################################")

The relevant output from the above message() commands are:

######################################################
Project Name    : fc06ff52-9182-4256-8065-655330a57fe9
Platform HAL    : esp32
Path ESPIDF root: C:\Dropbox\devs\ws\z-sdk\esp-idf
Path Application: C:/Dropbox/devs/ws/z-appl/irmacs
Path Appl Build : C:/Dropbox/devs/ws/z-appl/irmacs/build
Path Work Space : C:/Dropbox/devs/ws
Path Components : C:/Dropbox/devs/ws/z-components
Path Ext Support: C:/Dropbox/devs/ws/z-support
COMPONENT_REQUIRES  : C:/Dropbox/devs/ws/z-components/rtos-support
EXTRA_COMPONENT_DIRS: C:/Dropbox/devs/ws/z-components/rtos-support
EXTRA_CFLAGS        : -IC:/Dropbox/devs/ws/z-components/rtos-support
######################################################

Other than the / vs \ issue (when using \ in the set(P_W C:\Dropbox\devs\ws) result in error on "\D") I cannot see anything wrong yet the process terminates with the following output:

[1/10] Building C object esp-idf/main/CMakeFiles/__idf_main.dir/application.c.obj
FAILED: esp-idf/main/CMakeFiles/__idf_main.dir/application.c.obj
ccache C:\Users\andre\.espressif\tools\xtensa-esp32-elf\esp-2020r1-8.2.0\xtensa-esp32-elf\bin\xtensa-esp32-elf-gcc.exe -DHAVE_CONFIG_H -DMBEDTLS_CONFIG_FILE=\"mbedtls/esp_config.h\" -DUNITY_INCLUDE_CONFIG_H -DWITH_POSIX -Iconfig -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/newlib/platform_include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/freertos/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/freertos/xtensa/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/heap/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/log/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/lwip/include/apps -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/lwip/include/apps/sntp -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/lwip/lwip/src/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/lwip/port/esp32/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/lwip/port/esp32/include/arch -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/soc/src/esp32/. -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/soc/src/esp32/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/soc/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/esp_rom/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/esp_common/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/esp_system/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/xtensa/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/xtensa/esp32/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/esp32/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/driver/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/driver/esp32/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/esp_ringbuf/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/efuse/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/efuse/esp32/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/espcoredump/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/esp_timer/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/soc/soc/esp32/. -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/soc/soc/esp32/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/soc/soc/esp32/../include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/vfs/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/esp_wifi/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/esp_wifi/esp32/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/esp_event/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/esp_netif/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/esp_eth/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/tcpip_adapter/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/app_trace/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/mbedtls/port/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/mbedtls/mbedtls/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/mbedtls/esp_crt_bundle/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/bootloader_support/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/app_update/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/spi_flash/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/wpa_supplicant/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/wpa_supplicant/port/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/wpa_supplicant/include/esp_supplicant -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/nvs_flash/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/pthread/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/perfmon/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/asio/asio/asio/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/asio/port/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/cbor/port/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/coap/port/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/coap/port/include/coap -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/coap/libcoap/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/coap/libcoap/include/coap2 -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/console -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/nghttp/port/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/nghttp/nghttp2/lib/includes -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/esp-tls -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/esp_adc_cal/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/esp_gdbstub/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/tcp_transport/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/esp_http_client/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/esp_http_server/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/esp_https_ota/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/protobuf-c/protobuf-c -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/protocomm/include/common -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/protocomm/include/security -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/protocomm/include/transports -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/mdns/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/esp_local_ctrl/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/sdmmc/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/esp_serial_slave_link/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/esp_websocket_client/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/expat/expat/expat/lib -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/expat/port/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/wear_levelling/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/fatfs/diskio -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/fatfs/vfs -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/fatfs/src -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/freemodbus/common/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/idf_test/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/idf_test/include/esp32 -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/jsmn/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/json/cJSON -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/libsodium/libsodium/src/libsodium/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/libsodium/port_include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/mqtt/esp-mqtt/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/openssl/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/spiffs/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/ulp/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/unity/include -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/unity/unity/src -IC:/Dropbox/devs/ws/z-sdk/esp-idf/components/wifi_provisioning/include -mlongcalls -Wno-frame-address   -ffunction-sections -fdata-sections -fstrict-volatile-bitfields -Wall -Werror=all -Wno-error=unused-function -Wno-error=unused-but-set-variable -Wno-error=unused-variable -Wno-error=deprecated-declarations -Wextra -Wno-unused-parameter -Wno-sign-compare -ggdb -Og -std=gnu99 -Wno-old-style-declaration -D_GNU_SOURCE -DIDF_VER=\"v4.2-dev-1266-g2dc4bfe78-dirty\" -DESP_PLATFORM -MD -MT esp-idf/main/CMakeFiles/__idf_main.dir/application.c.obj -MF esp-idf\main\CMakeFiles\__idf_main.dir\application.c.obj.d -o esp-idf/main/CMakeFiles/__idf_main.dir/application.c.obj   -c ../main/application.c
../main/application.c:5:10: fatal error: FreeRTOS_Support.h: No such file or directory
 #include "FreeRTOS_Support.h"
          ^~~~~~~~~~~~~~~~~~~~
compilation terminated.

As we can see in the list the path specified in 3 different way are not included anywhere. All help to move this 1 step forward will be appreciated.

dobairoland commented 4 years ago

@ammaree Please try to move project(${PROJECT_NAME}) after you set EXTRA_CFLAGS. Does it make any difference?

I don't think you should try to set COMPONENT_REQUIRES.

ammaree commented 4 years ago

@dobairoland

Thanks for the response. Moving the project(${PROJECT_NAME}) to after the CFLAGS made a difference. Now looks like:

set(EXTRA_COMPONENT_DIRS    ${P_C}/rtos-support)
project(${PROJECT_NAME})                                # Actual PROJECT name specification

It gets past the first include hurdle and fails on the next include which helps. Now I only have to find out how to specify multiple includes with the save statement to proceed. Have tried separating multiple paths with ' ' ',' and ';' no luck. Any suggestions?

dobairoland commented 4 years ago

@ammaree Does "-Ipath1 -Ipath2 -Ipath3" help?

ammaree commented 4 years ago

Thanks, got beyond that one, error was due to individual component CMakeLists.txt files having syntax errors.

Have resolved the syntax errors in each CMakeLists.txt file but now every component is asking for additional includes from other components. I was under the impression that adding the whole list of components in the top level CMakeLists.txt would work for all components as well.

Am I missing something?