search

espressif / arduino-esp32

Arduino core for the ESP32
GNU Lesser General Public License v2.1
13.71k stars 7.43k forks source link

feat(platform)[2.x]: Use iwithprefixbefore to shorten the command lines #10622

Closed pillo79 closed 1 day ago

pillo79 commented 3 days ago

Description of Change

The command line for the ESP32 compilers is very long, and it is causing problems with the Arduino IDE on Windows systems which enforce a hard 32kB string limit.

This PR uses the -iprefix/-iwithprefixbefore options to shorten the compiler command line by changing

    "-I{compiler.sdk.path}/include/aaa"
    "-I{compiler.sdk.path}/include/bbb"
    "-I{compiler.sdk.path}/include/ccc"
    ...

to:

    -iprefix "{compiler.sdk.path}/include/"
    -iwithprefixbefore aaa
    -iwithprefixbefore bbb
    -iwithprefixbefore ccc
    ...

Some targets have up to 200 paths so this can save A LOT of text. On a sample run this patch shortened a single compile command from 28777 bytes to 10728.

Tests scenarios

Tested on Arduino IDE 2.3.3 on Windows 10 for the Arduino Nano ESP32 target (S3 based), with a username containing spaces and non-ASCII UTF-8 chars.

Related links

Related issue on the arduino-cli repo: https://github.com/arduino/arduino-cli/issues/2716

me-no-dev commented 3 days ago

@pillo79 do you plan to move away from 2.x?

pillo79 commented 3 days ago

@me-no-dev We definitely do, it's just that we need some careful pivot strategy instead of pushing a 3.x point blank to everyone. There's also a reported issue with the Nano and the recovery firmware we need to look into in more detail.

We will have to allow using 2.x for a long time I guess, and this linked issue was reported several times in the past as well from users with long names, so I looked into a fix. I've pushed this in our branch as a blanket "sed", but I was about to ask if that was the proper fix for upstream (or if you still accept developments on the 2.x release at all :slightly_smiling_face:).

me-no-dev commented 3 days ago

fix looks OK, we use the same trick for 3.x, though for different reasons.

We no longer push anything to 2.x and IDF 4.4 is EOL, but we will accept fixes. I suggest you look into 3.1 as it comes with ZigBee, Thread and Matter (along with many fixes and refactoring of included classes and tools)

pillo79 commented 2 days ago

I had initial reports that said it was working flawlessly, but one tester later reported issues such as the following: 

C:\Users\LocalComputerAdmin\AppData\Local\Arduino15\packages\esp32\tools\xtensa-esp32s3-elf-gcc\esp-2021r2-patch5-8.4.0/bin/xtensa-esp32s3-elf-g++ -DHAVE_CONFIG_H -DMBEDTLS_CONFIG_FILE="mbedtls/esp_config.h" -DUNITY_INCLUDE_CONFIG_H -DWITH_POSIX -D_GNU_SOURCE -DIDF_VER="v4.4.7-dirty" -DESP_PLATFORM -D_POSIX_READER_WRITER_LOCKS -iprefix C:\Users\LocalComputerAdmin\AppData\Local\Arduino15\packages\arduino\hardware\esp32\2.0.18-arduino.4/tools/sdk/esp32s3/include/ -iwithprefixbefore newlib/platform_include -iwithprefixbefore freertos/include -iwithprefixbefore freertos/include/esp_additions/freertos -iwithprefixbefore freertos/port/xtensa/include -iwithprefixbefore freertos/include/esp_additions -iwithprefixbefore esp_hw_support/include -iwithprefixbefore esp_hw_support/include/soc -iwithprefixbefore esp_hw_support/include/soc/esp32s3 -iwithprefixbefore esp_hw_support/port/esp32s3 -iwithprefixbefore esp_hw_support/port/esp32s3/private_include -iwithprefixbefore heap/include -iwithprefixbefore log/include -iwithprefixbefore lwip/include/apps -iwithprefixbefore lwip/include/apps/sntp -iwithprefixbefore lwip/lwip/src/include -iwithprefixbefore lwip/port/esp32/include -iwithprefixbefore lwip/port/esp32/include/arch -iwithprefixbefore soc/include -iwithprefixbefore soc/esp32s3 -iwithprefixbefore soc/esp32s3/include -iwithprefixbefore hal/esp32s3/include -iwithprefixbefore hal/include -iwithprefixbefore hal/platform_port/include -iwithprefixbefore esp_rom/include -iwithprefixbefore esp_rom/include/esp32s3 -iwithprefixbefore esp_rom/esp32s3 -iwithprefixbefore esp_common/include -iwithprefixbefore esp_system/include -iwithprefixbefore esp_system/port/soc -iwithprefixbefore esp_system/port/public_compat -iwithprefixbefore xtensa/include -iwithprefixbefore xtensa/esp32s3/include -iwithprefixbefore driver/include -iwithprefixbefore driver/esp32s3/include -iwithprefixbefore esp_pm/include -iwithprefixbefore esp_ringbuf/include -iwithprefixbefore efuse/include -iwithprefixbefore efuse/esp32s3/include -iwithprefixbefore vfs/include -iwithprefixbefore esp_wifi/include -iwithprefixbefore esp_event/include -iwithprefixbefore esp_netif/include -iwithprefixbefore esp_eth/include -iwithprefixbefore tcpip_adapter/include -iwithprefixbefore esp_phy/include -iwithprefixbefore esp_phy/esp32s3/include -iwithprefixbefore esp_ipc/include -iwithprefixbefore app_trace/include -iwithprefixbefore esp_timer/include -iwithprefixbefore mbedtls/port/include -iwithprefixbefore mbedtls/mbedtls/include -iwithprefixbefore mbedtls/esp_crt_bundle/include -iwithprefixbefore app_update/include -iwithprefixbefore spi_flash/include -iwithprefixbefore bootloader_support/include -iwithprefixbefore nvs_flash/include -iwithprefixbefore pthread/include -iwithprefixbefore esp_gdbstub/include -iwithprefixbefore esp_gdbstub/xtensa -iwithprefixbefore esp_gdbstub/esp32s3 -iwithprefixbefore espcoredump/include -iwithprefixbefore espcoredump/include/port/xtensa -iwithprefixbefore wpa_supplicant/include -iwithprefixbefore wpa_supplicant/port/include -iwithprefixbefore wpa_supplicant/esp_supplicant/include -iwithprefixbefore ieee802154/include -iwithprefixbefore console -iwithprefixbefore asio/asio/asio/include -iwithprefixbefore asio/port/include -iwithprefixbefore bt/common/osi/include -iwithprefixbefore bt/include/esp32c3/include -iwithprefixbefore bt/common/api/include/api -iwithprefixbefore bt/common/btc/profile/esp/blufi/include -iwithprefixbefore bt/common/btc/profile/esp/include -iwithprefixbefore bt/host/bluedroid/api/include/api -iwithprefixbefore bt/esp_ble_mesh/mesh_common/include -iwithprefixbefore bt/esp_ble_mesh/mesh_common/tinycrypt/include -iwithprefixbefore bt/esp_ble_mesh/mesh_core -iwithprefixbefore bt/esp_ble_mesh/mesh_core/include -iwithprefixbefore bt/esp_ble_mesh/mesh_core/storage -iwithprefixbefore bt/esp_ble_mesh/btc/include -iwithprefixbefore bt/esp_ble_mesh/mesh_models/common/include -iwithprefixbefore bt/esp_ble_mesh/mesh_models/client/include -iwithprefixbefore bt/esp_ble_mesh/mesh_models/server/include -iwithprefixbefore bt/esp_ble_mesh/api/core/include -iwithprefixbefore bt/esp_ble_mesh/api/models/include -iwithprefixbefore bt/esp_ble_mesh/api -iwithprefixbefore cbor/port/include -iwithprefixbefore unity/include -iwithprefixbefore unity/unity/src -iwithprefixbefore cmock/CMock/src -iwithprefixbefore coap/port/include -iwithprefixbefore coap/libcoap/include -iwithprefixbefore nghttp/port/include -iwithprefixbefore nghttp/nghttp2/lib/includes -iwithprefixbefore esp-tls -iwithprefixbefore esp-tls/esp-tls-crypto -iwithprefixbefore esp_adc_cal/include -iwithprefixbefore esp_hid/include -iwithprefixbefore tcp_transport/include -iwithprefixbefore esp_http_client/include -iwithprefixbefore esp_http_server/include -iwithprefixbefore esp_https_ota/include -iwithprefixbefore esp_https_server/include -iwithprefixbefore esp_lcd/include -iwithprefixbefore esp_lcd/interface -iwithprefixbefore protobuf-c/protobuf-c -iwithprefixbefore protocomm/include/common -iwithprefixbefore protocomm/include/security -iwithprefixbefore protocomm/include/transports -iwithprefixbefore mdns/include -iwithprefixbefore esp_local_ctrl/include -iwithprefixbefore sdmmc/include -iwithprefixbefore esp_serial_slave_link/include -iwithprefixbefore esp_websocket_client/include -iwithprefixbefore expat/expat/expat/lib -iwithprefixbefore expat/port/include -iwithprefixbefore wear_levelling/include -iwithprefixbefore fatfs/diskio -iwithprefixbefore fatfs/vfs -iwithprefixbefore fatfs/src -iwithprefixbefore freemodbus/freemodbus/common/include -iwithprefixbefore idf_test/include -iwithprefixbefore idf_test/include/esp32s3 -iwithprefixbefore jsmn/include -iwithprefixbefore json/cJSON -iwithprefixbefore libsodium/libsodium/src/libsodium/include -iwithprefixbefore libsodium/port_include -iwithprefixbefore mqtt/esp-mqtt/include -iwithprefixbefore openssl/include -iwithprefixbefore perfmon/include -iwithprefixbefore spiffs/include -iwithprefixbefore usb/include -iwithprefixbefore ulp/include -iwithprefixbefore wifi_provisioning/include -iwithprefixbefore rmaker_common/include -iwithprefixbefore esp_diagnostics/include -iwithprefixbefore rtc_store/include -iwithprefixbefore esp_insights/include -iwithprefixbefore json_parser/upstream/include -iwithprefixbefore json_parser/upstream -iwithprefixbefore json_generator/upstream -iwithprefixbefore esp_schedule/include -iwithprefixbefore espressif__esp_secure_cert_mgr/include -iwithprefixbefore esp_rainmaker/include -iwithprefixbefore gpio_button/button/include -iwithprefixbefore qrcode/include -iwithprefixbefore ws2812_led -iwithprefixbefore freertos/include/freertos -iwithprefixbefore arduino_tinyusb/tinyusb/src -iwithprefixbefore arduino_tinyusb/include -iwithprefixbefore esp_littlefs/include -iwithprefixbefore esp-dl/include -iwithprefixbefore esp-dl/include/tool -iwithprefixbefore esp-dl/include/typedef -iwithprefixbefore esp-dl/include/image -iwithprefixbefore esp-dl/include/math -iwithprefixbefore esp-dl/include/nn -iwithprefixbefore esp-dl/include/layer -iwithprefixbefore esp-dl/include/detect -iwithprefixbefore esp-dl/include/model_zoo -iwithprefixbefore esp32-camera/driver/include -iwithprefixbefore esp32-camera/conversions/include -iwithprefixbefore espressif__esp-dsp/modules/dotprod/include -iwithprefixbefore espressif__esp-dsp/modules/support/include -iwithprefixbefore espressif__esp-dsp/modules/support/mem/include -iwithprefixbefore espressif__esp-dsp/modules/windows/include -iwithprefixbefore espressif__esp-dsp/modules/windows/hann/include -iwithprefixbefore espressif__esp-dsp/modules/windows/blackman/include -iwithprefixbefore espressif__esp-dsp/modules/windows/blackman_harris/include -iwithprefixbefore espressif__esp-dsp/modules/windows/blackman_nuttall/include -iwithprefixbefore espressif__esp-dsp/modules/windows/nuttall/include -iwithprefixbefore espressif__esp-dsp/modules/windows/flat_top/include -iwithprefixbefore espressif__esp-dsp/modules/iir/include -iwithprefixbefore espressif__esp-dsp/modules/fir/include -iwithprefixbefore espressif__esp-dsp/modules/math/include -iwithprefixbefore espressif__esp-dsp/modules/math/add/include -iwithprefixbefore espressif__esp-dsp/modules/math/sub/include -iwithprefixbefore espressif__esp-dsp/modules/math/mul/include -iwithprefixbefore espressif__esp-dsp/modules/math/addc/include -iwithprefixbefore espressif__esp-dsp/modules/math/mulc/include -iwithprefixbefore espressif__esp-dsp/modules/math/sqrt/include -iwithprefixbefore espressif__esp-dsp/modules/matrix/mul/include -iwithprefixbefore espressif__esp-dsp/modules/matrix/add/include -iwithprefixbefore espressif__esp-dsp/modules/matrix/addc/include -iwithprefixbefore espressif__esp-dsp/modules/matrix/mulc/include -iwithprefixbefore espressif__esp-dsp/modules/matrix/sub/include -iwithprefixbefore espressif__esp-dsp/modules/matrix/include -iwithprefixbefore espressif__esp-dsp/modules/fft/include -iwithprefixbefore espressif__esp-dsp/modules/dct/include -iwithprefixbefore espressif__esp-dsp/modules/conv/include -iwithprefixbefore espressif__esp-dsp/modules/common/include -iwithprefixbefore espressif__esp-dsp/modules/matrix/mul/test/include -iwithprefixbefore espressif__esp-dsp/modules/kalman/ekf/include -iwithprefixbefore espressif__esp-dsp/modules/kalman/ekf_imu13states/include -iwithprefixbefore fb_gfx/include -IC:\Users\LocalComputerAdmin\AppData\Local\Arduino15\packages\arduino\hardware\esp32\2.0.18-arduino.4/tools/sdk/esp32s3/qio_opi/include -mlongcalls -ffunction-sections -fdata-sections -Wno-error=unused-function -Wno-error=unused-variable -Wno-error=deprecated-declarations -Wno-unused-parameter -Wno-sign-compare -ggdb -freorder-blocks -Wwrite-strings -fstack-protector -fstrict-volatile-bitfields -Wno-error=unused-but-set-variable -fno-jump-tables -fno-tree-switch-conversion -std=gnu++11 -fexceptions -fno-rtti -c -w -Os -w -x c++ -E -CC -DF_CPU=240000000L -DARDUINO=10607 -DARDUINO_NANO_ESP32 -DARDUINO_ARCH_ESP32 -DARDUINO_BOARD="NANO_ESP32" -DARDUINO_VARIANT="arduino_nano_nora" -DARDUINO_PARTITION_app3M_fat9M_fact512k_16MB -DESP32 -DCORE_DEBUG_LEVEL=0 -DARDUINO_RUNNING_CORE=1 -DARDUINO_EVENT_RUNNING_CORE=1 -DBOARD_HAS_PIN_REMAP -DBOARD_HAS_PSRAM -DUSB_MANUFACTURER="Arduino" -DUSB_PRODUCT="Nano ESP32" -DARDUINO_USB_MODE=0 -DARDUINO_USB_CDC_ON_BOOT=1 -DARDUINO_USB_MSC_ON_BOOT=0 -DARDUINO_USB_DFU_ON_BOOT=1 @C:\Users\LocalComputerAdmin\AppData\Local\Temp\arduino\sketches\3C772D064CB0C5F45E2365F3F0DC8163/build_opt.h @C:\Users\LocalComputerAdmin\AppData\Local\Temp\arduino\sketches\3C772D064CB0C5F45E2365F3F0DC8163/file_opts -IC:\Users\LocalComputerAdmin\AppData\Local\Arduino15\packages\arduino\hardware\esp32\2.0.18-arduino.4\cores\esp32 -IC:\Users\LocalComputerAdmin\AppData\Local\Arduino15\packages\arduino\hardware\esp32\2.0.18-arduino.4\variants\arduino_nano_nora C:\Users\LocalComputerAdmin\AppData\Local\Temp\arduino\sketches\3C772D064CB0C5F45E2365F3F0DC8163\sketch\Blink.ino.cpp -o nul
In file included from c:\users\localcomputeradmin\appdata\local\arduino15\packages\esp32\tools\xtensa-esp32s3-elf-gcc\esp-2021r2-patch5-8.4.0\xtensa-esp32s3-elf\include\c++\8.4.0\cstdlib:41,
                 from c:\users\localcomputeradmin\appdata\local\arduino15\packages\esp32\tools\xtensa-esp32s3-elf-gcc\esp-2021r2-patch5-8.4.0\xtensa-esp32s3-elf\include\c++\8.4.0\stdlib.h:36,
                 from C:\Users\LocalComputerAdmin\AppData\Local\Arduino15\packages\arduino\hardware\esp32\2.0.18-arduino.4/tools/sdk/esp32s3/include/newlib/platform_include/assert.h:21,
                 from c:\users\localcomputeradmin\appdata\local\arduino15\packages\esp32\tools\xtensa-esp32s3-elf-gcc\esp-2021r2-patch5-8.4.0\xtensa-esp32s3-elf\sys-include\sys\reent.h:503,
                 from C:\Users\LocalComputerAdmin\AppData\Local\Arduino15\packages\arduino\hardware\esp32\2.0.18-arduino.4/tools/sdk/esp32s3/include/newlib/platform_include/sys/reent.h:17,
                 from c:\users\localcomputeradmin\appdata\local\arduino15\packages\esp32\tools\xtensa-esp32s3-elf-gcc\esp-2021r2-patch5-8.4.0\xtensa-esp32s3-elf\sys-include\stdio.h:60,
                 from C:\Users\LocalComputerAdmin\AppData\Local\Arduino15\packages\arduino\hardware\esp32\2.0.18-arduino.4\cores\esp32/Arduino.h:27,
                 from C:\Users\LocalComputerAdmin\AppData\Local\Temp\arduino\sketches\3C772D064CB0C5F45E2365F3F0DC8163\sketch\Blink.ino.cpp:1:
c:\users\localcomputeradmin\appdata\local\arduino15\packages\esp32\tools\xtensa-esp32s3-elf-gcc\esp-2021r2-patch5-8.4.0\xtensa-esp32s3-elf\include\c++\8.4.0\xtensa-esp32s3-elf\no-rtti\bits\c++config.h:508:10: fatal error: bits/os_defines.h: No such file or directory
 #include <bits/os_defines.h>
          ^~~~~~~~~~~~~~~~~~~

The weird thing is that this file is there, in the same directory that contains the including file - which was obviously found. This was reproduced in a different PC... on a different file that gcc still "refuses" to find. We are investigating.

me-no-dev commented 2 days ago

Issue there is the length of the path on Windows. In newer versions we have renamed the packages with the toolchains to a very short ones in order to help the situation when users have longer usernames

pillo79 commented 1 day ago

Wow, I did not notice at first there were two different limits at play! :man_facepalming: I did exactly what you suggested to improve the MAX_PATH issue, but that is definitely a spot fix.

Since there's already a proper fix for this in 3.x and this repo is EOL'd, we'll keep this in our downstream for the time being. Again thanks for the help!