search

platformio / platformio-core

Your Gateway to Embedded Software Development Excellence :alien:
https://platformio.org
Apache License 2.0
7.84k stars 789 forks source link

ARM mbed: Compilation of project with long file paths not possible #2877

Open maxgerhardt opened 5 years ago

maxgerhardt commented 5 years ago

What kind of issue is this?

Configuration

Operating system: Windows 10 x64

PlatformIO Version (platformio --version): 4.0.1b3

Description of problem

When compiling the sample project given below, compilation fails as soon as the first source file of an external library or the main project is compiled. The occurring error is 

arm-none-eabi-g++: error: CreateProcess: No such file or directory

Which might hint at file path length problems (see https://github.com/MarlinFirmware/Marlin/issues/7967).

Steps to Reproduce

  1. Import the sample project given below
  2. Before compiling, execute git config --system core.longpaths true, otherwise th git clone will fail due to overlong filepaths
  3. Compile on Windows 10
  4. Observer error message

Actual Results

Processing nucleo_l152re (platform: ststm32; board: nucleo_l152re; framework: mbed)
CONFIGURATION: https://docs.platformio.org/page/boards/ststm32/nucleo_l152re.html
PLATFORM: ST STM32 5.5.0 > ST Nucleo L152RE
HARDWARE: STM32L152RET6 32MHz, 80KB RAM, 512KB Flash
DEBUG: Current (stlink) On-board (stlink) External (blackmagic, jlink)
PACKAGES: toolchain-gccarmnoneeabi 1.70201.0 (7.2.1), framework-mbed 5.51204.190701 (5.12.4)
Collecting mbed sources...
LDF: Library Dependency Finder -> http://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ soft
Found 1 compatible libraries
Scanning dependencies...
Dependency Graph
|-- <ArduinoCore-nRF528x-mbedos> #a4b622b [git+https://github.com/arduino/ArduinoCore-nRF528x-mbedos.git] (C:\Users\Maxi\Documents\stackoverflow_testing\.pio\libdeps\nucleo_l152re\ArduinoCore-nRF528x-mbedos)

arm-none-eabi-g++ -o .pio\build\nucleo_l152re\src\main_esp.o -c -std=gnu++98 -fno-rtti [...] @"C:\Users\Maxi\Documents\stackoverflow_testing\.pio\build\nucleo_l152re\longcmd-7828fe635133cd1a11dd36e80ff553da" src\main_esp.cpp

arm-none-eabi-g++: error: CreateProcess: No such file or directory

Expected Results

Compilation success.

If problems with PlatformIO Build System:

The content of platformio.ini:

[env:nucleo_l152re]
platform = ststm32
board = nucleo_l152re
framework = mbed
build_flags = -D PIO_FRAMEWORK_MBED_RTOS_PRESENT
lib_deps = https://github.com/arduino/ArduinoCore-nRF528x-mbedos.git

Source file to reproduce issue:

// Arduino code
#include "Arduino.h"

void setup() {
    pinMode(LED_BUILTIN, OUTPUT);
    Serial.begin(115200);
    Serial.println("Welcome to Arduino on Mbed OS");
}

void loop() {
    digitalWrite(LED_BUILTIN, HIGH);
    Serial.println("LED is now on!");
    delay(1000);
    digitalWrite(LED_BUILTIN, LOW);
    Serial.println("LED is now off!");
    delay(1000);
}

// Mbed OS code
int main() {
    setup();
    while (1) loop();
}
valeros commented 5 years ago

I don't think we can properly overcome this limitation as the input size of WINAPI function CreateProcess is limited to 32767 characters. As a workaround, you can install packages somewhere in the root directory of your drive (e.g C:\) by using packages_dir option.

maxgerhardt commented 5 years ago

The problem still persists when the library installed in C:\PLIB\ though :(. 

[platformio]
default_envs = nucleo_l152re
libdeps_dir = C:\PLIB\

[env:nucleo_l152re]
platform = ststm32
board = nucleo_l152re
framework = mbed
build_flags = -D PIO_FRAMEWORK_MBED_RTOS_PRESENT
lib_deps = https://github.com/arduino/ArduinoCore-nRF528x-mbedos.git
|-- <ArduinoCore-nRF528x-mbedos> #a4b622b [git+https://github.com/arduino/ArduinoCore-nRF528x-mbedos.git] (C:\PLIB\nucleo_l152re\ArduinoCore-nRF528x-mbedos)
...
arm-none-eabi-g++ -o .pio\build\nucleo_l152re\lib493\ArduinoCore-nRF528x-mbedos\cores\arduino\Interrupts.o [..] @"C:\Users\Maxi\Documents\stackoverflow_testing\.pio\build\nucleo_l152re\longcmd-5b09eca3e2f9c5ab662062cc89abf51b" C:\PLIB\nucleo_l152re\ArduinoCore-nRF528x-mbedos\cores\arduino\Interrupts.cpp

arm-none-eabi-g++: error: CreateProcess: No such file or directory

Is it the execution of the above arm-none-eabi-g++ command that fails? I cut it short for here but this doesn't seem like 32K of text to me in the console. Are we sure this is the cause of the problem? Are you e.g. able to reproduce that?

maxgerhardt commented 5 years ago

The command for the main_esp.cpp is 1645 characters long and references a longcmd of a size of 245724 characters. Basically the library causes every possible include path for all each of its subfolders to be included, leading to an enourmous amount of includes. Is that correct behaviour?

-I"C:/PLIB/nucleo_l152re/ArduinoCore-nRF528x-mbedos/cores/arduino/mbed/targets/TARGET_Analog_Devices/TARGET_ADUCM302X/TARGET_ADUCM3029" 
[..]
-I"C:/PLIB/nucleo_l152re/ArduinoCore-nRF528x-mbedos/cores/arduino/mbed/targets/TARGET_WIZNET/TARGET_W7500x/W7500x_Peripheral_Library"
valeros commented 5 years ago

There are plenty of includes in the framework itself so you need to move it to the root folder as well. Are you sure that ArduinoCore-nRF528x-mbedos can be used as a library without proper configuration? It seems that it's a full-fledged Arduino core.

I cut it short for here but this doesn't seem like 32K of text to me in the console. Are we sure this is the cause of the problem? Are you e.g. able to reproduce that?

I've bumped into similar problems before, all characters you pass to the compiler in form of raw arguments or in a special file are eventually transferred to the CreateProcess function and when the total length of these symbols is greater than 32KB it fails somewhere in toolchain core.

maxgerhardt commented 5 years ago

Are you sure that ArduinoCore-nRF528x-mbedos can be used as a library without proper configuration?

It's an mbed-os library which implements the Arduino HAL using mbed-os, see http://blog.janjongboom.com/2019/08/01/arduino-mbed.html

I've bumped into similar problems before, all characters you pass to the compiler in form of raw arguments or in a special file are eventually transferred to the CreateProcess function and when the total length of these symbols is greater than 32KB it fails somewhere in toolchain core.

If that is indeed the case, then the library must be modified to stop spamming 245KB in include paths. I'll look into this.

maxgerhardt commented 4 years ago

@valeros Another example of this problem has popped up in https://community.platformio.org/t/avr-g-error-createprocess-no-such-file-or-directory/12046/13?u=maxgerhardt, where compilation of a PlatformIO provided example project fails (https://github.com/platformio/platform-nordicnrf52/tree/master/examples/mbed-ble-thermometer).

I am able to reproduce that error 

arm-none-eabi-g++: error: CreateProcess: No such file or directory
work-mbed@5.51401.200110\components\802.15.4_RF\stm-s2lp-rf-driver\source\NanostackRfPhys2lp.cpp
arm-none-eabi-g++ -o .pio\build\delta_dfbm_nq620\FrameworkMbed\components\802.15.4_RF\stm-s2lp-rf-driver\source\at24mac_s2lp.o -c -std=gnu++14 -fno-rtti -Wvla -c -Wall -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -fmessage-length=0 -fno-exceptions -ffunction-sections -fdata-sections -funsigned-char -MMD -fno-delete-null-pointer-checks -fomit-frame-pointer -Os -g -DMBED_TRAP_ERRORS_ENABLED=1 -mcpu=cortex-m4 -mthumb -mfpu=fpv4-sp-d16 -mfloat-abi=softfp -DMBED_ROM_START=0x0 -DMBED_ROM_SIZE=0x80000 -DMBED_RAM_START=0x20000000 -DMBED_RAM_SIZE=0x10000 -include mbed_config.h -DPLATFORMIO=40202 -DARDUINO_GENERIC -DPIO_FRAMEWORK_MBED_RTOS_PRESENT -DARM_MATH_CM4 -DBOARD_PCA10040 -DCMSIS_VECTAB_VIRTUAL -DCMSIS_VECTAB_VIRTUAL_HEADER_FILE=\"cmsis_nvic.h\" -DCOMPONENT_NSPE=1 -DCOMPONENT_PSA_SRV_EMUL=1 -DCOMPONENT_PSA_SRV_IMPL=1 -DCONFIG_GPIO_AS_PINRESET -DDEVICE_ANALOGIN=1 -DDEVICE_FLASH=1 -DDEVICE_I2C=1 -DDEVICE_I2C_ASYNCH=1 -DDEVICE_INTERRUPTIN=1 -DDEVICE_LPTICKER=1 -DDEVICE_PORTIN=1 -DDEVICE_PORTINOUT=1 -DDEVICE_PORTOUT=1 -DDEVICE_PWMOUT=1 -DDEVICE_SERIAL=1 -DDEVICE_SERIAL_ASYNCH=1 -DDEVICE_SERIAL_FC=1 -DDEVICE_SLEEP=1 -DDEVICE_SPI=1 -DDEVICE_SPI_ASYNCH=1 -DDEVICE_SYSTICK_CLK_OFF_DURING_SLEEP=1 -DDEVICE_TRNG=1 -DDEVICE_USTICKER=1 -DFEATURE_BLE=1 -DMBED_MPU_CUSTOM -DMBED_TICKLESS -DNRF52 -DNRF52_PAN_12 -DNRF52_PAN_15 -DNRF52_PAN_20 -DNRF52_PAN_30 -DNRF52_PAN_31 -DNRF52_PAN_36 -DNRF52_PAN_51 -DNRF52_PAN_53 -DNRF52_PAN_54 -DNRF52_PAN_55 -DNRF52_PAN_58 -DNRF52_PAN_62 -DNRF52_PAN_63 -DNRF52_PAN_64 -DSWI_DISABLE0 -DTARGET_CORDIO -DTARGET_CORDIO_LL -DTARGET_CORTEX -DTARGET_CORTEX_M -DTARGET_DELTA_DFBM_NQ620 -DTARGET_FF_ARDUINO -DTARGET_LIKE_CORTEX_M4 -DTARGET_LIKE_MBED -DTARGET_M4 -DTARGET_MCU_NRF52832 -DTARGET_NAME=DELTA_DFBM_NQ620 -DTARGET_NORDIC -DTARGET_NORDIC_CORDIO -DTARGET_NRF52 -DTARGET_NRF52832 -DTARGET_NRF5x -DTARGET_RELEASE -DTARGET_RTOS_M4_M7 -DTARGET_SDK_15_0 -DTARGET_SOFTDEVICE_NONE -DTOOLCHAIN_GCC -DTOOLCHAIN_GCC_ARM -D__CMSIS_RTOS -D__CORTEX_M4 -D__FPU_PRESENT=1 -D__MBED_CMSIS_RTOS_CM -D__MBED__=1 @"C:\Users\Maxi\mbed-ble-thermometer\.pio\build\delta_dfbm_nq620\longcmd-4973fc6c3c0d3f729a439ffc951d0e04" C:\Users\Maxi\.platformio\packages\frameworarm-none-eabi-g++: error: CreateProcess: No such file or directory

And again the long command has 32720 bytes 

ls -l "C:\Users\Maxi\mbed-ble-thermometer\.pio\build\delta_dfbm_nq620\longcmd-4973fc6c3c0d3f729a439ffc951d0e04"
-rw-rw-rw-  1 Maxi 0 32720 2020-02-25 15:54 C:\Users\Maxi\mbed-ble-thermometer\.pio\build\delta_dfbm_nq620\longcmd-4973fc6c3c0d3f729a439ffc951d0e04

However the longcommand is also full of include directories for components not used in the project or thohse who would not appear during a real mbed-os build. It seems the generating code just recursively adds all folder paths to the list, while only excluding some? This code would need a rework if not even PIO's example projects can be compiled..

prostoAlexandr commented 4 years ago

I faced the same problem trying to compile blink example from mbed repository. Are there any known ways to fix that issue?

valeros commented 4 years ago

@prostoAlexandr It's a known issue, try to use packages_dir option so the include paths will be shorter, for example:

[platformio]
packages_dir = C:\\pio

[env:nucleo_l152re]
platform = ststm32
board = nucleo_l152re
framework = mbed
prostoAlexandr commented 4 years ago

@valeros It actually worked for me! Thanks

Barestos commented 4 years ago

Hello,

I'm facing the same issue with PIO : avr-g++: error: CreateProcess: No such file or directory What exactly do I need to do Valeros? I don't get it...

maxgerhardt commented 4 years ago

@Barestos

avr-g++: error: CreateProcess: No such file or directory

If you are using the AVR compiler you arent'c compiling for ARM mbed in which this problem was observed, so it may be something different.

What is the exact platformio.ini and code of your project? What's the build log when you do a Verbose Build / pio run -v? (You can use pastebin.com)

TD-er commented 3 years ago

This is really a nasty problem, as also discussed here: https://github.com/platformio/platform-espressif8266/issues/231

In short, there is a lot of path overhead in the longcmd file, so even when using short path names, you will run into this issue. Right now, I'm kinda stuck with my project as I am constantly running into this limitation where I can only have like 400 .cpp.o files that can be linked together to a binary. Some PIO environments I have, do have a longer env name and thus run into this issue even faster.

maxgerhardt commented 3 years ago

where I can only have like 400 .cpp.o files that can be linked together to a binary

But PlatformIO also archives libraries together, to form a .a archive from multiple .o object files. Is it maybe a remedy to move code from src/ (which would be linked together as individual object files) to lib/, to exchange multiple .o paths for one .a path? (That is, if the project doesn't require lib_archive = no for some reason..)

TD-er commented 3 years ago

Well, in my project the .cpp.o files of libraries also end up in the longcmd file. I haven't tried lib_archive settings myself, but I wonder if the linker is then able to remove the unused functions if it only handles .a files from a library. I'm constantly hitting all kind of limits regarding build size, so that would be a disaster if the unused functions of libraries could not be stripped from the final build.

HamzaHajeir commented 1 year ago

A cause of reaching the limit is the auto rename of the packages with a specific commit ID, such as: framework-arduinoespressif32@src-093fb8619fcf98944cc8fa3c6dce0b8d, which floods the command line, breaking the limit.

Can there be an option to the user to specify its name for that matter?

PIO Community post: https://community.platformio.org/t/windows-command-limit/34451

ivankravets commented 1 year ago

Can there be an option to the user to specify its name for that matter?

@HamzaHajeir, just remove all packages starting with framework-arduinoespressif32* from ~/.platformio/packages.

HamzaHajeir commented 1 year ago

This is what I'm doing now, the issue is that it will be forced to rename if any other project uses another version, as the other project uses unspecified version nor package of a platform.

On Mon, Jul 3, 2023, 12:38 Ivan Kravets @.***> wrote:

Can there be an option to the user to specify its name for that matter?

@HamzaHajeir https://github.com/HamzaHajeir, just remove all packages starting with framework-arduinoespressif32* from ~/.platformio/packages.

— Reply to this email directly, view it on GitHub https://github.com/platformio/platformio-core/issues/2877#issuecomment-1617731034, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH3O7J5HNOQODZ5BSGZ34YLXOKHK5ANCNFSM4IJ2JUTA . You are receiving this because you were mentioned.Message ID: @.***>