"Command line is too long" while building large project under Windows

nathanjel commented 3 years ago

[x] PlatformIO Core. If you’ve found a bug, please provide an information below.

Configuration

Windows 10 PlatformIO Core, version 5.0.4

Description of problem

When building my ESP32 project, I get "The command line is too long." at the final firmware linking step.

Steps to Reproduce

pio run -e esp32dev-21pin

Actual Results

Too long line issue - resulting linker command - Pastebin.com

The command line is too long.
*** [.pio\build\esp32dev-21pin\firmware.elf] Error 1

Expected Results

Successful build

If problems with PlatformIO Build System:

The content of platformio.ini: Too long line issue - platformio.ini - Pastebin.com

Additional info

The same project builds perfectly on Ubuntu 20.04.1 LTS, same version of PIO
Resulting linker command was captured by adding -v to the command line
Replication - as I work on a proprietary solution - is not straightforward, but what should help is that my project imports 76 external libraries thru lib_deps. That results in the final linker command of 10280 bytes. Please see my platformio.ini and the resulting linker command for building the 21-pin version - it should give You a good idea of how to reproduce this issue. I had to mask the files a bit - sorry for this.

justoke commented 3 years ago

I am having the same issue, although out of 3 env defined in the platformio.ini only the node32s builds, whereas the others fail with the "path too long". I tried moving the whole project to a folder in the root of my drive to short it as much as possible without any effect. I am using Windows 10.

nathanjel commented 3 years ago

I just tried with lib_archive = false and it fails on Windows 10

Linking .pio\build\esp32dev-21pin\bootloader.elf
Building .pio\build\esp32dev-21pin\bootloader.bin
esptool.py v3.0
Linking .pio\build\esp32dev-21pin\firmware.elf
xtensa-esp32-elf-g++: error: CreateProcess: No such file or directory
*** [.pio\build\esp32dev-21pin\firmware.elf] Error 1
=============================================================== [FAILED] Took 233.96 seconds ===============================================================
Environment     Status    Duration
--------------  --------  ------------
esp32dev-21pin  FAILED    00:03:53.956
========================================================== 1 failed, 0 succeeded in 00:03:53.956 ========================================================== 
PS C:\Work\E2-Sample-Demonstration>

justoke commented 3 years ago

I just tried with lib_archive = false and it fails on Windows 10
I tried the same with the same failure

nathanjel commented 3 years ago

I just managed to create a workaround based on original PIO builder/tools/piomaxlen.py. The fact it works confirms a need to further develop platformio-core. The existing version supports large linker command line only in case of large set of linker source files, but it does not handle libraries as of yet.

The workaround involves creating an additional script and referencing it from platformio.ini.

1. place `win_linker_workaround.py` in project root directory, aside `platformio.ini`

Import("env")

from os.path import isfile, join
import hashlib
import os

def _file_long_data_workaround(env, _data):
    data = env.subst(_data).replace("\\", "/")
    build_dir = env.subst("$BUILD_DIR")
    tmp_file = join(
        build_dir, "longcmd-%s" % hashlib.md5(data.encode()).hexdigest()
    )
    retval = '@"%s"' % tmp_file
    if not isfile(tmp_file):
        with open(tmp_file, "w") as fp:
            fp.write(data)
    return retval

def implant_hook_for_libflags():
    env.Replace(_long_libflags_hook=_file_long_data_workaround)
    coms = {}
    key = "LINKCOM"
    coms[key] = env.get(key, "").replace(
        "$_LIBFLAGS", "${_long_libflags_hook(__env__, _LIBFLAGS)}"
    )
    env.Replace(**coms)

implant_hook_for_libflags()

2. do needed changes to `platformio.ini`

...
[env]
...
extra_scripts =
  pre:win_linker_workaround.py
...

And for me it solves the problem:

Linking .pio\build\esp32dev-21pin\firmware.elf
Building .pio\build\esp32dev-21pin\firmware.bin
Retrieving maximum program size .pio\build\esp32dev-21pin\firmware.elf
Checking size .pio\build\esp32dev-21pin\firmware.elf
Advanced Memory Usage is available via "PlatformIO Home > Project Inspect"
RAM:   [=         ]  14.7% (used 48324 bytes from 327680 bytes)
Flash: [=======   ]  72.5% (used 1140630 bytes from 1572864 bytes)
esptool.py v3.0
============================================================================= [SUCCESS] Took 203.75 seconds =============================================================================

Environment     Status    Duration
--------------  --------  ------------
esp32dev-21pin  SUCCESS   00:03:23.755
============================================================================== 1 succeeded in 00:03:23.755 ==============================================================================

justoke commented 3 years ago

I've tried that but still get the error:

IS there anyway to confirm the script has run?

nathanjel commented 3 years ago

@justoke To my best guess - You're facing a different error - the one You reported occurs during source file compilation. My workaround only resolves the issue at final linking step. I guess in Your case a similar workaround for compile command could do the trick, but explaining it is beyond this forum.

Regarding checking if the script is being run - sure, add something like print("HELLO, IT's ME, THE SCRIPT!!!!!!!!!!!!!") in the _file_long_data_workaround function. You will notice this in the output.

justoke commented 3 years ago

@justoke To my best guess - You're facing a different error - the one You reported occurs during source file compilation. My workaround only resolves the issue at final linking step. I guess in Your case a similar workaround for compile command could do the trick, but explaining it is beyond this forum.

Regarding checking if the script is being run - sure, add something like print("HELLO, IT's ME, THE SCRIPT!!!!!!!!!!!!!") in the _file_long_data_workaround function. You will notice this in the output.

Thanks for the tip - you are right this is compile time, but this could be promising as a workaround. I'll share my findings

ivankravets commented 3 years ago

Please re-test with pio upgrade --dev Does it work now?

nathanjel commented 3 years ago

esp32dev-21pin  SUCCESS   00:00:43.636
=========================================== 1 succeeded in 00:00:43.636 ===========================================

Works! :) @justoke looking at the resolution in the code, which is perfect <3, I think Your issue and the overall problem will be finally gone for good!

Big thanks to PlatformIO team!!

justoke commented 3 years ago

Big thanks to PlatformIO team!!

Thank you!

platformio / platformio-core