collect2.exe: fatal error: CreateProcess: No such file or directory

[TD-er]

I've been getting this error when building on VScode in Windows. If I build the exact same code in Linux (using command line) it just works out fine. It happens with PIO 5.0.1 and the latest 5.0.2 dev version. (reinstalled PIO and all its cache and downloaded packages a few times already)

Building in release mode
Compiling .pio\build\custom_beta_ESP8266_4M1M\src\ESPEasy.ino.cpp.o
Linking .pio\build\custom_beta_ESP8266_4M1M\ESP_Easy_mega_20201012_custom_beta_ESP8266_4M1M.elf
collect2.exe: fatal error: CreateProcess: No such file or directory
compilation terminated.
*** [.pio\build\custom_beta_ESP8266_4M1M\ESP_Easy_mega_20201012_custom_beta_ESP8266_4M1M.elf] Error 1

The strange thing is, it only happens with some of my PIO environments and also depends on some commits I make. I have been working very hard on converting all my .ino files to .cpp/.h (see: https://github.com/platformio/platformio-vscode-ide/issues/2067 ) and in the process I do see this happen on some of my PIO environments and after a new commit (changing again a few dozen files) it happens on even more environments. If I checkout an older commit it can be built again in a few environments that failed before.

So it just seems like I'm hitting some resource limit when building in VScode? In my changes from .ino to .cpp/.h I do try to include as little as possible in .h files an try to only include in the .cpp file. Only exception is files like ESPEasy_common.h, which are included almost everywhere. N.B. my environments differ in the used ESP8266/Arduino core library and also the number of "plugins" of my code included in the build.

I created a branch of which the last commit made compiling impossible in VS code for the custom_ESP8266_4M1M environment. https://github.com/TD-er/ESPEasy/tree/build/failing_custom_windows_builds When going back 1 commit in that branch, it compiles again. Even building in verbose mode is not giving any clue why it failed to generate the .elf file.

So to summarize:

• Some env. builds in VScode (Windows) suddenly fail after converting some .ino files to .cpp/.h
• Command line build in the terminal of VScode also fails.
• The same code built in Linux (command line) works just fine
• Reverting back to a previous commit makes those envs build again in VScode.

[valeros]

Hi @TD-er ! On Windows the maximum length of the string (including all arguments) passed to CreateProcess() is 32768 characters and it looks like the final linker command exceeds that limit. Please attach here verbose output so I can try to come up with possible workarounds.

[TD-er]

Hmm that sounds plausible.

I first made a normal failing build and then a verbose build as it would then just fail on the linker part, right?

Building in release mode
xtensa-lx106-elf-g++ -o .pio\build\custom_ESP8266_4M1M\src\ESPEasy.ino.cpp.o -c -fno-rtti -std=c++11 -mtarget-align -fno-exceptions -O2 -Wno-deprecated-declarations -Os -mlongcalls -mtext-section-literals -falign-functions=4 -U__STRICT_ANSI__ -ffunction-sections -fdata-sections -fno-exceptions -Wall -DCONTROLLER_SET_ALL -DNOTIFIER_SET_NONE -DUSES_P001 -DUSES_P002 -DUSES_P004 -DUSES_P027 -DUSES_P028 -DUSES_P036 -DUSES_P045 -DUSES_P049 -DUSES_P052 -DUSES_P056 -DUSES_P059 -DUSES_P081 -DUSES_P082 -DUSES_P085 -DUSES_P100 -DUSES_C016 -DUSES_C018 -DFEATURE_MDNS -DFEATURE_SD -DFEATURE_I2CMULTIPLEXER -DUSE_SETTINGS_ARCHIVE -DPLATFORMIO=50001 -DESP8266 -DARDUINO_ARCH_ESP8266 -DARDUINO_ESP8266_ESP12 -DBUILD_GIT=\"\" -DNDEBUG -DVTABLES_IN_FLASH -DPIO_FRAMEWORK_ARDUINO_LWIP2_HIGHER_BANDWIDTH_LOW_FLASH -DPUYA_SUPPORT=1 -DCORE_POST_2_5_0 -DBEARSSL_SSL_BASIC -DCORE_POST_2_6_0 -DPIO_FRAMEWORK_ARDUINO_ESPRESSIF_SDK22x_190703 -DMQTT_MAX_PACKET_SIZE=1024 -DHTTPCLIENT_1_1_COMPATIBLE=0 -DPLUGIN_BUILD_CUSTOM -DF_CPU=80000000L -D__ets__ -DICACHE_FLASH -DARDUINO=10805 -DARDUINO_BOARD=\"PLATFORMIO_ESP12E\" -DFLASHMODE_DOUT -DLWIP_OPEN_SRC -DNONOSDK22x_190703=1 -DTCP_MSS=1460 -DLWIP_FEATURES=0 -DLWIP_IPV6=0 -Isrc -Ilib\Adafruit_SGP30-1.0.5 -Ilib\SparkFun_APDS-9960_Sensor_Arduino_Library\src -Ilib\Adafruit_MPR121 -Ilib\HT16K33 -Ilib\esp8266-oled-ssd1306 -Ilib\LiquidCrystal_I2C -IC:\users\gijsn\.platformio\packages\framework-arduinoespressif8266\libraries\ESP8266HTTPClient\src -IC:\users\gijsn\.platformio\packages\framework-arduinoespressif8266\libraries\ESP8266HTTPUpdateServer\src -IC:\users\gijsn\.platformio\packages\framework-arduinoespressif8266\libraries\ArduinoOTA -Ilib\pubsubclient\src -Ilib\I2Cdevlib -IC:\users\gijsn\.platformio\packages\framework-arduinoespressif8266\libraries\ESP8266WebServer\src -IC:\users\gijsn\.platformio\packages\framework-arduinoespressif8266\libraries\Servo\src -IC:\users\gijsn\.platformio\packages\framework-arduinoespressif8266\libraries\SD\src -IC:\users\gijsn\.platformio\packages\framework-arduinoespressif8266\libraries\SDFS\src -IC:\users\gijsn\.platformio\packages\framework-arduinoespressif8266\libraries\ESP8266SdFat\src -IC:\users\gijsn\.platformio\packages\framework-arduinoespressif8266\libraries\LittleFS\src -IC:\users\gijsn\.platformio\packages\framework-arduinoespressif8266\libraries\ESP8266mDNS\src -IC:\users\gijsn\.platformio\packages\framework-arduinoespressif8266\libraries\DNSServer\src -Ilib\TinyGPSPlus-1.0.2\src -Ilib\SDM_Energy_Meter -Ilib\RN2483-Arduino-Library\src -Ilib\Regexp\src -Ilib\MechInputs -Ilib\NewPingESP8266 -Ilib\SerialDevices -Ilib\HLW8012_1.1.1\src -Ilib\HeatpumpIR -Ilib\IRremoteESP8266\src -Ilib\Blynk\src -IC:\users\gijsn\.platformio\packages\framework-arduinoespressif8266\libraries\ESP8266WiFi\src -Ilib\ArduinoJson-6.x\src -Ilib\AM2320 -Ilib\Adafruit_NeoPixel -Ilib\Adafruit_Motor_Shield_V2 -Ilib\Adafruit_Motor_Shield_V2\utility -Ilib\ccronexpr -Ilib\AS_BH1750 -Ilib\Adafruit_TSL2591 -Ilib\Adafruit_Sensor -Ilib\Adafruit_TCS34725 -Ilib\LOLIN_EPD\src "-I.pio\libdeps\custom_ESP8266_4M1M\Adafruit ILI9341" "-I.pio\libdeps\custom_ESP8266_4M1M\Adafruit GFX Library" "-I.pio\libdeps\custom_ESP8266_4M1M\Adafruit BusIO" -I.pio\libdeps\custom_ESP8266_4M1M\ESPeasySerial -Ilib\SC16IS752 -IC:\users\gijsn\.platformio\packages\framework-arduinoespressif8266\libraries\Wire -IC:\users\gijsn\.platformio\packages\framework-arduinoespressif8266\libraries\SPI -I.pio\build\custom_ESP8266_4M1M\core -IC:\users\gijsn\.platformio\packages\framework-arduinoespressif8266\tools\sdk\include -IC:\users\gijsn\.platformio\packages\framework-arduinoespressif8266\tools\sdk\libc\xtensa-lx106-elf\include -IC:\users\gijsn\.platformio\packages\framework-arduinoespressif8266\cores\esp8266 -IC:\users\gijsn\.platformio\packages\framework-arduinoespressif8266\tools\sdk\lwip2\include -IC:\users\gijsn\.platformio\packages\framework-arduinoespressif8266\variants\nodemcu src\ESPEasy.ino.cpp
xtensa-lx106-elf-g++ -o .pio\build\custom_ESP8266_4M1M\ESP_Easy_mega_20201012_custom_ESP8266_4M1M.elf -T eagle.flash.4m1m.ld -Os -nostdlib -Wl,--no-check-sections -Wl,-static -Wl,--gc-sections -Wl,-wrap,system_restart_local -Wl,-wrap,spi_flash_read -u app_entry -u _printf_float -u _scanf_float -u _DebugExceptionVector -u _DoubleExceptionVector -u _KernelExceptionVector -u _NMIExceptionVector -u _UserExceptionVector @"C:\GitHub\TD-er\ESPEasy\.pio\build\custom_ESP8266_4M1M\longcmd-a302cf0692c4fc5d40ee988c2df28d2a" -L.pio\build\custom_ESP8266_4M1M -L.pio\build\custom_ESP8266_4M1M\ld -LC:\users\gijsn\.platformio\packages\framework-arduinoespressif8266\tools\sdk\lib -LC:\users\gijsn\.platformio\packages\framework-arduinoespressif8266\tools\sdk\ld -LC:\users\gijsn\.platformio\packages\framework-arduinoespressif8266\tools\sdk\libc\xtensa-lx106-elf\lib -LC:\users\gijsn\.platformio\packages\framework-arduinoespressif8266\tools\sdk\lib\NONOSDK22x_190703 -Wl,--start-group .pio\build\custom_ESP8266_4M1M\libFrameworkArduinoVariant.a .pio\build\custom_ESP8266_4M1M\libFrameworkArduino.a -lstdc++ -lhal -lphy 
-lpp -lnet80211 -lwpa -lcrypto -lmain -lwps -lbearssl -laxtls -lespnow -lsmartconfig -lairkiss -lwpa2 -lstdc++ -lm -lc -lgcc -llwip2-1460 -Wl,--end-group
xtensa-lx106-elf-g++: error: CreateProcess: No such file or directory
*** [.pio\build\custom_ESP8266_4M1M\ESP_Easy_mega_20201012_custom_ESP8266_4M1M.elf] Error 1
=========================================================================================================================================================================== [FAILED] Took 9.96 seconds ===========================================================================================================================================================================

This is less than 6k, so I am not entirely sure it will be the true error.

If you like, I can also make a full verbose build, but I guess the log will not entirely fit in the terminal window.

[valeros]

The linker also expands C:\GitHub\TD-er\ESPEasy\.pio\build\custom_ESP8266_4M1M\longcmd-a302cf0692c4fc5d40ee988c2df28d2a file when finally invokes CreateProcess. How many symbols are there?

[TD-er]

Just made a new test build on this laptop to check. Size of the file: 34.1 KB (34,986 bytes) It contains all .cpp.o files incl path, so that does explain why some envs had this issue while others didn't experience it.

For example:

".pio/build/custom_alt_wifi_ESP8266_4M1M/src/src/WebServer/WebServer_login.cpp.o"

[TD-er]

Right now, quite a lot of function still have a global scope (left over from converting from .ino files) Would it be helpful for this particular issue if I move some into their own class? Then the separate files would first be linked into a new (bigger) .o file. Disadvantage is that the linker may not be as effective in leaving out unused functions.

[valeros]

Does it make sense to move some common functionality to standalone libraries? PlatformIO will be able to link them as static archives so it should help with the long linker command.

[TD-er]

I can try, to see if this file will be smaller. But still, if functions are not used, they will not be left out of the final binary right? If something is used from the "library", then the entire object will be linked.

[valeros]

Thanks to the -Wl,--gc-sections flag, unused functions won't be linked.

[TD-er]

Also if they are part of a library?

[valeros]

Doesn't matter, the linker is smart enough to throw out unnecessary symbols even from libraries.

[TD-er]

Good to know. I will try to split some into the lib dir.

[TD-er]

I now was looking again into this issue and thus also did look at the longcommand-<hexvalue> file. It does have an entry per .cpp.o file, including all library code used.

Just a small section of this:

".pio/build/custom_ESP8266_4M1M/libf62/SPI/SPI.cpp.o"
".pio/build/custom_ESP8266_4M1M/lib525/Wire/Wire.cpp.o"
".pio/build/custom_ESP8266_4M1M/lib551/SC16IS752/SC16IS752.cpp.o"
".pio/build/custom_ESP8266_4M1M/lib9e8/ESPeasySerial/ESPEasySC16IS752_Serial.cpp.o"
".pio/build/custom_ESP8266_4M1M/lib9e8/ESPeasySerial/ESPEasySerial_ESP32.cpp.o"
".pio/build/custom_ESP8266_4M1M/lib9e8/ESPeasySerial/ESPEasySerial_ESP8266_noSWserial.cpp.o"
".pio/build/custom_ESP8266_4M1M/lib9e8/ESPeasySerial/ESPEasySoftwareSerial.cpp.o"
".pio/build/custom_ESP8266_4M1M/lib9e8/ESPeasySerial/ESPeasySerial_ESP8266.cpp.o"

(It is all in one line, but for readability, I placed each on a single line) As you can see, the .cpp.o files here are from a library but still are used in this longcommand.

So I don't think it will help here to move parts to their own library, or am I missing something here?

[TD-er]

Just as an idea... All lines use the same path prefix, which in my example uses 30 bytes per entry and this (failed) longcommand file has 426 entries. So if the longcommand could be generated relative to the common path, it would save over 12 kByte in size.

Could this file be used with some relative path?

[TD-er]

I just found a bit more room by going over the libraries mentioned in the lib_ignore part of an environment definition. Still this does not appear to be a fix for all my environments, so I will continue to dig further.

Edit: Setting lib_ldf_mode = deep+ does help here, but not in all environments. :(

[TD-er]

OK, dug a bit deeper into what is happening here and I found the deep+ does only scan for libraries to be included or ignored, not individual cpp/h files (which is also implied by the name of lib_ldf_mode) This means I cannot continue to convert the rest of the .ino files in my project to .cpp/.h files or else I will run into build issues I cannot find a work-around for.

To explain this, you need to know the structure of my project.

• ESPEasy has a number of core elements (Rules processing / WiFi / event handling / webserver / etc)
• Interaction with sensor hardware is done via so called plugins.
• Sending out data is done via Controller plugins (e.g. to MQTT broker)

These plugins and controllers are now coded in .ino files, but some already use objects coded in .cpp/.h files Those are only included when the specific .ino Plugin file is meant to be included in the build. But the .cpp.o files of those all occur in the longcmd file. Meaning if I convert the other Plugin .ino files to .h/.cpp files they will all end up in the longcmd file and thus every build will fail. (currently we have over 100 plugin and controller .ino files)

The only thing I can think of right now, will be a true nightmare to maintain as it means every plugin and every controller will end up in a separate library.

So I really hope there is a better solution for this. Maybe @ivankravets may have a suggestion here to overcome this issue?

Edit: There may be a really dirty work-around for it, if I don't convert the plugin .ino files to .h files instead of .cpp files. These .h files must then be included in a single .cpp file (only once) acting as a wrapper. But I really dislike this idea due to the fact all code in such a .h file will be inline. Meaning if a function is defined there and called twice, it will result in a larger executable file. And still I need to have code in separate files which will be present in all compile runs.

It is something I really don't like to do.

[valeros]

The only thing I can think of right now, will be a true nightmare to maintain as it means every plugin and every controller will end up in a separate library.

Could you please explain in a few words what exactly is hard to maintain it that case? In my opinion, isolated modules (the granularity of that modules may vary as your project evolves) should be much easier to develop and implicitly lead to better software design.

The main issue here is that all project files are located in the src folder, all these files are linked as objects in the final command. We need to move away from that structure by extracting related parts of the project into separate modules that can be linked as static archives.

[TD-er]

Could you please explain in a few words what exactly is hard to maintain it that case? In my opinion, isolated modules (the granularity of that modules may vary as you project evolves) should be much easier to develop and implicitly lead to better software design.

Well it depends on what PlatformIO may find acceptable as a library. If the "library per plugin" can simply be a directory in the libs folder without an existing git repository, then it only adds some overhead in library.json and library.properties file. But it can still reside in the main repository of my project. This way, if I change something that may affect a number of plugins, then it could still all be in a single commit. If I also need to update dozens of separate repositories, then it will be a nightmare to maintain.

Right now I am still in the process of refining the interface between the core and the plugins, so the interface between the plugins and core is still a bit in flux. Meaning some changes may affect quite some plugin files.

I wonder how building it in Arduino IDE will be affected by such changes. How does Arduino IDE link all .cpp.o files? Do they also run into these Windows command line length limitations?

[valeros]

If the "library per plugin" can simply be a directory in the libs folder without an existing git repository, then it only adds some overhead in library.json and library.properties file. But it can still reside in the main repository of my project.

You can group your libs as you like and keep them in your main repository, that's what the lib folder was made for. You don't even need the library.json file by default.

How does Arduino IDE link all .cpp.o files? Do they also run into these Windows command line length limitations?

Arduino IDE (as any other app) is exposed to that limitation since it's imposed by the OS.

[TD-er]

I do understand why it may fail, if it indeed tries to generate a similar line. The main problem for me is that PlatformIO can be instructed to ignore a library, but not sure how Arduino IDE may handle this. Does Arduino IDE try to create separate intermediate files first or does it also generate .o files from every .cpp file it finds and then trying to link it at once.

[valeros]

Does Arduino IDE try to create separate intermediate files first or does it also generate .o files from every .cpp file it finds and then trying to link it at once.

If I'm not mistaken, Arduino IDE links only framework files and libraries as static archives. Project source files are linked as objects.

[TD-er]

OK, that would make sense. Probably still need to check it before I make my final decision on how to proceed with this.

[TD-er]

I was just thinking about this a bit more and I realized this (which I hope isn't correct).

Imagine I will create a library of every single plugin and controller (e.g. just about any .ino file left in my project) These plugins do need parts of the core of my project, but you just can't include parts of the main project in the library. This implies the entire core should also be moved into a library. Thus my entire project would then consist of a single file including all libraries. Isn't that a bit defeating the whole idea of making if more structured? And it is all needed to overcome a truly silly limitation, that we cannot freely create a build based on a number of pre-selected compiled object files.

I just got the feeling it is heading in a completely wrong direction.

[thomastech]

Assuming I grasp the problem, perhaps Platformio's linker could copy all the files to a temporary root folder, then link the files there. Afterwords, cleanup by deleting the temp folder.

It seems to me that would reduce the command line length in a typical build environment. And this alternate linking method could be an option enabled by an entry in platformio.ini, where the drive root folder could be specified.

• Thomas

[mcspr]

Just to note, this might also affect lib_deps=... entries, which was noticed some time ago here: https://gitter.im/tinkerman-cat/espurna?at=5f32a2b3b103e45c830df0f5 https://github.com/xoseperez/espurna itself builds purely from .cpp files, but as noticed in the same thread, we allow to reduce the amount of .o files by doing something like the current .ino behaviour - concating all the .cpp files (another example is cmake's unity builds or this Wikipedia page (edited))

Re: above, it might be a good idea to launch the linker with .pio/build/\<env> as pwd? If this is the main problem here.

(I also believe this could be migrated to the platformio-core issue tracker?)

[TD-er]

Yep it seems it is indeed a PIO core issue.

And stripping the shared path of all .cpp.o file entries might save roughly 12k in my project's longcmd file.

I will have a look at what was discussed in the topics you linked.

[Jason2866]

@TD-er i reorganized the libs in Tasmota in subfolders. Reason was not a compile fail. The compile time was growing more and more because Platformio compiles all libs even when not used. Tasmota does not use libdeps (no easy change...) So i grouped the libs, and activate only the groups needed for the variant build. No code change needed and the compile time is reduced nicely. Maybe a way for your issue?

[TD-er]

@Jason2866 Yep, I was also thinking about something similar. Another option could be to use the Python scripts I already use in my PIO envs to deduct the needed libraries and thus generate a lib_ignore for a specific build.

This means I have to transfer my build defines to the Python layer, which may not be a bad thing. Right now I do have the problem that I sometimes need to define things at compile arguments and not in my C++ project code as they are needed for libraries. So it is already a bit of a mix between where to put the defines, which makes it not always clear to users where to put the defines.

[Jason2866]

@TD-er Had the idea with dynamic disabling libs too. After working on it i trashed the idea. The amount of (extra) stuff needs maintaining was more than i thought. After looking which libs are used for the different Tasmota versions it was easy with libs folder groups. A nice automatic disabling libs (python) PIO thing would be still great. If you have something please let me know

[TD-er]

Yep as soon as I found the best fix for ESPEasy, I will let you know.

Still it would also be nice if we didn't hit this limit and it is fixed in PlatformIO :)

[ivankravets]

Hi all,

Could you show an example of platformio.ini and explain what you would like to achieve? Maybe I can give you some advice. Thanks!

[TD-er]

Right now we do hit a limit in the longcmd file generated. So if the path to all .cpp.o files could be shortened, then the problem "magically" disappears (for a while)

On the other hand, if we could somehow determine what library is used, it does not have to be compiled (thus saving time and an entry in the longcmd file)

So there either needs to be a practical way to populate the lib_ignore from a Python script during build, or a list of files (not folders) that can be excluded/included using the same Python script route.

Edit: See my setup: https://github.com/letscontrolit/ESPEasy For example this line: https://github.com/letscontrolit/ESPEasy/blob/d0faa2dd4d5a00deac6583e63665d09c8d681fcf/platformio_esp82xx_envs.ini#L24 If you remove the IRremoteESP8266, HeatpumpIR no ESP8266 "custom" build can be built again.

[Jason2866]

"Golden solution" I would love if PlatformIO does a scan and compiles only the librarys used in the environment. Solves TDer issue and reduces compile time a lot in projects which needs many differents Libs (use of libs depends on user configurable settings) . https://github.com/arendst/Tasmota/blob/development/platformio.ini#L86 and https://github.com/arendst/Tasmota/blob/development/platformio_tasmota_env.ini#L48 is the not elegant way to reduce compile by using defined library groups More info https://github.com/arendst/Tasmota/blob/development/platformio_override_sample.ini#L78

[ivankravets]

lib_deps instructs PlatformIO to download this dependency, install on your machine, and include in ba build process. Do you mean that you want only "INSTALL" stage? And later just use lib_ldf_mode = chain+.

In this case, you need to do the next:

1. Do not use lib_deps if you don't need this library in a build process
2. Create PRE extra script and manage dependencies on a disk manually. See lines of code https://github.com/platformio/platformio-core/blob/develop/platformio/builder/tools/piolib.py#L885

lm = LibraryPackageManager(
            env.subst(os.path.join("$PROJECT_LIBDEPS_DIR", "$PIOENV"))
        )
for spec in custom_specs:
   if lm.get_package(spec):
      lm.install(spec)

custom_specs - could be an option from platformio.ini, for example, custom_lib_deps = .... Then use

env.GetProjectConfig().get(
                "env:" + self.env["PIOENV"], "custom_lib_deps"
            )

[Jason2866]

In Tasmota we do NOT use lib_deps all libs are local (adopt for Tasmota). We use already lib_ldf_mode=chain+ So the way you described is the way that fits Thx. It is on my todo list. Idk when i have time for. At the moment the way we do is good enough

[ivankravets]

If you use lib_ldf_mode=chain+ - how it is possible that PlatformIO builds libraries that you do not need?

[Jason2866]

Idk. Without the libs are missing. With it it compiles every lib found in folder if used in code or not It is not in the resulting firmware. Linker does his job.

[TD-er]

Even with deep+ the local libraries are still built. And the files all appear in the longcmd file. Only when ignoring the libs, the files are not built.

Just like Tasmota, the ESPEasy libs folder is filled with adapted versions of libraries. And I am considering to "misuse" the libs idea to convert all plugins of ESPEasy into "libraries" so you can exclude them from the build (meaning the number of libraries will then be 100 libraries more, so probably hit another limit?)

I'm not sure what config the ldf scanner uses to determine whether a library is needed? Does it process the #define checks?

[ivankravets]

Does it process the #define checks?

Yes, it follows macros but is not so smart. It does not have information about toolchain's macros. We do not use native GCC preprocessor and use soft version written in Python (part of SCons). We have a feature request to use native GCC Preprocessor but this way will significantly increase compilation time.

[Jason2866]

That explains it. We hope the feature request will be done (soon)😍 Thanks for your awesome work and your great support. Without PlatformIO Tasmota would not be possible!

[TD-er]

Would it be possible to make a filter on files to include/exclude in a build? For example, if I have a list of defines in Python like these:

if os.path.isfile('src/Custom.h'):
  env.Append(CPPDEFINES=["USE_CUSTOM_H"])
else:
  env.Append(CPPDEFINES=[
    "CONTROLLER_SET_ALL",
    "NOTIFIER_SET_NONE",
#    "PLUGIN_BUILD_NORMAL",
    "USES_P001",  # Switch
    "USES_P002",  # ADC
    "USES_P004",  # Dallas DS18b20
    "USES_P100",  # Pulse Counter - DS2423

    "USES_C016",  # Cache Controller
    "USES_C018",  # TTN/RN2483

    "FEATURE_MDNS",
    "FEATURE_SD",
    "FEATURE_I2CMULTIPLEXER",

    "USE_SETTINGS_ARCHIVE"
  ])

As you can see, there is quite a simple structure, which can also be found in the .cpp and .h files (at this moment still .ino files, but I can transform them into .h/.cpp files) My other idea was to convert them into libraries, which can be added to either lib_deps or lib_ignore. I can also create folders, but I'm not sure when I will hit another command line limit when basing a filter on source folders.

[ivankravets]

It's possible to tune your build process following macros which are visible to SCons. You can see them in pio run -v or pio run -t envdump and check flags.

Thank you too for using PlatformIO!

[ivankravets]

If you put dependency includes under #ifdef USES_P002 (for example), the LDF should understand this and ignore these includes.

[TD-er]

I think they are all used in such a structure. Only problem right now is that the plugins still are in .ino files, which may cause the LDF to find them, even though the includes that refer to the libraries are all within the #ifdef USES_Pxxx checks. Problem is, if I convert them to .cpp/.h too, I will very soon hit the limit of the longcmd again.

And about using PIO... I truly have no idea how to work without it anymore. Some users still want to build using Arduino IDE, but I really don't understand why...

[TD-er]

I am again running into this nasty command line limit and am a bit out of options now to think of new work arounds for it. The only thing I can now think of is creating a single large .cpp file of all separate .cpp files in a specific directory. This is of course not a desired fix.

Another thing I can do is:

• move the .cpp files to another directory away from the .h files
• Exclude that dir from building
• let the pre-build Python script concatenate all .cpp files to a single .cpp file and move that one to a dir where it can be built (to keep include paths working)

This way I still have the files separate so a change on a single file is still usable in Git blame.

[TD-er]

I tried to make some Python scripts to concatenate .cpp files in a specific directory and exclude the original dir. See https://github.com/letscontrolit/ESPEasy/pull/3409

Problem here is that the source filter not appears to be working to exclude the orginal .cpp files.

Import("env")
import os
import glob

cpp_files = []

cpp_path = './src/src/PluginStructs'

cpp_path_out = './src/src/PluginStructs_tmp'

if not os.path.exists(cpp_path_out):
    os.makedirs(cpp_path_out)

tmp_cpp_file = os.path.join(cpp_path_out, '__tmpfile.cpp')

if os.path.exists(tmp_cpp_file):
    os.remove(tmp_cpp_file)

for root, dirs, files in os.walk(cpp_path):
    for file in files:
        if file.endswith('.cpp'):
            fullname = os.path.join(root, file)
            cpp_files.append(fullname)
            print("Add {}".format(fullname))

with open(tmp_cpp_file,'wb') as newf:
    for filename in cpp_files:
        with open(filename,'rb') as hf:
            newf.write(hf.read())

I tried several filters for the src_filter, but it doesn't appear to be working.

Specifically excluding the files that are concatenated:

src_filter                = +<*> -<.git/> -<.svn/> -<example/> -<examples/> -<test/> -<tests/> -<P*_data_struct.cpp>

Or excluding the original path:

src_filter                = +<*> -<.git/> -<.svn/> -<example/> -<examples/> -<test/> -<tests/> -<src/src/PluginStructs/>

The output of the linker still suggests the old .cpp files are being compiled:

c:/users/gijsn/.platformio/packages/toolchain-xtensa@2.40802.200502/bin/../lib/gcc/xtensa-lx106-elf/4.8.2/../../../../xtensa-lx106-elf/bin/ld.exe: .pio/build/custom_ESP8266_4M1M/src/src/PluginStructs_tmp/__tmpfile.cpp.o: in function `P045_data_struct::loop()':
__tmpfile.cpp:(.text._ZN16P045_data_struct4loopEv+0x1c): multiple definition of `P045_data_struct::loop()'; .pio/build/custom_ESP8266_4M1M/src/src/PluginStructs/P045_data_struct.cpp.o:P045_data_struct.cpp:(.text._ZN16P045_data_struct4loopEv+0x1c): first defined here

@ivankravets I hope you can push me in the right direction here, to create a work-around for this issue.

[TD-er]

Ah.... this one appears to be working:

src_filter                = +<*> -<.git/> -<.svn/> -<example/> -<examples/> -<test/> -<tests/> -<*/PluginStructs/>

[TD-er]

OK, I now have managed to limit the size of the longcmd file by selectively concatenating .cpp files in a specific directory into a temp dir at the same level.

e.g. ./src/src/Commands/*.cpp => ./src/src/Commands_tmp/__tmpfile.cpp

Measures taken:

• Generate __tmpfile.cpp of selected directories
• Exclude selected directories via src_filter in platformio.ini
• Added newline after each file to prevent issues where a file ends with #endif without a newline
• Make sure all includes in a .cpp file to a .h file does use the parent dir of that .h file (e.g. Commands/Common.cpp includes its .h file like this: #include "../Commands/Common.h"
• Remove __tmpfile.cpp of selected directories as post action when a bin file has been created (so build errors can still be accessed inside the __tmpfile.cpp)

See: https://github.com/letscontrolit/ESPEasy/blob/3341bdf74427af9ffeb2522f92f33db1710e4351/tools/pio/concat_cpp_files.py https://github.com/letscontrolit/ESPEasy/blob/3341bdf74427af9ffeb2522f92f33db1710e4351/tools/pio/remove_concat_cpp_files.py https://github.com/letscontrolit/ESPEasy/blob/3341bdf74427af9ffeb2522f92f33db1710e4351/platformio.ini#L91

[TD-er]

I'm hitting these issues again and now I'm running out of my own files to concatenate in the Python script mentioned above. Right now I also tried to concatenate some library which does have quite a lot of .cpp files in it which makes it possible to compile on Windows again but since that library was never intended to have all .cpp files concatenated into a single large file I am running into other issues like 0 maintainability on upgrades of that library and linker issues on Linux.

I also found this PR, which should fix this issue as far as I could understand, but apparently it doesn't for me. (issue it tried to fix: https://github.com/platformio/platformio-core/issues/3792 )

But maybe it isn't the exact same problem as my linker error is:

xtensa-esp32-elf-g++: error: CreateProcess: No such file or directory

I really would like to see a proper fix as my patch to concatenate files in the build process does have some side-effects like Intellisense often pointing me to the wrong file and I only find out I was editing the wrong file after all my edits were lost when building. (happened a lot, really frustrating)

So please, can we somehow make a proper fix for this? Or is there already one magic project setting which I simply overlooked?

[Jason2866]

Probably this will be the solution when the toolchain is available and the option used. https://github.com/espressif/esp-idf/issues/7864 7 To have this change in esp8266 toolchain the maintainer needs to be contacted. https://github.com/earlephilhower/esp-quick-toolchain

[TD-er]

That would be great if it is fixed in the tool chain. Right now it is mainly a problem for me with ESP32 builds, but it is just a matter of time before it will happen again on ESP8266.