Espressif arduino framework gets rebuild every time a source file is added to project

[Bascy]

What kind of issue is this?

• Development Platform or Board.

Using PlatformIO Core, version 6.0.0a2

When developing a project for esp32dev board, platform arduino, everything including all the framework files get rebuild every time you add a new source file to the project. This seems unneeded.

Side note: I have seen this also happening when just modifying project files, or files from dependent packages in./pio/libdeps but I can;t seem to get this reproduced in a small project that I can share here

platformio.ini

    [env:esp32dev]
    platform = espressif32@^4.0.0
    board = esp32dev
    framework = arduino

How to reproduce

• Create a new project
• build
• change main.cpp and build, notice that only main.cpp is rebuild
• add an empty test.cpp file
• build, notice all files are rebuild (see logging beneath)
• Remove test.cpp from project, rebuild. All files are rebuild, including framework which is not needed

> Executing task in folder TestBuild: C:\Users\Bas\.platformio\penv\Scripts\platformio.exe run <

Processing esp32dev (platform: espressif32@^4.0.0; board: esp32dev; framework: arduino)
--------------------------------------------------------------------------------------------------------------
Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/espressif32/esp32dev.html
PLATFORM: Espressif 32 (4.1.0) > Espressif ESP32 Dev Module
HARDWARE: ESP32 240MHz, 320KB RAM, 4MB Flash
DEBUG: Current (esp-prog) External (esp-prog, iot-bus-jtag, jlink, minimodule, olimex-arm-usb-ocd, olimex-arm-usb-ocd-h, olimex-arm-usb-tiny-h, olimex-jtag-tiny, tumpa)
PACKAGES:
 - framework-arduinoespressif32 @ 3.20001.0 (2.0.1) 
 - tool-esptoolpy @ 1.30100.210531 (3.1.0)
 - toolchain-xtensa-esp32 @ 8.4.0+2021r2-patch3
LDF: Library Dependency Finder -> https://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ soft
Found 31 compatible libraries
Scanning dependencies...
No dependencies
Building in release mode
Compiling .pio\build\esp32dev\src\main.cpp.o
Compiling .pio\build\esp32dev\src\test.cpp.o
Generating partitions .pio\build\esp32dev\partitions.bin
Archiving .pio\build\esp32dev\libFrameworkArduinoVariant.a
Compiling .pio\build\esp32dev\FrameworkArduino\Esp.cpp.o
Compiling .pio\build\esp32dev\FrameworkArduino\FirmwareMSC.cpp.o
Compiling .pio\build\esp32dev\FrameworkArduino\FunctionalInterrupt.cpp.o
Compiling .pio\build\esp32dev\FrameworkArduino\HWCDC.cpp.o
Compiling .pio\build\esp32dev\FrameworkArduino\HardwareSerial.cpp.o
Compiling .pio\build\esp32dev\FrameworkArduino\IPAddress.cpp.o
Compiling .pio\build\esp32dev\FrameworkArduino\IPv6Address.cpp.o
Compiling .pio\build\esp32dev\FrameworkArduino\MD5Builder.cpp.o
Compiling .pio\build\esp32dev\FrameworkArduino\Print.cpp.o
Compiling .pio\build\esp32dev\FrameworkArduino\Stream.cpp.o
Compiling .pio\build\esp32dev\FrameworkArduino\StreamString.cpp.o
Compiling .pio\build\esp32dev\FrameworkArduino\USB.cpp.o
Compiling .pio\build\esp32dev\FrameworkArduino\USBCDC.cpp.o
Compiling .pio\build\esp32dev\FrameworkArduino\USBMSC.cpp.o
Compiling .pio\build\esp32dev\FrameworkArduino\WMath.cpp.o
Compiling .pio\build\esp32dev\FrameworkArduino\WString.cpp.o
Compiling .pio\build\esp32dev\FrameworkArduino\base64.cpp.o
Compiling .pio\build\esp32dev\FrameworkArduino\cbuf.cpp.o
Compiling .pio\build\esp32dev\FrameworkArduino\esp32-hal-adc.c.o
Compiling .pio\build\esp32dev\FrameworkArduino\esp32-hal-bt.c.o
Compiling .pio\build\esp32dev\FrameworkArduino\esp32-hal-cpu.c.o
Compiling .pio\build\esp32dev\FrameworkArduino\esp32-hal-dac.c.o
Compiling .pio\build\esp32dev\FrameworkArduino\esp32-hal-gpio.c.o
Compiling .pio\build\esp32dev\FrameworkArduino\esp32-hal-i2c-slave.c.o
Compiling .pio\build\esp32dev\FrameworkArduino\esp32-hal-i2c.c.o
Compiling .pio\build\esp32dev\FrameworkArduino\esp32-hal-ledc.c.o
Compiling .pio\build\esp32dev\FrameworkArduino\esp32-hal-matrix.c.o
Compiling .pio\build\esp32dev\FrameworkArduino\esp32-hal-misc.c.o
Compiling .pio\build\esp32dev\FrameworkArduino\esp32-hal-psram.c.o
Compiling .pio\build\esp32dev\FrameworkArduino\esp32-hal-rmt.c.o
Compiling .pio\build\esp32dev\FrameworkArduino\esp32-hal-sigmadelta.c.o
Compiling .pio\build\esp32dev\FrameworkArduino\esp32-hal-spi.c.o
Compiling .pio\build\esp32dev\FrameworkArduino\esp32-hal-time.c.o
Compiling .pio\build\esp32dev\FrameworkArduino\esp32-hal-timer.c.o
Compiling .pio\build\esp32dev\FrameworkArduino\esp32-hal-tinyusb.c.o
Compiling .pio\build\esp32dev\FrameworkArduino\esp32-hal-touch.c.o
Compiling .pio\build\esp32dev\FrameworkArduino\esp32-hal-uart.c.o
Compiling .pio\build\esp32dev\FrameworkArduino\firmware_msc_fat.c.o
Compiling .pio\build\esp32dev\FrameworkArduino\libb64\cdecode.c.o
Compiling .pio\build\esp32dev\FrameworkArduino\libb64\cencode.c.o
Compiling .pio\build\esp32dev\FrameworkArduino\main.cpp.o
Compiling .pio\build\esp32dev\FrameworkArduino\stdlib_noniso.c.o
Compiling .pio\build\esp32dev\FrameworkArduino\wiring_pulse.c.o
Compiling .pio\build\esp32dev\FrameworkArduino\wiring_shift.c.o
Archiving .pio\build\esp32dev\libFrameworkArduino.a
Linking .pio\build\esp32dev\firmware.elf
Retrieving maximum program size .pio\build\esp32dev\firmware.elf
Checking size .pio\build\esp32dev\firmware.elf
Advanced Memory Usage is available via "PlatformIO Home > Project Inspect"
RAM:   [          ]   4.9% (used 16140 bytes from 327680 bytes)
Flash: [==        ]  15.5% (used 203797 bytes from 1310720 bytes)
Building .pio\build\esp32dev\firmware.bin
esptool.py v3.1
Merged 2 ELF sections
======================================== [SUCCESS] Took 13.65 seconds ========================================
Terminal will be reused by tasks, press any key to close it.

[ivankravets]

This is normal behavior. If you change the project file structure, PlatformIO deletes build caches. The issue might arise if you add or remove header files and compilation logic changes (macros inside headers are changed, etc).

[Bascy]

Doesn't it compare source and object file timestamps? Or include of changed files? It looks to me like this will rebuild a lot of unneeded files (It takes almost 3 minutes to build all files in my project ...)

[ivankravets]

It compares a project's file structure, not file modifications. So, if you move files and add new directories, PlatformIO cleans caches.

Do you often delete/add files?

[ivankravets]

If you frequently delete/add files in your project, this option will help https://docs.platformio.org/en/latest/core/userguide/cmd_run.html#cmdoption-pio-run-disable-auto-clean

[Bascy]

No not that often but i'm trying to figure out why its recompiling the whole project so often in my situation, Also it has happend regularly that after a succesfull build, I start build and upload and that again starts building all files again

[Bascy]

Does the lib_ldf_mode have any impact on the amount of files recompiled?

[ivankravets]

If you modify the project structure all caches are deleted, including compiled library dependencies.