Speeding up the build time

[Gregwar]

Hello,

Building time (pio run) with platformio and mbed is really long, even with minor changes in application files (and no change in mbed itself). I am now using a workaround involving tweaking scons itself to ignore the check of changes on mbed files themselves and is explained here: https://gregwar.github.io/speedup-pio-mbed-build-and-dfu-util#speed-up-the-build-time-pio-run-w-mbed

This is really dirty but speeds up the build time by a great factor

I think it might be possible to have a faster build with other methods, one of them would be building a separate library for mbed and either build it or just use it if it exists without asking scons to scan all the files (which takes like 20s on my computer)

[ivankravets]

Hi @Gregwar,

Sorry for the issue. We got similar reports and can not reproduce this issue on our side.

1. What is your OS?
2. Could you provide a remote session to debug this issue on your side under your control?

/cc @valeros

[Gregwar]

I am giving lecture with students on various systems and they all experience the same issue

It is actually easy to reproduce:

# Cloning a minimal example
git clone https://github.com/Gregwar/mbed_debug.git
cd mbed_debug
pio run  # Doing an initial build
time pio run # The time it takes to do pio run doing nothing on my computer: 17.8s
# The same issue apply if I just do minor change in the app code

[Gregwar]

As far as I understood in my investigations in scons, almost 100% of the CPU time is spent scanning for the mbed framework headers and source files changes, while we know for sure they didn't change because we only have to build mbed once (if I change some files in src/ it should not impact it)

It could be a solution based on an explicit user flag like -t nobuild_mbed

[ivankravets]

Could provide an output of 2nd attempt to run pio run?

/t/mbed_debug (master|✔) $ pio run
Processing debug_mbed (board: nucleo_f446re; platform: ststm32; framework: mbed)
----------------------------------------------------------
Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/ststm32/nucleo_f446re.html
PLATFORM: ST STM32 (11.0.0) > ST Nucleo F446RE
HARDWARE: STM32F446RET6 180MHz, 128KB RAM, 512KB Flash
DEBUG: Current (stlink) On-board (stlink) External (blackmagic, cmsis-dap, jlink)
PACKAGES:
 - framework-mbed 6.60600.201227 (6.6.0)
 - toolchain-gccarmnoneeabi 1.90201.191206 (9.2.1)
Collecting mbed sources...
LDF: Library Dependency Finder -> http://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ soft
Found 0 compatible libraries
Scanning dependencies...
No dependencies
Building in release mode
Checking size /private/tmp/pio-workspaces/mbed_debug-9080d73fbe/build/debug_mbed/firmware.elf
Advanced Memory Usage is available via "PlatformIO Home > Project Inspect"
RAM:   [=         ]   7.4% (used 9640 bytes from 131072 bytes)
Flash: [=         ]   6.1% (used 32116 bytes from 524288 bytes)
====================== [SUCCESS] Took 7.69 seconds =======================

It took me 8 secs. But, my machine is very fast.

If you don't have SSD, then 17 secs is OK to check if 650Mb of files are modified.

[Gregwar]

It took me 8 secs. But, my machine is very fast.

If you don't have SSD, then 17 secs is OK to check if 650Mb of files are modified.

Your PC is fast, mine is slower, and my students are even slower (it takes minutes to wait on some of them, making pio+mbed almost not practicable) The point is that we could avoid checking if those 650Mb of files were modified when we are focusing on a few bytes in the src/ folder (but I understand it might not be easy to implement)

(With my hack, see blog post above, it takes only 2s instead of 17s)

[ivankravets]

Have you tried https://docs.platformio.org/en/latest/projectconf/section_platformio.html#build-cache-dir

[platformio]
; Set a path to a cache folder
build_cache_dir = .cache

[Gregwar]

If I understand well, this cache directory allows you to share object files across projects to avoid re-building them But yet the files are scanned anyway and this is what is taking time (because nothing new is actually built), so this option appear to have no effect for me

[ivankravets]

Thanks, we agree with you and will investigate how to disable this modification checking for frameworks.

[ivankravets]

@Gregwar could you help to investigate this issue? There is Decider in SCons and I tried to use it but don't see significant improvement.

def _decider(*args, **kwargs):
    return False

env.Decider(_decider)

You can use it with PRE script https://docs.platformio.org/en/latest/projectconf/advanced_scripting.html

[Gregwar]

My hunch would be:

1. Make framework mbed a separate static library target (like libFrameworkMbed.a)
2. When compiling, link user code against that librart
3. Give the user option so that if .a already exists we don't build it again but just link against it

But I have little knowledge of scons and I am not sure how to achieve that

[Copper-Bot]

Hello @Gregwar,

Another solution is to apply a .mbedignore file into PIO MBED framework folder. This will speed up MBED build process, to avoid compiling unnecessary modules.

A community script already exists, you can have detailed instruction here to how to set up the mbedignore script.

If students need to debug too, you can add build_type = debug in your platformio.ini file, to avoid rebuild all sources between debug mode and a normal build.

[zackees]

I'm getting the same issue. I'm trying to improve the build times for FastLED. I'm seeing the same files get compiled over and over again. I'm using --keep-build-dir but it doesn't help.

We have a build matrix of five boards times nineteen tests. It currently takes 7 minutes to run through the whole test suite. And this speed is by telling pio to build each ino for all board variants in one command. It would be nice if we could at least get pio to build multiple *.ino using each board, or have the build artifacts re-used for subsequent builds.

[valeros]

Hi @zackees, it seems you writing in a wrong repository. Anyway, I guess the build_cache_dir option is what you're looking for as it allows you to share build cache between environments. It's also available as an environment variable PLATFORMIO_BUILD_CACHE_DIR, so should be easy to use in your CI environment.