SuperBoss9 commented 4 years ago

Hello everyone!

The problem is in mbed support in PIO. If I enable RTOS support via recommended macro then PIO recompiles everything that is in MBED framework (all features, all drives and everything). This operation takes a lot of resources even at a very powerful PC. If PIO recompiles everything only once it is possible to survive. But it recompiles everything at the following situations:

I add some more include predicates into my code.
I'd like to debug.

That makes work non possible because every compilation takes from 400 (i7) sec to 20 minutes (i3).

One more thing that mbed supports called .mbedignore that can switch off non necessary features is not supported in PIO.

pfeerick commented 4 years ago

valeros commented 4 years ago

Hi @SuperBoss9 ! Thanks for reporting. I'm afraid that's default build logic for mbed framework, all suitable source files are collected based on the target configuration. Don't mbed tools (mbed-cli or mbed IDE) also compile the entire framework?

What do you mean? The project will be rebuilt if your platformio.ini file is changed, no workaround here. If you add includes only to individual files, only they should be recompiled.
Since the debug session requires additional debug flags, the entire project will be recompiled.

Regarding .mbedignore file, it should work with PlatformIO as well. Are you sure you're correctly using it? How did you specify paths to the folders you want to ignore? Please share here contents of this file and the location where you placed it. Thanks!

SuperBoss9 commented 4 years ago

Hi @SuperBoss9 ! Thanks for reporting. I'm afraid that's default build logic for mbed framework, all suitable source files are collected based on the target configuration. Don't mbed tools (mbed-cli or mbed IDE) also compile the entire framework?

What do you mean? The project will be rebuilt if your platformio.ini file is changed, no workaround here. If you add includes only to individual files, only they should be recompiled.

Since the debug session requires additional debug flags, the entire project will be recompiled.

Regarding .mbedignore file, it should work with PlatformIO as well. Are you sure you're correctly using it? How did you specify paths to the folders you want to ignore? Please share here contents of this file and the location where you placed it.

Hello valeros! Regarding to your questions:

If I include an include then the whole set of libs and features are rebuild. It shouldn't be so.
Mbed studio doesn't rebuild mbed core files if I'd like to debug. PIO does.

PIO with VScode is a great thing but it is not usable with mbed RTOS functions due to the compilation issue. Mbed Studio doesn't recompile everything until I ask it to do that clearly. Moreover I can use RTOS functionality with Mbed Studio just adding thread, ticker or other includes. But I need to use special macros with PIO in order to be able to include these functionality.

Regarding to .mbedignore it works with mbed Studio, where I can just put the file into a project root dir. In PIO I tested any projects dirs without any effect (and with any combinations of pathes).

mbed ignore: mbed-os/features/lorawan/ mbed-os/features/nfc/ mbed-os/features/cryptocell/*

Unfortunately I won't be able to provide PIO .mbedignore example until end of this week.

valeros commented 4 years ago

If I include an include then the whole set of libs and features are rebuild. It shouldn't be so.

Couldn't reproduce this behavior. In my case PlatformIO recompiles only changed files.

Moreover I can use RTOS functionality with Mbed Studio just adding thread, ticker or other includes. But I need to use special macros with PIO in order to be able to include these functionality.

The reason why you need to specify a special flag is because we support both variations of mbed framework: mbed2 (framework without rtos library) and mbed-os. If you don't switch between mbed2 and mbed-os what is the problem to specify -DPIO_FRAMEWORK_MBED_RTOS_PRESENT once?

I'm not familiar with Mbed studio, but you can always build your project in debug configuration or even start the debug session without compiling your project?

mbed ignore: mbed-os/features/lorawan/ mbed-os/features/nfc/ mbed-os/features/cryptocell/*

Where did you place this file? The paths are relative and we don't even have mbed-os folder in our framework package, so this won't work with PlatformIO as in our case framework is located in a special folder.

SuperBoss9 commented 4 years ago

If I include an include then the whole set of libs and features are rebuild. It shouldn't be so.

Couldn't reproduce this behavior. In my case PlatformIO recompiles only changed files.

Moreover I can use RTOS functionality with Mbed Studio just adding thread, ticker or other includes. But I need to use special macros with PIO in order to be able to include these functionality.

The reason why you need to specify a special flag is because we support both variations of mbed framework: mbed2 (framework without rtos library) and mbed-os. If you don't switch between mbed2 and mbed-os what is the problem to specify -DPIO_FRAMEWORK_MBED_RTOS_PRESENT once?

I've double check the behaviour at my i7:

1st compiling with RTOS enabled: [SUCCESS] Took 136.24 seconds

2st compiling with RTOS enables [SUCCESS] Took 31.22 seconds

3rd compiling with RTOS enabled and added include of my own class located in the project [SUCCESS] Took 130.97 seconds

I didn't change platformio.ini just add include of my own class to main and that caused mandatory clean build.

I'm not familiar with Mbed studio, but you can always build your project in debug configuration or even start the debug session without compiling your project?

mbed ignore: mbed-os/features/lorawan/ mbed-os/features/nfc/ mbed-os/features/cryptocell/*

Where did you place this file? The paths are relative and we don't even have mbed-os folder in our framework package, so this won't work with PlatformIO as in our case framework is located in a special folder.

.mbedignore from above is from Mbed studio project where it works well. Closer to the end of the week I'll post the file I used in PIO (tryed to put it at any folder I have in my project). But anyway .mbedignore is not a complete workaround: it is not possible to switch off all the features due to dependences between them.

PS. I've checked mbed versions: PIO: PACKAGES: toolchain-gccarmnoneeabi 1.70201.0 (7.2.1), framework-mbed 5.51204.190701 (5.12.4) MBED Studio: Running mbed-os 5.14.0

pfeerick commented 4 years ago

.mbedignore from above is from Mbed studio project where it works well.

That might be the case, but since the paths used are relative (not absolute), if not placed relative to the mbed-os folder, it's not going to work, is it? And as valeros pointed out that there is no mbed-os folder in the platformio framework... so the likelihood of it working are ... zilch, nada, nope.

I noticed in another forum thread someone suggested that upgrading from python 2.7 to python 3 fixed what they referred to as 'random rebuilds'... maybe worth checking if that's the culprit? Maybe it was just a coincidence?

valeros commented 4 years ago

@SuperBoss9 Could you please provide a minimal project to reproduce the spurious recompilation?

SuperBoss9 commented 4 years ago

@SuperBoss9 Could you please provide a minimal project to reproduce the spurious recompilation?

Yes, sure https://github.com/SuperBoss9/MBEDRecompileIssue.git

It seems that I've found the exact place of the issue - adding a new class/.h via include forces recompilation of everything.

Recompilation 1

include

//#include //#include

Recompilation 2

include

//#include

Recompilation 3

include

//#include

include

No Recompilation

include

//#include

valeros commented 4 years ago

It seems that I've found the exact place of the issue - adding a new class/.h via include forces recompilation of everything.

I tried your project (Python 2&3) and didn't notice this issue. What is your OS, version of Python, version of PlatformIO?

SuperBoss9 commented 4 years ago

It seems that I've found the exact place of the issue - adding a new class/.h via include forces recompilation of everything.

I tried your project (Python 2&3) and didn't notice this issue. What is your OS, version of Python, version of PlatformIO?

I have the same issue at at least 3 of my computers. All of them under Win10Prox64 PIO Home 2.3.3·Core 4.0.3 (PIO IDE with VSCode) Python Python 2.7.16 (v2.7.16:413a49145e, Mar 4 2019, 01:30:55) [MSC v.1500 32 bit (Intel)] on win32

valeros commented 4 years ago

Compiled your project on 2 different machines with Win10 (Python 2&3) and still can't reproduce the issue. Could you try another python version?

SuperBoss9 commented 4 years ago

I think that you need to create a fresh class after 1st build. Here is a video of the process https://youtu.be/_Gl8Fm_d7WE

Also will try the same at my other 2 PCs. Check Python version.

valeros commented 4 years ago

@SuperBoss9 Thanks for the video, it helped me understand what we are talking about. So when you add new file it rebuilds the whole environment and at the moment this is expected behavior. As a workaround your can try build-cache-dir option, it should help you avoid recompilation of the entire framework.

SuperBoss9 commented 4 years ago

Thank you

DesktopMan commented 4 years ago

If I build my project, then click build again without changing anything that second build randomly takes either 15 seconds (OK) or 3 minutes (not OK).

This can't possibly be expected behavior? Core 4.3.1, Home 3.1.1, ST STM32 6.0.0.

[env:disco_l475vg_iot01a] platform = ststm32 board = disco_l475vg_iot01a framework = mbed build_flags = -D PIO_FRAMEWORK_MBED_RTOS_PRESENT

This is with the rtos_basic example.

valeros commented 4 years ago

Hi @DesktopMan ! Could you please try to run pio platform update ststm32 and try to compile your project a few times in a row? Does it work better now?

DesktopMan commented 4 years ago

I did a fresh Windows install and all the tools in a virtual machine to make sure the test is correct. Building the empty Mbed project twice takes 20 seconds each time, even without changes. It compiles the HAL and platform source every time.

If I enable the OS with _build_flags = -D PIO_FRAMEWORK_MBED_RTOSPRESENT the initial build took 5 minutes and the next two 6 minutes, so same result there.

Since this happens on a clean install and doesn't require hardware to test it should be possible to replicate.

Mixerito commented 4 years ago

This issue should be reopened, since the problem is not solved by any of proposed workarounds.

build_cache_dir does not help.

.mbedignore could help to speed up the build as it could at least omit unused components and features (still does not help with repeatedly built components that we use and were not changed) BUT we are not able to ignore some components for one project and use them in another project, since the .mbedignore file must be placed inside ".platformio/packages/framework-mbed/" folder. Still unresolved issue: https://github.com/platformio/builder-framework-mbed/issues/13

With _-D PIO_FRAMEWORK_MBED_RTOSPRESENT every build (without any change) compiles all mbed libraries.

It takes :

cca 4 minutes in Win7/VSCode/Platformio/mbed
cca 2 minutes in Win7/CLion/Platformio/mbed

The development with these timing constraints is very uneffective.

Mixerito commented 3 years ago

With a big help of Valeros, I found a problem in my project.

Firstly in file c:\Users\YOUR_USER.platformio\penv\Lib\site-packages\platformio\platform_run.py (Win) or /home/YOUR_USER/.platformio/penv/Lib/site-packages/platformio/platform/_run.py (Linux) activate the --debug=explain option.

        args = [
            proc.get_pythonexe_path(),
            script_path,
            "-Q",      
            "--debug=explain",
            "--warn=no-no-parallel-support",
            "--jobs",
            str(jobs),
            "--sconstruct",
            os.path.join(fs.get_source_dir(), "builder", "main.py"),
        ]

This will reveal you the reason of recompile (during build).

In my case it was:

rebuilding because the contents of the build action changed

There are some reasons for build action changed:

some .py pre-scripts modifying the build flags or include libs
some dynamic directives in platformio.ini build_flags (my case: -D COMPILE_TIME=$UNIX_TIME)

Hope this helps someone

platformio / platform-ststm32

PIO compiles everything in MBED framework when RTOS is enabled #288

include

include

include

include

include

include

include