search

ARMmbed / mbed-os

Arm Mbed OS is a platform operating system designed for the internet of things
https://mbed.com
Other
4.67k stars 2.98k forks source link

mbed compile rebuilds from scratch #10564

Closed cederom closed 4 years ago

cederom commented 5 years ago

Hello world, I noticed that mbedOS 5.12 on my macOS using VirtualEnv Python3.7 and ARM_GCC very very often rebuilds the whole source tree from scratch, what takes really lots of time.. is this only my problem here?

cederom commented 5 years ago

This is really annoying and waste of time :-( What could be the problem that 90% of cases mbed compile rebuilds all mbed-os from scratch???

ciarmcom commented 5 years ago

Internal Jira reference: https://jira.arm.com/browse/MBOCUSTRIA-1191

cederom commented 5 years ago

Thank you for looking into the issue :-) I am working on macOS now with APFS.. only minor modification to my application source code (not touching the mbed-os/ at all) results in full OS rebuild. I have custom target defined in custom_targets.json maybe that is somehow related?

cederom commented 5 years ago

Hmm, that one may be related with how Target is treated. My Custom_Target inherits from some generic target that is also placed in custom_targets.json (https://github.com/ARMmbed/mbed-os/issues/10562). When chaning that parent to public I can now rebuild only parts of changes code on 5.10.4. On 5.12.3 I have to rebuild everything each time :-(

I still get that warning on each build start:

WARNING: Custom target "MCU_NRF52832" cannot replace existing target.
cederom commented 5 years ago

Really I have no idea and cannot find a pattern why this happens.

Another scenario: I am changing a line of my firmware code, mbed compile gets only into that changed file, finds an error, I am fixing the error, then compile again, I am expecting the one file compilation and linking.. then out of nowhere full mbedOS build happens.

This takes really tremendous amount of time. Especially if I need to finish tomorrow :-P

40Grit commented 5 years ago

@cederom we see and have seen weird stuff with builds as well and are currently using custom_target.json

@trowbridgec was this one we ran into?

cederom commented 5 years ago

Thank you @40Grit, not really happy to have the problem, but glad I am not alone.. I suspected there may be something wrong on my side..

microbit-carlos commented 5 years ago

It's not the same problem, as after I build a project the first time it then correctly recompiles only the files I've touched, but maybe this could still be related? https://github.com/ARMmbed/mbed-cli/issues/894 (from a clean blinky project the first mbed compile takes an hour because it builds all of Mbed OS)

cederom commented 5 years ago

Thanks @microbit-carlos but all releases after 5.9.7 [1] for me rebuilds that whole mbed-os almost every time I change a line of code in my project not only the first initial build.. thus I am working on 5.9.7 right now.

[1] https://github.com/ARMmbed/mbed-os/releases/tag/mbed-os-5.9.7

cederom commented 5 years ago

There are some bugs in 5.9.7, so I have to move to 5.12.3 to test against, but 5.9.7 is also the last one that did not rebuild the whole mbed-os from scratch each time. This is kind of blocker for me.. as I have to wait severa minutes on each change.. and this happens very very often :-(

I guess it would be safer to assume mbed-os is untouched in most cases, so user only needs to build it once, rather than building it each time..?

40Grit commented 5 years ago

@theotherjimmy

Any insight?

0xc0170 commented 5 years ago

@ARMmbed/mbed-os-tools Please review

mark-edgeworth commented 5 years ago

Hi, thanks for raising this. I'll let you know if I can find a solution...

cederom commented 5 years ago

Hello and thank you @mark-edgeworth @0xc0170 @40Grit :-) I thought that maybe my Custom Target configuration may be invalid and cause of the problem, but I have updated to 5.12.4 and used bare nRF52_DK target, and problem did not disappear, I still have to rebuild whole mbed-os from scratch most of the time. Any hints welcome :-)

40Grit commented 5 years ago

@aglass0fmilk @farrenv @trowbridgec

I know for a bit this was something we just dealt with.

Our CI currently rebuilds everytime since we have cacheing disabled, but do we see similar behavior in local builds?

trowbridgec commented 5 years ago

We actually don't have caching disabled, but we have to rebuild every time because we use the Mbed Client CLI which accepts a custom command line boot message via a MACRO. We generate this MACRO at compile time with the git branch, git commit, and build time/date and pass it in via a command line argument to mbed compile. When that happens, the compiler is forced to rebuild everything...

TeroJaasko commented 5 years ago

As a workaround, one can use ccache, which reduces even the forced full rebuild (started with -c switch on mbed-cli) times by 75% on my machine. Trick is that one needs to set MBED_BUILD_TIMESTAMP to a constant value, so the input macros do not change on each build.

There is a documentation request on this issue at https://github.com/ARMmbed/mbed-os/issues/10228

ciarmcom commented 4 years ago

Thank you for raising this detailed GitHub issue. I am now notifying our internal issue triagers. Internal Jira reference: https://jira.arm.com/browse/IOTOSM-2300

cederom commented 4 years ago

I am working on FreeBSD 5.15 MBED and latest MED-CLI the problem does not occur here / anymore.. will reopen when occur again.