How to compile only changed modules?

[kevinresol]

I am try to debug and add features to the firmware (dev-esp32 branch). But using docker to compile it takes like 30+ minutes. I would like know if there is a way to only compile changed source code and reuse other compiled objects to link a new binary?

[marcelstoer]

There should be I suspect...However, I've got no clue about make or the the firmware build internals.

For the ESP8266 build @nwf contributed a ccache integration in #28. Maybe something similar would be possible for the ESP32 build.

Currently the build script just calls make w/o any arguments: https://github.com/marcelstoer/docker-nodemcu-build/blob/master/build-esp32#L64

[kevinresol]

Thanks and I have inspected the build-esp32 script, by appending $@ to make at least I can pass argument to the make call. But the firmware's makefile itself is pretty convoluted, I am not sure how to achieve cache there. For now I run build-esp32 component-modules-build (replace "modules" with other component name if needed), then rm build/NodeMCU.* then build-esp32 `pwd`/build/NodeMCU.bin to shorten the build. Not sure if that is correct though.

[marcelstoer]

@devsaurus how do you use this in your environment?

[devsaurus]

To build local changes I just call make from bash which typically compiles only the modified source files, creates intermediate libraries, and links the bin files. I tried with the docker image and found that the build-esp32 script modifies user_version.h each time it is invoked. This triggers re-compilation of source files in components/lua and components/module which adds some overhead.

The bare make can be used in docker as well:

1. Launch a shell in the docker image: docker run --rm -ti -v `pwd`:/opt/nodemcu-firmware marcelstoer/nodemcu-build bash
2. Inside this docker shell install the python add-ons manually: /usr/bin/python -m pip install --user -r /opt/nodemcu-firmware/sdk/esp32-esp-idf/requirements.txt
3. Modify a source file, e.g. components/modules/gpio.c
4. Start the compilation inside the docker shell: make
5. Optionally generate the bin file manually inside the docker shell: srec_cat -output nodemcu_debug.bin -binary build/bootloader/bootloader.bin -binary -offset 0x1000 -fill 0xff 0x0000 0x8000 build/partitions_singleapp.bin -binary -offset 0x8000 -fill 0xff 0x8000 0x10000 build/NodeMCU.bin -binary -offset 0x10000

Step 4 just does the essential things:

Setting IDF_PATH and re-invoking...
make[1]: Entering directory '/opt/nodemcu-firmware'
Toolchain path: /opt/nodemcu-firmware/tools/toolchains/esp32/bin//xtensa-esp32-elf-gcc
Toolchain version: crosstool-ng-1.22.0-80-g6c4433a5
Compiler version: 5.2.0
make[2]: Entering directory '/opt/nodemcu-firmware/sdk/esp32-esp-idf/components/bootloader/subproject'
make[2]: Leaving directory '/opt/nodemcu-firmware/sdk/esp32-esp-idf/components/bootloader/subproject'
Python requirements from /opt/nodemcu-firmware/sdk/esp32-esp-idf/requirements.txt are satisfied.
make[2]: Entering directory '/opt/nodemcu-firmware/build/modules'
CC build/modules/gpio.o
AR build/modules/libmodules.a
make[2]: Leaving directory '/opt/nodemcu-firmware/build/modules'
LD build/NodeMCU.elf
...
make[1]: Leaving directory '/opt/nodemcu-firmware'

You can repeat steps 3 to 5 as often as needed.

Maybe this speeds up things.

[kevinresol]

Thanks for the detailed explanations!

[marcelstoer]

I'd like to keep this open for now; it should be addressed.

the build-esp32 script modifies user_version.h each time it is invoked. This triggers re-compilation of source files in components/lua and components/module

That's correct. We do this to produce (more) meaningful definitions such as

#define NODE_VERSION    "NodeMCU ESP32" " built with Docker provided by frightanic.com\n\tbranch: dev-esp32\n\tcommit: e38fc7ac66ab979690e10890231804d653aa7294\n\tSSL: true\n\tmodules: crypto,encoder,file,gpio,http,i2c,net,node,ow,tmr,wifi\n"
#ifndef BUILD_DATE
#define BUILD_DATE    "created on 2019-01-23 21:53\n"
#endif

There's still a comment at https://github.com/marcelstoer/docker-nodemcu-build/blob/master/build-esp8266#L80 which shows that I once tried to use EXTRA_CCFLAGS to avoid altering that file - I failed miserably 😞

Do you see any way around that? Other things to explore in order to fix this issue properly?

[devsaurus]

Do you see any way around that?

Since make decides for rebuild based on the files' timestamps, you'd have to avoid touching user_version.h if its content won't change. E.g.

1. Generate a temporary file based on required logic.
2. Compare this with the real user_version.h
3. If content differs: overwrite user_version.h
4. If content is the same: don't touch user_version.h

Just a random idea, didn't try it actually myself.

[marcelstoer]

Due to BUILD_DATE user_version.h will always be different ☹️

Since make decides for rebuild based on the files' timestamps

It would help if I could tell make to ignore certain files. Sounds like something that should be a standard feature - will investigate.

[nwf]

@marcelstoer Please don't circumvent make's already quite fragile dependency tracking. Better to make the script not set BUILD_DATE, IMHO.

[marcelstoer]

@nwf thanks for the warning! As I said, I don't know make at all and any piece of advice is certainly appreciated.

I find that having all the information I pack into NODE_VERSION a tremendous improvement.

Looks like I need to make another attempt at setting those variables through EXTRA_CCFLAGS rather than user_version.h. Sigh...

[devsaurus]

Looks like I need to make another attempt at setting those variables through EXTRA_CCFLAGS rather than user_version.h

Hm, maybe the whole escaping can be isolated to a central instance with

EXTRA_CFLAGS="-DBUILD_DATE=\"\`cat `pwd`/builddatefile\`\"" make

Works for me with mildly complex content in builddate:

"some 'strange' string\n\ttab's are ok\n\t[hyphens] need escapes\""

Just figured that components/lua/lua.h is also tweaked. This one actually triggers rebuild of modules and lua sources, not user_version.h.

[devsaurus]

And while we're at it - Recompilation of some source files due to volatile dependencies isn't actually a big deal on my machine here. More time is sunken with pip install, making each re-run painfully slow with bad internet connection. The python update appears to be volatile and doesn't remain for subsequent runs. Is this a characteristic of Docker containers?

[marcelstoer]

Just figured that components/lua/lua.h is also tweaked. This one actually triggers rebuild of modules and lua sources, not user_version.h.

Yeah, that one can easily be changed i.e. I can place the ESP-IDF version somewhere else.

The python update appears to be volatile and doesn't remain for subsequent runs. Is this a characteristic of Docker containers?

Yes, containers are immutable by default. If you want to persist data generated during operations running inside the container you can mount a host system volume into the container. That's what

-v `pwd`:/opt/nodemcu-firmware

upon container start does (host-path:container-path).

The ESP-IDF Python requirements are, well, dependent on the ESP-IDF version and can therefore not be packaged into the Docker image. Everything else comes with the image as I declare it to be baked into it: https://github.com/marcelstoer/docker-nodemcu-build/blob/6e35bbaa5d50dcd8475c710aff8b724509ce2a04/Dockerfile#L13-L23

With some Python magic it may be possible to have it store (and find!) those dependencies in a mounted directory. That way it would only have to download the artifacts once. However, the pip install step is really fast in my environment.

[marcelstoer]

Avoiding to touch components/lua/lua.h yields good results - as long as I'm working inside the running container (e.g. by booting into bash). Subsequent make calls do no longer re-compile the whole firmware.

However, when the container is restarted running the whole build-esp32 script, still minus the lua.h manipulation, make still triggers a complete compile & link cycle. I need more time to figure out why that is i.e. which manipulations run by build-esp32 cause this. Both git status and git diff --submodule=diff confirm that components/platform/include/user_version.h is the only modified file under source control.

[devsaurus]

I need more time to figure out why that is i.e. which manipulations run by build-esp32 cause this.

Sounds like a tough job. Maybe make -d provides some pointers.

[marcelstoer]

Not being familiar with make is definitely is a serious handicap for this task. As the build script really only modifies those two header files https://github.com/marcelstoer/docker-nodemcu-build/blob/master/build-esp32#L15-L16 I suspect it or the compiler keep some temp data somewhere. As containers are immutable those would be gone the next time a build is triggered

[marcelstoer]

when the container is restarted [...] make still triggers a complete compile & link cycle.

Note: the ESP8266 build is also affected.

[mk-pmb]

Jumping in late and haven't read most of the thread, but as for the original concern:

But using docker to compile it takes like 30+ minutes.

… using a faster docker host might help. The free ones provided by GitHub Actions seem to be able to build the firmware in less than 4 minutes: Example