Enabling faster compile times using make -jN switch

[rodgergr]

• Pycom board: LoPy4
• Firmware version: 1.20.2r1

Pycom Micropython projects ignore the '-j'or '--jobs' switch on the make command line, whereas vanilla micropython.org projects allow this.

To understand this better, I timed a standard build for my project after using make clean at 2 min 29 s.

Investigating, I found the following two lines of code

https://github.com/pycom/pycom-micropython-sigfox/blob/d574024b715990fa3323073f9eb1fe327987be81/esp32/application.mk#L907-L908

that seemed salient. These were rules with a target of .NOTPARALLEL and various prerequisites

The GNU Make manual states

You can inhibit parellism in a particular makefile with the .NOTPARALLEL pseudotarget

https://www.gnu.org/software/make/manual/make.html#Parallel

Further,section 4.9 of the same manual (Special Built-in Target Names) says

.NOTPARALLEL

If .NOTPARALLEL is mentioned as a target, then this invocation of make will be run serially, even if the -j option is given. Any recursively invoked make command will still run recipes in parallel (unless its makefile also contains this target). Any prerequisites on this target are ignored (italics mine)

https://www.gnu.org/software/make/manual/make.html#Special-Targets

So, if the prerequisites are ignored by make, the above two lines are equivalent to

.NOTPARALLEL:

which simply disables parallellism for the whole makefile.

Commenting out these two lines and rerunning make with a -j9 switch resulted in a 50% reduction in compile and build time down to only 1 min 11 s, a worthwhile saving in a code-build-upload-test cycle..

As make defaults to -j1 anyway, could the developer be given the choice to select -jN in the make command line to allow for much faster compile times if there are no other issues preventing parallelism? This could be achieved by simply deleting these two apparently redundant lines in the code.

I would be happy to submit a PR on this, but thought I would raise it here first in case I am missing an obvious showstopper.

p.s. This is my first time raising an issue, so please forgive and point out any unintentional etiquette errors.

[peter-pycom]

Thanks for the well written report rodgergr! Yes the nonparallel build is annoying. My main problem with changing this is however not about obvious showstoppers but rather about the non-obvious aspect. Why has it been put in place? Maybe something in the dependency determination could break under certain circumstances?! I always wanted to check the git history why it was put in place, but it keeps falling to the bottom of my list :(

Oct 12, 2020 02:52:49 rodgergr notifications@github.com:

• Pycom board: LoPy4

• Firmware version: 1.20.2r1 Pycom Micropython projects ignore the '-j' or '--jobs' switch on the make command line, whereas vanilla micropython.org projects allow this.

  To understand this better, I timed a standard build for my project after using make clean at 2 min 29 s.

  Investigating, I found the following two lines of code at lines 907 and 908 in the esp32/application.mk file that seemed salient. These were rules with a target of .NOTPARALLEL and various prerequisites

  .NOTPARALLEL: CHECK_DEP $ (OBJ) .NOTPARALLEL: CHECK_DEP $ (BOOT_OBJ)The GNU Make manual states

  You can inhibit parellism in a particular makefile with the .NOTPARALLEL pseudotarget

  https://www.gnu.org/software/make/manual/make.html#Parallel

  Further,section 4.9 of the same manual (Special Built-in Target Names) says

  .NOTPARALLEL

  If .NOTPARALLEL is mentioned as a target, then this invocation of make will be run serially, even if the -j option is given. Any recursively invoked make command will still run recipes in parallel (unless its makefile also contains this target). Any prerequisites on this target are ignored (italics mine)

  https://www.gnu.org/software/make/manual/make.html#Special-Targets

  So, if the prerequisites are ignored by make, the above two lines are equivalent to

  .NOTPARALLEL:

  which simply disables parallellism for the whole makefile.

  Commenting out these two lines and rerunning make with a -j9 switch resulted in a 50% reduction in compile and build time down to only 1 min 11 s, a worthwhile saving in a code-build-upload-test cycle..

  As make defaults to -j1 anyway, could the developer be given the choice to select -jN in the make command line to allow for much faster compile times if there are no other issues preventing parallelism? This could be achieved by simply deleting these two apparently redundant lines in the code.

  I would be happy to submit a PR on this, but thought I would raise it here first in case I am missing an obvious showstopper.

  p.s. This is my first time raising an issue, so please forgive and point out any unintentional etiquette errors.

  — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub[https://github.com/pycom/pycom-micropython-sigfox/issues/485], or unsubscribe[https://github.com/notifications/unsubscribe-auth/AOUQ75TCUJYFPRXSGF6R2X3SKJHN5ANCNFSM4SMEQPCQ]. [https://github.com/notifications/beacon/AOUQ75RQ3MHVENQ2S4QOHVLSKJHN5A5CNFSM4SMEQPC2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4KW2SMLA.gif]