Enhancement: Improve calc Makefiles to work with parallel make

[pmetzger]

Right now, I have to instruct MacPorts not to do make -jN when building because parallel builds seem to consistently fail during configuration. I'm not sure if you have tested parallel make, but it would be nice (for speed) to be able to remove the restriction that calc be built serially.

For example, on my machine, where -j16 would probably be appropriate, I get errors like:

make[1]: *** No rule to make target `../endian_calc.h', needed by `custtbl.o'.  Stop.

after which everything dies. (Everything is fine if -j16 is omitted by the build system.)

My guess is that there are just a bunch of dependencies on .h files built during configuration that aren't reflected in the make rules.

[pmetzger]

Okay, from a cursory look, it appears that custom/Makefile needs to be kept from being invoked until after endian_calc.h is built, but beyond that I'm clueless here.

[lcn2]

We will look at this issue before the version 2.14.0.1 release and see if we can improve things.

[lcn2]

It turns out that the dependency issues for parallel make are very complex. As such we might not fix this issue, let alone in time for version 2.14.0.1.

We will keep trying.

[lcn2]

Wow .. just had a chat with another person who deals with parallel GNU make.

Unfortunately due to the complex dependency issues between Makefile, Makefile.ship and custom/Makefile, parallel GNU make is NOT recommended. Sorry (tm Canada) :)

It would take a major rewrite of the calc build system, and even then parallel GNU make still might not work.

[pmetzger]

That's unfortunate. On a 16 core machine one would like to make use of all 16 cores for the build.

[lcn2]

To make real progress there needs to be a completely new way to build calc AND a new way to configure the calc build. That way could have as a requirement, parallel builds.

What do you think of the GNU-style configure tool? It was a mix of pluses and minuses the last time I looked at it a few years ago.

Another path is to have a simple text file where options are set that in turn, builds a Makefile.

Maybe a new calc issue needs to be created to re-design and improve the calc build process?

Comments?

[pmetzger]

Gnu's autoconf +configure system is a bit messy and intricate, but it works relatively well, and it would eliminate your having to maintain something quite like it on your own. Most C packages one finds for POSIX systems these days seem to use it.

[lcn2]

If autoconf + configure were to be used, then a number of the have_XXX.h files would be replaced by autoconf-ness.

There is also CMake.

There is also SCons.

There may be other stuff ...

There is also "roll my own for exactly eat calc needs"

In general the process might be:

0) configure special stuff (with sane defaults if this is not done) 1) build Makefiles 2) make special things that depend on special configuration stuff 3) make everything else in parallel

Something like that.

comments?

[pmetzger]

My general guess is that the autotools stuff looks like it is the smallest possible change to what you're doing now, given how your current approach works. I'd skim the manual and see if it looks like it would be feasible.

[lcn2]

We are making another attempt, @pmetzger, given the build changes that are on the way for 2.14.0.13.

Right now, various build stages with with make -j, while other stages do not.

Question: If you had a recipe to build calc, but in stages such as:

  make -j foo ; make -j bar ; make -j baz

There each stage can be compiled in parallel (i.e., make -j), nevertheless you would need perform several stages in a sequence. Would that help?

[pmetzger]

That would certainly help, and would be entirely feasible. Long term, I'd suggest that autoconf is probably your friend, but that's obviously going to be far far more work.

[lcn2]

Hello @pmetzger,

We are not sure that moving to autoconf would fix this issue. While there are benefits of autoconf (and there are negative aspects of doing so), the issue is larger than just avoiding the effects of:

make hsrc   # (i.e., replacing have_xxx.h with autoconf stuff)

doing the following:

make -j clobber hsrc all

fails. I.e., there are other parallel make issues with how calc is built, such as the issue of calc static vs. calc dynamic libraries.

=-=

We recommend, however, that you try the following:

make -j clobber hsrc calc-dynamic-only all install BLD_TYPE=calc-dynamic-only

While not ideal, that seems to work more often than not.

Let us know, @pmetzger, if that "work-around" works for your parallel build environment.

[lcn2]

Actually @pmetzger,

When you test, you should make the following edits to disable .NOTPARALLEL:

NOTPARALLEL.ptch.txt

When you do, you will find that the parallel make appears to make things in an order that it shouldn't in terms of dependencies.

We were informed by a supercomputer user that they have a proprietary parallel make that, when the above patch is applied, is able to maximally fully clobber, compile, link, build auxiliary files and install in parallel.  So we conclude that the common gnu-like "make -j" parallelism is not sufficiently cleaver enough to analyze where parallelism may or may not be applied.

While it WON'T compile calc in parallel full, WITHOUT the above patch:

make -j clobber hsrc calc-dynamic-only all install BLD_TYPE=calc-dynamic-only

Without the NOTPARALLEL.ptch.txt, build environments that want to automatically use "make -j" should allow for the dynamically compiled calc to be built in full.

Comments and patches welcome.

[lcn2]

Let us know, @pmetzger, if you please, if you were able to, after applying the above NOTPARALLEL.ptch.txt patch to the top of the master branch, perform the following build and install of the dynamically compiled calc works in your environment:

make -j clobber hsrc calc-dynamic-only all install BLD_TYPE=calc-dynamic-only

If the above command does not work, then what went wrong?

[lcn2]

This is just a reply to comment 992832831 and not a reason why this issue was closed. We added this comment because someone else asked about autoconf for their project.

Calc does not use autoconf(1). Calc predates such systems. While we have looked at the autoconf system, we have found it to be more problematic that beneficial. So the historic use of the Makefile creating "have_foo.h" remains as our custom alternative to autoconf(1).

[lcn2]

We have, over the past 2 years, make changes that should improve compilation performance. There are some things that are serialized, such as the make hsrc stage that creates "have_foo.h" files that need to be done before much of the rest of calc can be compiled.

In our testing of 2 parallel make systems, we found them to be somewhat naive about compile dependencies: making mistakes about what can be compiled in parallel.

We improved the Makefile dependency rules in the hope that a parallel make can do the right thing, if they are able to do so. Nevertheless we found that parallel make tools didn't work well.

Sorry (tm Canada 🇨🇦).

P.S., If someone wanted to provide a pull request that worked for some parallel make, we would be very open to considering it.