`make -j` explodes, sometimes

[jpco]

es is quick enough to build that this doesn't seem terribly material, but it's likely a sign of a missing dependency link somewhere in the Makefile that should be fixed up. It seems more likely if optimization is off, like -O0.

The errors seem to all be around y.tab.c.

[jpco]

Okay, as soon as I filed this it stopped happening. I'm gonna leave this open for now to see if I can figure out how to get it to start happening again. It really was just as simple as

; make clean; make -j

and about one in four builds would fail catastrophically while trying to compile y.tab.c.

[jpco]

Aha, I'm getting repros again! Yay!

The failures all happen while trying to produce y.tab.o. I notice in the Makefile there is no config line referring to dependencies of y.tab.o -- I suspect it may be trying to start compiling y.tab.c too early due to a missing dependency statement.

Experimentally, it seems like adding the line

y.tab.o : y.tab.c y.tab.h 

stops the errors happening. But this is all weird and probabilistic behavior so I'd love to find some documentation that actually helps me know what the Right Thing to do here would be.

[jpco]

Here's my best guess right now as to what's going on:

make doesn't know that a single yacc invocation produces both y.tab.c and y.tab.h. So what happens is, if make tries to get at y.tab.c it'll run yacc, and then if it also wants y.tab.h before yacc is done the first time, it'll run yacc again, re-generating both. And then, if make starts trying to build y.tab.o while y.tab.c is being re-generated, it'll blow up due to whatever weird write race going on.

This would explain both why yacc runs twice when make is invoked with -j but not without, and why adding y.tab.h as a dependency for y.tab.o fixes the crashes despite y.tab.h not having anything to do (directly) with actually compiling y.tab.c.

Turns out GNU make 4.3 added a feature called "grouped targets" for just this scenario! That makes me more confident this is what's happening here.

Unfortunately that feature isn't very portable, so unless we expect everybody to use a recent-ish GNU make to build es, it might be better to just specify y.tab.o : y.tab.c y.tab.h and allow yacc to get run twice, sometimes.

[memreflect]

make doesn't know that a single yacc invocation produces both y.tab.c and y.tab.h.

That's correct. make has targets and prerequisites, not file inputs and file outputs.

So what happens is, if make tries to get at y.tab.c it'll run yacc, and then if it also wants y.tab.h before yacc is done the first time, it'll run yacc again, re-generating both. And then, if make starts trying to build y.tab.o while y.tab.c is being re-generated, it'll blow up due to whatever weird write race going on.

Exactly right, and i agree that it's a really weird race, but that's also why newer build tools like Ninja and new meta-build systems like CMake support multiple outputs. The Automake manual sort of mentions this at the end of its 8.8 Yacc and Lex section. While its context is output from multiple yacc/lex processes executing in parallel as a result of multiple yacc/lex source files, the idea is the same: multiple yacc processes executing in parallel as a result of requiring multiple yacc output files at the same time.

This would explain both why yacc runs twice when make is invoked with -j but not without, and why adding y.tab.h as a dependency for y.tab.o fixes the crashes despite y.tab.h not having anything to do (directly) with actually compiling y.tab.c.

One of the BUILT_SOURCES examples in the Automake manual seems to have a similar example in "Recording Dependencies manually":

foo.$(OBJEXT): bindir.h

Another option might be to have y.tab.c depend on y.tab.h and specify y.tab.h as the only target with parse.y as a prerequisite. It's still not pretty, but it's perhaps more logical to have a source file depend on a header and an object file depend on that source file.

[jpco]

Another option might be to have y.tab.c depend on y.tab.h and specify y.tab.h as the only target with parse.y as a prerequisite.

Ooh, this is much nicer than what I came up with. I think we should do this (likely with a brief explanatory comment).