ArcticaProject / nx-libs

nx-libs
Other
120 stars 39 forks source link

Random parallel build failures #1072

Open Apteryks opened 7 months ago

Apteryks commented 7 months ago

Hi,

The build of nx-libs appears to suffer from non-deterministic build failures when parallelized (make -jN). Here's an example:

making all in programs/Xserver...
make[6]: Entering directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11/programs'
make[6]: warning: -j24 forced in submake: resetting jobserver mode.
making all in programs/Xserver/include...
making all in programs/Xserver/dix...
making all in programs/Xserver/os...
making all in programs/Xserver/mi...
making all in programs/Xserver/Xext...
making all in programs/Xserver/Xi...
making all in programs/Xserver/xkb...
making all in programs/Xserver/GL...
making all in programs/Xserver/dbe...
make[7]: Entering directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11/programs/Xserver'
make[7]: warning: -j24 forced in submake: resetting jobserver mode.
making all in programs/Xserver/record...
make[7]: Nothing to be done for 'all'.
make[7]: Leaving directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11/programs/Xserver/include
'
making all in programs/Xserver/randr...
make[7]: Entering directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11/programs/Xserver'
make[7]: warning: -j24 forced in submake: resetting jobserver mode.
make[7]: Entering directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11/programs/Xserver'
make[7]: warning: -j24 forced in submake: resetting jobserver mode.
making all in programs/Xserver/render...
make[7]: Entering directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11/programs/Xserver'
make[7]: warning: -j24 forced in submake: resetting jobserver mode.
make[7]: Entering directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11/programs/Xserver'
make[7]: warning: -j24 forced in submake: resetting jobserver mode.
making all in programs/Xserver/xfixes...
make[7]: Entering directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11/programs/Xserver'
make[7]: warning: -j24 forced in submake: resetting jobserver mode.
making all in programs/Xserver/damageext...
make[7]: Entering directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11/programs/Xserver'
make[7]: warning: -j24 forced in submake: resetting jobserver mode.
make[7]: *** No rule to make target 'all'.  Stop.
make[7]: Leaving directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11/programs/Xserver/os'
making all in programs/Xserver/miext/cw...
make[6]: *** [Makefile:682: os] Error 2
make[6]: *** Waiting for unfinished jobs....

It'd be nice if it worked normally. I also not that parallel build is disabled in the Debian rules file (--no-parallel).

uli42 commented 5 months ago

You ouput is unclear to me. What subdir exactly fails?

Is it damageext?

Apteryks commented 5 months ago

It's not clear to me either, but that's what I see! It does look like the error is triggered from the damageext directory.

uli42 commented 4 months ago

Just stumbled over this in nx-X11/programs/Xserver/Imakefile:

#if HasParallelMake
MakeMutex($(NXAGENTDIRS) $(NXCOMPEXT) $(NXAGENTOBJS) $(NXAGENTLIBS) $(NXCOMPEXTSYSLIBS) $(NXAGENTSYSLIBS))
#endif

So you probably have to run the topdir makefile with IMAKE_DEFINES=-DHasParallelMake=1

Apteryks commented 4 months ago

Hm, I tried that, but it didn't seem to make any difference:

make[5]: Leaving directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11/include/GL'
make[4]: Leaving directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11/include'
including in ./config...
make[4]: Entering directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11/config'
including in config/cf...
make[5]: Entering directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11/config/cf'
make[5]: Nothing to be done for 'includes'.
make[5]: Leaving directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11/config/cf'
make[4]: Leaving directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11/config'
including in ./programs...
make[4]: Entering directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11/programs'
including in programs/Xserver...
make[5]: Entering directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11/programs/Xserver'
Makefile:539: *** missing separator.  Stop.
make[5]: Leaving directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11/programs/Xserver'
make[4]: *** [Makefile:615: includes] Error 2
make[4]: Leaving directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11/programs'
make[3]: *** [xmakefile:674: includes] Error 2
make[3]: Leaving directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11'
make[2]: *** [Makefile:102: includes] Error 2
make[2]: Leaving directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11'
make[1]: *** [Makefile:148: build-env] Error 2
make[1]: Leaving directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source'
make: *** [Makefile:220: build] Error 2
error: in phase 'build': uncaught exception:
%exception #<&invoke-error program: "make" arguments: ("-j" "180" "PREFIX=/gnu/store/xz96mncfbz47spmdllxyki4hv6vaal4a-nx-libs-3.5.99.27" "ETCDIR_NX=/gnu/store/xz96mncfbz47spmdllxyki4hv6vaal4a-nx-libs-3.5.99.27/etc" "LOCAL_LDFLAGS=-Wl,-rpath=/gnu/store/xz96mncfbz47spmdllxyki4hv6vaal4a-nx-libs-3.5.99.27/lib,-rpath=/gnu/store/xz96mncfbz47spmdllxyki4hv6vaal4a-nx-libs-3.5.99.27/lib/nx/X11" "IMAKE_DEFINES=-DUseTIRPC=1 -DHasParallelMake=1 -DBourneShell=/gnu/store/9vw5slrffp27rzy2i2plnw7xfqjyk7m4-bash-minimal-5.1.16/bin/sh -DProjectRoot=/gnu/store/xz96mncfbz47spmdllxyki4hv6vaal4a-nx-libs-3.5.99.27 -DDefaultUserPath=/gnu/store/xz96mncfbz47spmdllxyki4hv6vaal4a-nx-libs-3.5.99.27/bin" "CONFIG_SHELL=/gnu/store/9vw5slrffp27rzy2i2plnw7xfqjyk7m4-bash-minimal-5.1.16/bin/sh" "SHELL=/gnu/store/9vw5slrffp27rzy2i2plnw7xfqjyk7m4-bash-minimal-5.1.16/bin/sh" "CONFIGURE=/gnu/store/9vw5slrffp27rzy2i2plnw7xfqjyk7m4-bash-minimal-5.1.16/bin/sh ./configure --prefix=/gnu/store/xz96mncfbz47spmdllxyki4hv6vaal4a-nx-libs-3.5.99.27" "VERBOSE=1") exit-status: 2 term-signal: #f stop-signal: #f> 
phase `build' failed after 5.9 seconds
command "make" "-j" "180" "PREFIX=/gnu/store/xz96mncfbz47spmdllxyki4hv6vaal4a-nx-libs-3.5.99.27" "ETCDIR_NX=/gnu/store/xz96mncfbz47spmdllxyki4hv6vaal4a-nx-libs-3.5.99.27/etc" "LOCAL_LDFLAGS=-Wl,-rpath=/gnu/store/xz96mncfbz47spmdllxyki4hv6vaal4a-nx-libs-3.5.99.27/lib,-rpath=/gnu/store/xz96mncfbz47spmdllxyki4hv6vaal4a-nx-libs-3.5.99.27/lib/nx/X11" "IMAKE_DEFINES=-DUseTIRPC=1 -DHasParallelMake=1 -DBourneShell=/gnu/store/9vw5slrffp27rzy2i2plnw7xfqjyk7m4-bash-minimal-5.1.16/bin/sh -DProjectRoot=/gnu/store/xz96mncfbz47spmdllxyki4hv6vaal4a-nx-libs-3.5.99.27 -DDefaultUserPath=/gnu/store/xz96mncfbz47spmdllxyki4hv6vaal4a-nx-libs-3.5.99.27/bin" "CONFIG_SHELL=/gnu/store/9vw5slrffp27rzy2i2plnw7xfqjyk7m4-bash-minimal-5.1.16/bin/sh" "SHELL=/gnu/store/9vw5slrffp27rzy2i2plnw7xfqjyk7m4-bash-minimal-5.1.16/bin/sh" "CONFIGURE=/gnu/store/9vw5slrffp27rzy2i2plnw7xfqjyk7m4-bash-minimal-5.1.16/bin/sh ./configure --prefix=/gnu/store/xz96mncfbz47spmdllxyki4hv6vaal4a-nx-libs-3.5.99.27" "VERBOSE=1" failed with status 2
builder for `/gnu/store/zik74m5vcclr9bcs44sfqh46i2rh0rv3-nx-libs-3.5.99.27.drv' failed with exit code 1
build of /gnu/store/zik74m5vcclr9bcs44sfqh46i2rh0rv3-nx-libs-3.5.99.27.drv failed
View build log at '/var/log/guix/drvs/zi/k74m5vcclr9bcs44sfqh46i2rh0rv3-nx-libs-3.5.99.27.drv.gz'.
guix build: error: build of `/gnu/store/zik74m5vcclr9bcs44sfqh46i2rh0rv3-nx-libs-3.5.99.27.drv' failed

real    0m42.514s
user    0m6.178s
sys     0m0.594s
uli42 commented 4 months ago

I see "./configure" in the output. That is used for some components. Of course that is not synced with the imake build. So I think you have to modify the main makefile to introduce some "barriers" First compile the automake parts and then the imake parts (xserver). Be aware that nxcomp and nxcompshad are required by the other parts. And the server depends on both and libNX_x11.

If you are wiling to help there's an automake branch that needs some polishing. It should be findable in the pull-requests. Let me know if you don't find it.

Xorg uses meson nowadays so the optimum was to restructure everything to use just that.

Uli

Apteryks @.***> schrieb am So., 19. Mai 2024, 03:11:

Hm, I tried that, but it didn't seem to make any difference:

make[5]: Leaving directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11/include/GL' make[4]: Leaving directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11/include' including in ./config... make[4]: Entering directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11/config' including in config/cf... make[5]: Entering directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11/config/cf' make[5]: Nothing to be done for 'includes'. make[5]: Leaving directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11/config/cf' make[4]: Leaving directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11/config' including in ./programs... make[4]: Entering directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11/programs' including in programs/Xserver... make[5]: Entering directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11/programs/Xserver' Makefile:539: missing separator. Stop. make[5]: Leaving directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11/programs/Xserver' make[4]: [Makefile:615: includes] Error 2 make[4]: Leaving directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11/programs' make[3]: [xmakefile:674: includes] Error 2 make[3]: Leaving directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11' make[2]: [Makefile:102: includes] Error 2 make[2]: Leaving directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source/nx-X11' make[1]: [Makefile:148: build-env] Error 2 make[1]: Leaving directory '/tmp/guix-build-nx-libs-3.5.99.27.drv-0/source' make: [Makefile:220: build] Error 2 error: in phase 'build': uncaught exception: %exception #<&invoke-error program: "make" arguments: ("-j" "180" "PREFIX=/gnu/store/xz96mncfbz47spmdllxyki4hv6vaal4a-nx-libs-3.5.99.27" "ETCDIR_NX=/gnu/store/xz96mncfbz47spmdllxyki4hv6vaal4a-nx-libs-3.5.99.27/etc" "LOCAL_LDFLAGS=-Wl,-rpath=/gnu/store/xz96mncfbz47spmdllxyki4hv6vaal4a-nx-libs-3.5.99.27/lib,-rpath=/gnu/store/xz96mncfbz47spmdllxyki4hv6vaal4a-nx-libs-3.5.99.27/lib/nx/X11" "IMAKE_DEFINES=-DUseTIRPC=1 -DHasParallelMake=1 -DBourneShell=/gnu/store/9vw5slrffp27rzy2i2plnw7xfqjyk7m4-bash-minimal-5.1.16/bin/sh -DProjectRoot=/gnu/store/xz96mncfbz47spmdllxyki4hv6vaal4a-nx-libs-3.5.99.27 -DDefaultUserPath=/gnu/store/xz96mncfbz47spmdllxyki4hv6vaal4a-nx-libs-3.5.99.27/bin" "CONFIG_SHELL=/gnu/store/9vw5slrffp27rzy2i2plnw7xfqjyk7m4-bash-minimal-5.1.16/bin/sh" "SHELL=/gnu/store/9vw5slrffp27rzy2i2plnw7xfqjyk7m4-bash-minimal-5.1.16/bin/sh" "CONFIGURE=/gnu/store/9vw5slrffp27rzy2i2plnw7xfqjyk7m4-bash-minimal-5.1.16/bin/sh ./configure --prefix=/gnu/store/xz96mncfbz47spmdllxyki4hv6vaal4a-nx-libs-3.5.99.27" "VERBOSE=1") exit-status: 2 term-signal: #f stop-signal: #f> phase build' failed after 5.9 seconds command "make" "-j" "180" "PREFIX=/gnu/store/xz96mncfbz47spmdllxyki4hv6vaal4a-nx-libs-3.5.99.27" "ETCDIR_NX=/gnu/store/xz96mncfbz47spmdllxyki4hv6vaal4a-nx-libs-3.5.99.27/etc" "LOCAL_LDFLAGS=-Wl,-rpath=/gnu/store/xz96mncfbz47spmdllxyki4hv6vaal4a-nx-libs-3.5.99.27/lib,-rpath=/gnu/store/xz96mncfbz47spmdllxyki4hv6vaal4a-nx-libs-3.5.99.27/lib/nx/X11" "IMAKE_DEFINES=-DUseTIRPC=1 -DHasParallelMake=1 -DBourneShell=/gnu/store/9vw5slrffp27rzy2i2plnw7xfqjyk7m4-bash-minimal-5.1.16/bin/sh -DProjectRoot=/gnu/store/xz96mncfbz47spmdllxyki4hv6vaal4a-nx-libs-3.5.99.27 -DDefaultUserPath=/gnu/store/xz96mncfbz47spmdllxyki4hv6vaal4a-nx-libs-3.5.99.27/bin" "CONFIG_SHELL=/gnu/store/9vw5slrffp27rzy2i2plnw7xfqjyk7m4-bash-minimal-5.1.16/bin/sh" "SHELL=/gnu/store/9vw5slrffp27rzy2i2plnw7xfqjyk7m4-bash-minimal-5.1.16/bin/sh" "CONFIGURE=/gnu/store/9vw5slrffp27rzy2i2plnw7xfqjyk7m4-bash-minimal-5.1.16/bin/sh ./configure --prefix=/gnu/store/xz96mncfbz47spmdllxyki4hv6vaal4a-nx-libs-3.5.99.27" "VERBOSE=1" failed with status 2 builder for/gnu/store/zik74m5vcclr9bcs44sfqh46i2rh0rv3-nx-libs-3.5.99.27.drv' failed with exit code 1 build of /gnu/store/zik74m5vcclr9bcs44sfqh46i2rh0rv3-nx-libs-3.5.99.27.drv failed View build log at '/var/log/guix/drvs/zi/k74m5vcclr9bcs44sfqh46i2rh0rv3-nx-libs-3.5.99.27.drv.gz'. guix build: error: build of `/gnu/store/zik74m5vcclr9bcs44sfqh46i2rh0rv3-nx-libs-3.5.99.27.drv' failed

real 0m42.514s user 0m6.178s sys 0m0.594s

— Reply to this email directly, view it on GitHub https://github.com/ArcticaProject/nx-libs/issues/1072#issuecomment-2119049992, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABQHBZBVX4SJEFQJGEEHA4DZC734PAVCNFSM6AAAAABELVQQ2OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMJZGA2DSOJZGI . You are receiving this because you commented.Message ID: @.***>

Apteryks commented 4 months ago

If you are wiling to help there's an automake branch that needs some polishing. It should be findable in the pull-requests. Let me know if you don't find it.

That's tempting but am I bit swamped with things to do at the moment. I'll try to remember revisiting this in the future!

Apteryks commented 4 months ago

Xorg uses meson nowadays so the optimum was to restructure everything to use just that.

"Was" makes it sound like the migration of the build system to Meson already happened? Is that so?

uli42 commented 4 months ago

No, "was" is subjunctive, like "If I were rich" ;-)

Uli

On Wed, May 22, 2024 at 1:58 AM Apteryks @.***> wrote:

Xorg uses meson nowadays so the optimum was to restructure everything to use just that.

"Was" makes it sound like the migration of the build system to Meson already happened? Is that so?

— Reply to this email directly, view it on GitHub https://github.com/ArcticaProject/nx-libs/issues/1072#issuecomment-2123611828, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABQHBZCFZI4T3U4SOUZODDTZDPNTTAVCNFSM6AAAAABELVQQ2OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMRTGYYTCOBSHA . You are receiving this because you commented.Message ID: @.***>

Apteryks commented 4 months ago

Ah, I see :-) I guess I'll look at that Autotools branch at some point; it'd be a nice improvement over a build system whose default assumption is that make doesn't support parallel builds ^^'.

uli42 commented 4 months ago

;-)

Here are the respective PRs: https://github.com/ArcticaProject/nx-libs/pull/947 https://github.com/ArcticaProject/nx-libs/pull/946

On Wed, May 22, 2024 at 2:33 PM Apteryks @.***> wrote:

Ah, I see :-) I guess I'll look at that Autotools branch at some point; it'd be a nice improvement over a build system whose default assumption is that make doesn't support parallel builds ^^'.

— Reply to this email directly, view it on GitHub https://github.com/ArcticaProject/nx-libs/issues/1072#issuecomment-2124679613, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABQHBZA65DCW5IQUFD7P6QLZDSGAJAVCNFSM6AAAAABELVQQ2OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMRUGY3TSNRRGM . You are receiving this because you commented.Message ID: @.***>