ROOT fails to build many builtins with CMake 3.30 (>=3.28) (when building in parallel (-j N))

[andresailer]

Check duplicate issues.

• [X] Checked for duplicates

Description

With newer versions of cmake like 3.30.5 at least some of ExternalProject_Add'ed builtins fail to compile with an error like

gmake[6]: *** read jobs pipe: Bad file descriptor.  Stop.
gmake[6]: *** Waiting for unfinished jobs....
gmake[5]: *** [CMakeFiles/Makefile2:97: CMakeFiles/pcre.dir/all] Error 2
gmake[4]: *** [CMakeFiles/Makefile2:104: CMakeFiles/pcre.dir/rule] Error 2
gmake[3]: *** [Makefile:179: pcre] Error 2

(here builtin PCRE failed)

I don't know with which version of cmake this starts failing, but we (SPI/LCG) moved from 3.26.2 to 3.30. Unless one limits oneself to a small number of --jobs?

Reproducer

On lxplus or equivalent machines with CVMFS for example

export PATH=/cvmfs/sft.cern.ch/lcg/contrib/CMake/3.30.5/Linux-x86_64/bin/:$PATH
git clone https://github.com/root-project/root.git
cd root
mkdir build
cd build
cmake -D builtin_pcre=ON ..
make -j8

ROOT version

any? (definitely HEAD)

Installation method

build from source

Operating system

Linux

Additional context

No response

[bellenot]

Is it really a ROOT bug? Or a CMake one? 🤔

[bellenot]

OK, so apparently it's only builtin_pcre which is causing problems (i.e. pcre-8.43.tar.bz2)

[bellenot]

FYI there is the same error with pcre 8.45 (which is the latest available). Investigating...

[bellenot]

OK, I found a work-around. PR is coming soon. And note that the issue is not present on Windows

[andresailer]

It probably depends on which builtins you enable? I also see this for builtin_XROOTD

gmake[4]: *** read jobs pipe: Bad file descriptor.  Stop.
gmake[4]: *** Waiting for unfinished jobs....
gmake[3]: *** [Makefile:146: all] Error 2

[bellenot]

Oh crap! I can't build XRootD... OK, thanks! I'll do the same with XRootD. Anything else?

[bellenot]

Note that would be interesting to see if it's possible to build XRootD in parallel outside ROOT

[andresailer]

I haven't found anything else so far. Do you have a cmake command to enable all builtins?

And yes we build xrootd outside of ROOT for the LCG stacks with cmake 3.30 without problems. Heck, I can build pcre outside of ROOT without problems as well.

[bellenot]

I haven't found anything else so far. Do you have a cmake command to enable all builtins?

I don't think so

And yes we build xrootd outside of ROOT for the LCG stacks with cmake 3.30 without problems. Heck, I can build pcre outside of ROOT without problems as well.

OK, so I'll have to spend more time on this... Thanks Andre!

[andresailer]

ccmake to the rescue... TBB

gmake[5]: *** read jobs pipe: Bad file descriptor.  Stop.
gmake[5]: *** Waiting for unfinished jobs....
gmake[4]: *** [CMakeFiles/Makefile2:134: src/tbb/CMakeFiles/tbb.dir/all] Error 2
gmake[3]: *** [Makefile:156: all] Error 2

FFTW3

make[5]: *** read jobs pipe: Bad file descriptor.  Stop.
make[5]: *** Waiting for unfinished jobs....
make[4]: *** [Makefile:708: all-recursive] Error 1
make[3]: *** [Makefile:552: all] Error 2

freetype (in the configure step??? :confused: )

make[3]: *** read jobs pipe: Bad file descriptor.  Stop.
make[3]: *** Waiting for unfinished jobs....

[andresailer]

cfitsio

gmake[5]: *** read jobs pipe: Bad file descriptor.  Stop.
gmake[5]: *** Waiting for unfinished jobs....
gmake[4]: *** [CMakeFiles/Makefile2:83: CMakeFiles/cfitsio.dir/all] Error 2
gmake[4]: Target 'all' not remade because of errors.
gmake[3]: *** [Makefile:136: all] Error 2
gmake[3]: Target 'default_target' not remade because of errors.

I couldn't enable builtin_zlib, gave some other cmake error

CMake Error at interpreter/llvm-project/llvm/lib/Support/CMakeLists.txt:303 (get_property):
  The LOCATION property may not be read from target "ZLIB".  Use the target
  name directly with add_custom_command, or use the generator expression
  $<TARGET_FILE>, as appropriate.

CMake Error at interpreter/llvm-project/llvm/lib/Support/CMakeLists.txt:306 (get_property):
  The LOCATION property may not be read from target "ZLIB".  Use the target
  name directly with add_custom_command, or use the generator expression
  $<TARGET_FILE>, as appropriate.

CMake Error at interpreter/llvm-project/llvm/lib/Support/CMakeLists.txt:308 (get_library_name):
  get_library_name Function invoked with incorrect arguments for function
  named: get_library_name

[bellenot]

OK, thanks André! It looks like CMake is badly broken...

[bellenot]

Looking there: https://www.kitware.com/cmake-3-28-0-rc5-is-ready-for-testing/ It seems adding BUILD_JOB_SERVER_AWARE true in ExternalProject_Add fixes the issue (at least for PCRE). Trying it right now (but xrootd fails for some reason...)

[bellenot]

Adding BUILD_JOB_SERVER_AWARE 1 partially fixes the issue, but adding it only for CMake version > 3.28 in something like 50 different places would be a pain in the neck... see also https://gitlab.kitware.com/cmake/cmake/-/issues/26398

[andresailer]

For XROOTD I didn't see BUILD_JOB_SERVER_AWARE helping, the doc also says that it only applies to explicit BUILD_COMMANDS, which XROOTD doesn't have, it only has an explicit INSTALL_COMMAND. Without these explicit commands it seems to work better.

For FREETYPE I saw no difference until I changed LOG_CONFIGURE 0 instead of LOG_CONFIGURE 1

[bellenot]

See also the discussion here: https://gitlab.kitware.com/cmake/cmake/-/issues/26398

[andresailer]

Ubuntu24 comes with cmake version 3.28.3. (cf. #16654, #16682)

[bellenot]

Thanks André!

[bellenot]

OK, so simply disabling the LOG_CONFIGURE, LOG_BUILD, LOG_INSTALL when using CONFIGURE_COMMAND, BUILD_COMMNAD, INSTALL_COMMAND seems to fix the issue (i.e. no BUILD_JOB_SERVER_AWARE involved), but then we have no log...