falcosecurity / libs

libsinsp, libscap, the kernel module driver, and the eBPF driver sources
https://falcosecurity.github.io/libs/
Apache License 2.0
227 stars 162 forks source link

parallel build sometime fails #57

Closed leogr closed 3 years ago

leogr commented 3 years ago

Describe the bug

When building libs with parallel jobs (e.g., make -j 24), sometimes the process fails on the first run. Usually, after failure, just running make again solves the issue. This behavior might indicate that the dependency building order is wrong, likely because a dependency declaration is missing within the cmake files.

How to reproduce it

mkdir -p build
cd build
cmake -DUSE_BUNDLED_DEPS=On ..
make -j24

If the problem does not happen, remove the build folder and try again.

Expected behaviour

No failure.

Screenshots

make[4]: *** No rule to make target 'libb64.a', needed by 'c-example1'.  Stop.
make[3]: *** [Makefile:8: all_examples] Error 2
make[3]: *** Waiting for unfinished jobs....
-- [download 23% complete]
-- [download 26% complete]
-- [download 28% complete]
-- [download 31% complete]
-- [download 32% complete]
ar: creating libb64.a
a - cencode.o
a - cdecode.o
make[2]: *** [CMakeFiles/b64.dir/build.make:86: b64-prefix/src/b64-stamp/b64-build] Error 2
make[1]: *** [CMakeFiles/Makefile2:346: CMakeFiles/b64.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....

Environment

N/A

Additional context

Probably, this issue is the root cause of the random failures we are recently encountering on Falco's CI (here is an example of job failure on CircleCI).

leogr commented 3 years ago

I can confirm the problem is related to b64.

❯ cmake -DUSE_BUNDLED_DEPS=On ..
CMake Deprecation Warning at CMakeLists.txt:34 (cmake_minimum_required):
  Compatibility with CMake < 2.8.12 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.

-- The C compiler identification is GNU 11.1.0
-- The CXX compiler identification is GNU 11.1.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Using bundled zlib in '/home/leogr/code/github.com/falcosecurity/libs/build/zlib-prefix/src/zlib'
-- Looking for sys/mkdev.h
-- Looking for sys/mkdev.h - not found
-- Looking for sys/sysmacros.h
-- Looking for sys/sysmacros.h - found
-- Using bundled tbb in '/home/leogr/code/github.com/falcosecurity/libs/build/tbb-prefix/src/tbb'
-- Using bundled jsoncpp in '/home/leogr/code/github.com/falcosecurity/libs/cmake/modules/../../userspace/libsinsp/third-party/jsoncpp'
-- Using bundled c-ares in '/home/leogr/code/github.com/falcosecurity/libs/build/c-ares-prefix/src/c-ares'
-- Using bundled b64 in '/home/leogr/code/github.com/falcosecurity/libs/build/b64-prefix/src/b64'
-- Using bundled jq in '/home/leogr/code/github.com/falcosecurity/libs/build/jq-prefix/src/jq'
-- Bundled jq: include: /home/leogr/code/github.com/falcosecurity/libs/build/jq-prefix/src/jq/target/include, lib: /home/leogr/code/github.com/falcosecurity/libs/build/jq-prefix/src/jq/target/lib/libjq.a
-- Using bundled jsoncpp in '/home/leogr/code/github.com/falcosecurity/libs/cmake/modules/../../userspace/libsinsp/third-party/jsoncpp'
-- Using bundled openssl in '/home/leogr/code/github.com/falcosecurity/libs/build/openssl-prefix/src/openssl'
-- Using bundled protobuf in '/home/leogr/code/github.com/falcosecurity/libs/build/protobuf-prefix/src/protobuf'
-- Using bundled grpc in '/home/leogr/code/github.com/falcosecurity/libs/build/grpc-prefix/src/grpc'
-- Using bundled curl in '/home/leogr/code/github.com/falcosecurity/libs/build/curl-prefix/src/curl'
-- Using SSL for curl in '--with-ssl=/home/leogr/code/github.com/falcosecurity/libs/build/openssl-prefix/src/openssl/target'
-- Using bundled jsoncpp in '/home/leogr/code/github.com/falcosecurity/libs/cmake/modules/../../userspace/libsinsp/third-party/jsoncpp'
CMake Deprecation Warning at CMakeLists.txt:17 (cmake_minimum_required):
  Compatibility with CMake < 2.8.12 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.

-- Configuring done
-- Generating done
-- Build files have been written to: /home/leogr/code/github.com/falcosecurity/libs/build/googletest-download
[ 11%] Creating directories for 'googletest'
[ 22%] Performing download step (git clone) for 'googletest'
Cloning into 'googletest-src'...
Already on 'master'
Your branch is up to date with 'origin/master'.
[ 33%] Performing update step for 'googletest'
HEAD is now at 8d51ffdf Googletest export
[ 44%] No patch step for 'googletest'
[ 55%] No configure step for 'googletest'
[ 66%] No build step for 'googletest'
[ 77%] No install step for 'googletest'
[ 88%] No test step for 'googletest'
[100%] Completed 'googletest'
[100%] Built target googletest
-- Found Python: /usr/bin/python3.9 (found version "3.9.6") found components: Interpreter 
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE  
-- Configuring done
-- Generating done
-- Build files have been written to: /home/leogr/code/github.com/falcosecurity/libs/build
❯ make -j24 b64
[ 25%] Creating directories for 'b64'
[ 50%] Performing download step (download, verify and extract) for 'b64'
-- Downloading...
   dst='/home/leogr/code/github.com/falcosecurity/libs/build/b64-prefix/src/v1.2.1.tar.gz'
   timeout='none'
   inactivity timeout='none'
-- Using src='https://github.com/libb64/libb64/archive/v1.2.1.tar.gz'
-- verifying file...
       file='/home/leogr/code/github.com/falcosecurity/libs/build/b64-prefix/src/v1.2.1.tar.gz'
-- Downloading... done
-- extracting...
     src='/home/leogr/code/github.com/falcosecurity/libs/build/b64-prefix/src/v1.2.1.tar.gz'
     dst='/home/leogr/code/github.com/falcosecurity/libs/build/b64-prefix/src/b64'
-- extracting... [tar xfz]
-- extracting... [analysis]
-- extracting... [rename]
-- extracting... [clean up]
-- extracting... done
[ 75%] No update step for 'b64'
[ 75%] No patch step for 'b64'
[ 75%] No configure step for 'b64'
[100%] Performing build step for 'b64'
make[5]: *** No rule to make target 'libb64.a', needed by 'c-example1'.  Stop.
make[4]: *** [Makefile:8: all_examples] Error 2
make[4]: *** Waiting for unfinished jobs....
ar: creating libb64.a
a - cencode.o
a - cdecode.o
make[3]: *** [CMakeFiles/b64.dir/build.make:86: b64-prefix/src/b64-stamp/b64-build] Error 2
make[2]: *** [CMakeFiles/Makefile2:346: CMakeFiles/b64.dir/all] Error 2
make[1]: *** [CMakeFiles/Makefile2:353: CMakeFiles/b64.dir/rule] Error 2
make: *** [Makefile:195: b64] Error 2