HAS_FUZZ_FLAGS fails on OSX & Linux

[steven-johnson]

~The fuzz tests won't run on OSX, because (apparently?) we are using the Xcode default version of Clang to execute check_cxx_source_compiles(), which fails. Can we make this work on OSX?~

Update: Fails for me on Linux as well. What special build magic is needed to get this to pass?

[steven-johnson]

attn @silvergasp

[steven-johnson]

Huh... it looks like the answer is "we don't attempt to build or test these at all inside Halide" -- that's not a good idea. Maybe these are intended mostly for use with the OSS-Fuzz integration, but we really need a way to at least build these as part of our CI, just to verify syntactic issues; otherwise the bugs will come from downstream things like OSS-Fuzz integration, or Google/Adobe bug reports, etc.

[nathaniel-brough]

Hmm that's odd. I have been testing the more recent changes in the oss fuzz build containers where it seems to be working fine.

So it's likely something to do with the oss fuzz environment that's a little different to your local system. What happens if you run.

export CFLAGS=-fsanitize=fuzzer-no-link,address
export CXXFLAGS=$CFLAGS
export LDFLAGS=-fsanitize=fuzzer, address
export CC=clang
export CXX=clang++

Before building/configuring? I'll be away from my computer for most of this afternoon and tomorrow but after that I can have a look in more detail.

[steven-johnson]

something to do with the oss fuzz environment

See, I think that may be the issue: do you need an "oss fuzz environment" (e.g. a container of some sort to build these)? If so, that needs documentation, because I don't see any indication of what's needed to build it via CMake on a posixy system.

What happens if you run

Failures of the form:

  The C compiler

    "/usr/bin/clang"

  is not able to compile a simple test program.

  It fails with the following output:
      <omitted>

[nathaniel-brough]

Failures of the form:

I may have made a bit of a careless mistake copy and pasting the above code, so my apologies. Anyway see below for a fully working set of build steps.

See, I think that may be the issue: do you need an "oss fuzz environment" (e.g. a container of some sort to build these)? If so, that needs documentation, because I don't see any indication of what's needed to build it via CMake on a posixy system.

Well I don't think that you should need an "oss-fuzz environment" to build these fuzz tests, and in fact I started building these just on my regular ubuntu 20.04 install without oss-fuzz involved.

I then switched over to the oss-fuzz build environment to make sure that everything worked there, which actually turned out to be quite a challenge. Everything in oss-fuzz is linked against libc++ instead of stlibc++ which meant building libllvm from source with ASAN instrumentation and linking against libc++ etc. There where a bunch of other issue getting it to work in oss-fuzz as well but that's not really important here.

At some point I've lost track of some important change between the environments and it's not reproducibly building on a regular linux install which is far from ideal, and I'm sure there is some small change somewhere that could fix the problem.

So I had a quick look into this on my system again, and it looks like you are correct in that it doesn't just build straight off the bat. But I was able to build on my Ubuntu 20.04 machine using the follow commands;

export CC=clang-15
export CXX=clang++15
export CFLAGS="-fsanitize=fuzzer-no-link,address"
export CXXFLAGS="-fsanitize=fuzzer-no-link,address"
export LDFLAGS="-fsanitize=address"

cmake -G Ninja  -S . -B build -DCMAKE_BUILD_TYPE=Debug \
   -DLLVM_DIR=$LLVM_ROOT/lib/cmake/LLVM/LLVMConfig.cmake \
   -DTARGET_WEBASSEMBLY=OFF \
   -DWITH_TUTORIALS=OFF \
   -DWITH_UTILS=OFF \
   -DWITH_PYTHON_BINDINGS=OFF \
   -DWITH_TESTS=ON \
   -DWITH_TEST_AUTO_SCHEDULE=OFF \
   -DWITH_TEST_CORRECTNESS=OFF  \
   -DWITH_TEST_ERROR=OFF \
   -DWITH_TEST_WARNING=OFF \
   -DWITH_TEST_PERFORMANCE=OFF \
   -DWITH_TEST_RUNTIME=OFF \
   -DWITH_TEST_GENERATOR=OFF \
   -DWITH_TEST_FUZZ=ON  \
   -DBUILD_SHARED_LIBS=OFF

cmake --build ./build -j$(nproc)

Note that I installed clang-15 and llvm-15 from the llvm apt repositories

If you can confirm that it works at least on your system I'd be happy to open a PR documenting the process/improving the situation :)

[steven-johnson]

Note that I installed clang-15 and llvm-15 from the llvm apt repositories

That really only works well for Linux systems; Halide has significant developer usage on macOS (and also Windows) so a Linux-only solution is not really ideal. Maybe it can be made to work by pointing CC etc at the locally-built Clang from our captive LLVM? I'll give that a try.

[steven-johnson]

Nope: on my macOS laptop, using the above but with CC and CXX pointing at LLVM_ROOT/bin/clang[++], I fail with

CMake Error at /Users/srj/.pyenv/versions/3.10.10/lib/python3.10/site-packages/cmake/data/share/cmake-3.26/Modules/CMakeTestCCompiler.cmake:67 (message):
  The C compiler

    "/Users/srj/llvm-17-install/bin/clang-17"

  is not able to compile a simple test program.

  It fails with the following output:

    Change Dir: /Users/srj/GitHub/Halide/build/CMakeFiles/CMakeScratch/TryCompile-Ak45mN

    Run Build Command(s):/Users/srj/.pyenv/versions/3.10.10/bin/ninja -v cmTC_54379 && [1/2] /Users/srj/llvm-17-install/bin/clang-17   -fsanitize=fuzzer-no-link,address  -isysroot /Users/srj/Downloads/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.3.sdk -MD -MT CMakeFiles/cmTC_54379.dir/testCCompiler.c.o -MF CMakeFiles/cmTC_54379.dir/testCCompiler.c.o.d -o CMakeFiles/cmTC_54379.dir/testCCompiler.c.o -c /Users/srj/GitHub/Halide/build/CMakeFiles/CMakeScratch/TryCompile-Ak45mN/testCCompiler.c
    [2/2] : && /Users/srj/llvm-17-install/bin/clang-17 -fsanitize=fuzzer-no-link,address  -isysroot /Users/srj/Downloads/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.3.sdk -Wl,-search_paths_first -Wl,-headerpad_max_install_names -fsanitize=address CMakeFiles/cmTC_54379.dir/testCCompiler.c.o -o cmTC_54379   && :
    FAILED: cmTC_54379
    : && /Users/srj/llvm-17-install/bin/clang-17 -fsanitize=fuzzer-no-link,address  -isysroot /Users/srj/Downloads/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.3.sdk -Wl,-search_paths_first -Wl,-headerpad_max_install_names -fsanitize=address CMakeFiles/cmTC_54379.dir/testCCompiler.c.o -o cmTC_54379   && :
    ld: file not found: /Users/srj/llvm-17-install/lib/clang/17/lib/darwin/libclang_rt.asan_osx_dynamic.dylib
    clang-17: error: linker command failed with exit code 1 (use -v to see invocation)
    ninja: build stopped: subcommand failed.

[steven-johnson]

Also fails on my Ubuntu machine with a very similar error. I suspect the issue here is that you need to compile LLVM with ~LLVM_ENABLE_PROJECTS=clang-tools-extra~ LLVM_ENABLE_RUNTIMES="compiler-rt;libcxx;libcxxabi;libunwind" if you want ASAN-related stuff... which we don't currently, because that adds nontrivial extra time to LLVM builds, which are huge already. But maybe that would fix it? Let me try.

[nathaniel-brough]

Well we don't necessarily need ASAN support, it's just probably THE most useful sanitizer to pair with fuzzing as it can detect common high risk vulnerabilities. But we could modify the flags above to just do the minimum instrumentation for the fuzzer e.g.

export CFLAGS="-fsanitize=fuzzer-no-link"
export CXXFLAGS="-fsanitize=fuzzer-no-link"
# export LDFLAGS="-fsanitize=address"
unset LDFLAGS

I think that should work with your current system, but I don't have a mac so I can't really test this.