Open silvergasp opened 9 months ago
DavidKorczynski
commented
9 months ago I can see the CMake was a somewhat recent addition https://github.com/google/fuzztest/commits/main/doc/quickstart-cmake.md -- I wasn't aware it was supported! I'll take a look at supporting this in the coming days!
vrabaud
commented
9 months ago For now, these are the hoops I had to jump through to get it working: https://github.com/AOMediaCodec/libavif/pull/1633/files
Basically just copy what is done at https://github.com/google/oss-fuzz/blob/31ac7244748ea7390015455fb034b1f4eda039d9/infra/base-images/base-builder/compile_fuzztests.sh#L59
On the project side, it also has to handle a local fuzztest folder, as recommended upstream. And you have to symlink that folder to the Dockerfile one, cf ln -s $SRC/fuzztest $SRC/libavif/ext/fuzztest
vrabaud
commented
8 months ago Hi, @DavidKorczynski , if you have any beta version you'd like me to try, please let me know !
DavidKorczynski
commented
7 months ago Apologies for the delay. I will start this today!
DavidKorczynski
commented
7 months ago @vrabaud you got very close but missed a single thing! The following diff fixes it:
$ git diff ./
diff --git a/tests/oss-fuzz/build.sh b/tests/oss-fuzz/build.sh
index a1e6756..968a39d 100755
--- a/tests/oss-fuzz/build.sh
+++ b/tests/oss-fuzz/build.sh
@@ -50,7 +50,9 @@ cmake -G Ninja -DBUILD_SHARED_LIBS=OFF -DAVIF_CODEC_AOM=LOCAL -DAVIF_CODEC_DAV1D
-DAVIF_LOCAL_FUZZTEST=ON \
-DAVIF_LOCAL_GTEST=ON -DAVIF_LOCAL_JPEG=ON -DAVIF_LOCAL_LIBSHARPYUV=ON \
-DAVIF_LIBYUV=LOCAL \-DAVIF_LOCAL_ZLIBPNG=ON \
- -DAVIF_BUILD_TESTS=ON -DAVIF_ENABLE_GTEST=OFF -DAVIF_ENABLE_FUZZTEST=ON ..
+ -DAVIF_BUILD_TESTS=ON -DAVIF_ENABLE_GTEST=OFF -DAVIF_ENABLE_FUZZTEST=ON \
+ -DFUZZTEST_COMPATIBILITY_MODE=libfuzzer \
+ ..
ninjaApply the diff on the code here: https://github.com/vrabaud/libavif/blob/7857b566b2031cb2a66528985fe68b0219b89369/tests/oss-fuzz/build.sh#L48-L54
The explanation is that OSS-Fuzz relies on FuzzTest's ability to be compatible with libFuzzer, but you must tell FuzzTest this: https://github.com/google/fuzztest/blob/9c6d264c9bd77c2dcff17424e7e40ccb49f8e09d/CMakeLists.txt#L6
In FuzzTest builds on OSS-Fuzz that uses Bazel this is achieved by using the bazel OSS-Fuzz configurations from FuzzTest (https://github.com/google/fuzztest/blob/9c6d264c9bd77c2dcff17424e7e40ccb49f8e09d/bazel/setup_configs.sh#L110-L168) which specifically sets the relevant option here: https://github.com/google/fuzztest/blob/9c6d264c9bd77c2dcff17424e7e40ccb49f8e09d/bazel/setup_configs.sh#L116
Once you've added the patch, you can run the fuzzers again and you will see output that is output from libFuzzer:
$ python3 infra/helper.py run_fuzzer libavif avif_fuzztest_enc_dec_incr@EncodeDecodeAvifFuzzTest.EncodeDecodeGridValid
INFO:__main__:Running: docker run --rm --privileged --shm-size=2g --platform linux/amd64 -i -e FUZZING_ENGINE=libfuzzer -e SANITIZER=address -e RUN_FUZZER_MODE=interactive -e HELPER=True -v /home/dav/code/2023/fuzztest-cmake/oss-fuzz/build/out/libavif:/out -t gcr.io/oss-fuzz-base/base-runner run_fuzzer avif_fuzztest_enc_dec_incr@EncodeDecodeAvifFuzzTest.EncodeDecodeGridValid.
/out/avif_fuzztest_enc_dec_incr@EncodeDecodeAvifFuzzTest.EncodeDecodeGridValid -rss_limit_mb=2560 -timeout=25 /tmp/avif_fuzztest_enc_dec_incr@EncodeDecodeAvifFuzzTest.EncodeDecodeGridValid_corpus < /dev/null
Note: Google Test filter = EncodeDecodeAvifFuzzTest.EncodeDecodeGridValid
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from EncodeDecodeAvifFuzzTest
[ RUN ] EncodeDecodeAvifFuzzTest.EncodeDecodeGridValid
FUZZTEST_PRNG_SEED=i_RIzcSrHuZv8MgWT0aD7RqTzandDuOcWpXLCZ40vzE
INFO: found LLVMFuzzerCustomMutator (0x69e6f0). Disabling -len_control by default.
INFO: libFuzzer ignores flags that start with '--'
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 4111442667
INFO: Loaded 1 modules (188028 inline 8-bit counters): 188028 [0x35630e0, 0x3590f5c),
INFO: Loaded 1 PC tables (188028 PCs): 188028 [0x3590f60,0x386f720),
INFO: 0 files found in /tmp/avif_fuzztest_enc_dec_incr@EncodeDecodeAvifFuzzTest.EncodeDecodeGridValid_corpus
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
INFO: A corpus is not provided, starting from an empty corpus
#2 INITED cov: 1 ft: 1 corp: 1/1b exec/s: 0 rss: 50Mb
Mutated data is larger than the limit. Returning the original data.
NEW_FUNC[1/109]: 0x514c70 in avif::testutil::ImageToGrid(avifImage const*, unsigned int, unsigned int) /src/libavif/tests/gtest/aviftest_helpers.cc:581
NEW_FUNC[2/109]: 0x531d10 in avif::testutil::(anonymous namespace)::EncodeDecodeGridValid(std::__1::unique_ptr<avifImage, avif::UniquePtrDeleter>, std::__1::unique_ptr<avifEncoder, avif::UniquePtrDeleter>, std::__1::unique_ptr<avifDecoder, avif::UniquePtrDeleter>, unsigned int, unsigned int, bool, bool) /src/libavif/tests/gtest/avif_fuzztest_enc_dec_incr.cc:30
#4 NEW cov: 433 ft: 435 corp: 2/724b lim: 4096 exec/s: 2 rss: 81Mb L: 723/723 MS: 2 Custom-Custom-
#5 NEW cov: 433 ft: 436 corp: 3/1447b lim: 4096 exec/s: 2 rss: 81Mb L: 723/723 MS: 1 Custom-
#6 NEW cov: 434 ft: 437 corp: 4/2170b lim: 4096 exec/s: 3 rss: 81Mb L: 723/723 MS: 1 Custom-
#8 pulse cov: 434 ft: 438 corp: 4/2170b lim: 4096 exec/s: 4 rss: 81Mb
#8 NEW cov: 435 ft: 438 corp: 5/2894b lim: 4096 exec/s: 4 rss: 81Mb L: 724/724 MS: 2 Custom-Custom-
#16 pulse cov: 435 ft: 438 corp: 5/2894b lim: 4096 exec/s: 8 rss: 81Mb
#20 NEW cov: 438 ft: 441 corp: 6/3618b lim: 4096 exec/s: 10 rss: 81Mb L: 724/724 MS: 2 Custom-Custom-
#32 pulse cov: 438 ft: 441 corp: 6/3618b lim: 4096 exec/s: 16 rss: 81Mb
#35 NEW cov: 438 ft: 442 corp: 7/4350b lim: 4096 exec/s: 17 rss: 81Mb L: 732/732 MS: 5 Custom-Custom-Custom-Custom-Custom-
#39 NEW cov: 451 ft: 468 corp: 8/5121b lim: 4096 exec/s: 19 rss: 81Mb L: 771/771 MS: 4 Custom-Custom-Custom-Custom-
#64 pulse cov: 451 ft: 468 corp: 8/5121b lim: 4096 exec/s: 32 rss: 81Mb
#80 REDUCE cov: 451 ft: 468 corp: 8/5120b lim: 4096 exec/s: 40 rss: 81Mb L: 770/770 MS: 1 Custom-
NEW_FUNC[1/569]: 0x515ac0 in avif::testutil::UniquePtrToRawPtr(std::__1::vector<std::__1::unique_ptr<avifImage, avif::UniquePtrDeleter>, std::__1::allocator<std::__1::unique_ptr<avifImage, avif::
UniquePtrDeleter> > > const&) /src/libavif/tests/gtest/aviftest_helpers.cc:624
NEW_FUNC[2/569]: 0x515f20 in std::__1::vector<avifImage const*, std::__1::allocator<avifImage const*> >::reserve(unsigned long) /usr/local/bin/../include/c++/v1/vector:1483
#121 NEW cov: 5092 ft: 5151 corp: 9/5847b lim: 4096 exec/s: 60 rss: 88Mb L: 727/770 MS: 5 Custom-Custom-Custom-Custom-Custom-
#128 pulse cov: 5092 ft: 5151 corp: 9/5847b lim: 4096 exec/s: 64 rss: 88Mb
#152 REDUCE cov: 5092 ft: 5151 corp: 9/5846b lim: 4096 exec/s: 76 rss: 88Mb L: 731/770 MS: 1 Custom-
#158 REDUCE cov: 5092 ft: 5151 corp: 9/5845b lim: 4096 exec/s: 79 rss: 88Mb L: 769/769 MS: 1 Custom-
#180 REDUCE cov: 5092 ft: 5151 corp: 9/5844b lim: 4096 exec/s: 90 rss: 88Mb L: 730/769 MS: 2 Custom-Custom-
#256 pulse cov: 5092 ft: 5151 corp: 9/5844b lim: 4096 exec/s: 128 rss: 88Mb
#272 NEW cov: 5097 ft: 5183 corp: 10/9649b lim: 4096 exec/s: 136 rss: 88Mb L: 3805/3805 MS: 2 Custom-Custom-
#283 NEW cov: 5108 ft: 5247 corp: 11/10379b lim: 4096 exec/s: 141 rss: 91Mb L: 730/3805 MS: 1 Custom-Note that there are still some (3) of the fuzzers that won't get passed check_build after you have deployed the diff above, since they don't succeed the first fuzz iteration -- is this something in the fuzzers?
vrabaud
commented
7 months ago Thx @DavidKorczynski but I could not get to your result. I have patched as you said (cf https://github.com/AOMediaCodec/libavif/compare/main...vrabaud:libavif:oss_fuzz) and my oss-fuzz branch has been patched to refer to that branch.
Here is the repro:
git clone https://github.com/vrabaud/oss-fuzz.git
cd oss-fuzz
python3 infra/helper.py build_image libavif
python3 infra/helper.py build_fuzzers libavifAnd I get the following:
python3 infra/helper.py run_fuzzer libavif avif_fuzztest_enc_dec_incr@EncodeDecodeAvifFuzzTest.EncodeDecodeGridValid
INFO:__main__:Running: docker run --rm --privileged --shm-size=2g --platform linux/amd64 -i -e FUZZING_ENGINE=libfuzzer -e SANITIZER=address -e RUN_FUZZER_MODE=interactive -e HELPER=True -v /usr/local/google/home/vrabaud/tmp/oss-fuzz/build/out/libavif:/out -t gcr.io/oss-fuzz-base/base-runner run_fuzzer avif_fuzztest_enc_dec_incr@EncodeDecodeAvifFuzzTest.EncodeDecodeGridValid./out/avif_fuzztest_enc_dec_incr@EncodeDecodeAvifFuzzTest.EncodeDecodeGridValid -rss_limit_mb=2560 -timeout=25 /tmp/avif_fuzztest_enc_dec_incr@EncodeDecodeAvifFuzzTest.EncodeDecodeGridValid_corpus < /dev/null
Note: Google Test filter = EncodeDecodeAvifFuzzTest.EncodeDecodeGridValid
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from EncodeDecodeAvifFuzzTest [ RUN ] EncodeDecodeAvifFuzzTest.EncodeDecodeGridValid
FUZZTEST_PRNG_SEED=Ha3FS2wsFuz1GPmiHlwJKXVA3aXVHTraBb4Nlo6O8zg
[!] To fuzz, please build with --config=fuzztest.
/src/libavif/ext/fuzztest/./fuzztest/googletest_adaptor.h:62: Failure Expected equality of these values: 0 test->RunInFuzzingMode(argc_, argv_)
Which is: 1
Fuzzing failure.
=================================================================
=== Fuzzing stats
Elapsed time: 227.183us
Total runs: 0
Edges covered: 0
Total edges: 0
Corpus size: 0
Max stack used: 0
=================================================================
=== SETUP FAILURE! It does run with libfuzzer, but it seems to be compiled without compatibility.
DavidKorczynski
commented
7 months ago Ah, apologies @vrabaud -- I thought that did the job but the problem is we need FUZZTEST_COMPATIBILITY_MODE when in the compiler flags when building fuzzers as well:
$ git diff ./
diff --git a/tests/oss-fuzz/build.sh b/tests/oss-fuzz/build.sh
index c4819e9..921539b 100755
--- a/tests/oss-fuzz/build.sh
+++ b/tests/oss-fuzz/build.sh
@@ -42,6 +42,8 @@ ln -s $SRC/fuzztest $SRC/libavif/ext/fuzztest
cd ext && bash aom.cmd && bash dav1d.cmd && bash googletest.cmd && bash libjpeg.cmd && \
bash libsharpyuv.cmd && bash libyuv.cmd && bash zlibpng.cmd && cd ..
+export CXXFLAGS="${CXXFLAGS} -DFUZZTEST_COMPATIBILITY_MODE"
+
# build libavif
mkdir build
cd buildI had this in my own patch but didn't think it was needed -- it is :)
vrabaud
commented
7 months ago I confirm it works, thx ! The failures are due to TEST_DATA_DIRS not being specified but that is very specific to libavif so I'll debug that.
Now, how do we get what I did upstream? We can create the scripts from CMake and dump them in the /out folder.
DavidKorczynski
commented
7 months ago Now, how do we get what I did upstream? We can create the scripts from CMake and dump them in the
/outfolder.
Hmm, I'm not entirely sure. In the bazel approach, we created something general but it turned out projects still needed to control various fields of the process. So, the thing that is perhaps the most OSS-Fuzz specific is the loop here: https://github.com/google/oss-fuzz/blob/4016dacde1ae1f2a33a1518a51f2cb6bbcf08896/infra/base-images/base-builder/compile_fuzztests.sh#L64-L80 -- I wonder if it would make sense to make that a standalone script wrap_fuzztest_binary_to_ossfuzz_fuzzers or so, and then perhaps document the need for DFUZZTEST_COMPATIBILITY_MODE?
vrabaud
commented
7 months ago That could all be done in CMake no? For each fuzztest test, you would basically have a function that does:
function(add_fuzztest TEST_NAME)
add_executable(${TEST_NAME} ${ARGN})
set(FUZZTEST_OUT "$ENV{OUT}")
set_target_properties(${TEST_NAME} RUNTIME_OUTPUT_DIRECTORY ${FUZZTEST_OUT})
add_custom_command(OUTPUT FUZZTEST_TESTS
COMMAND ${TEST_NAME} --list_fuzz_tests | cut -d ' ' -f 4
WORKING_DIRECTORY ${FUZZTEST_OUT}
DEPENDS ${TEST_NAME})
foreach(ENTRYPOINT ${FUZZTEST_TESTS})
set(TARGET_FUZZER TEST_NAME@${ENTRYPOINT})
file(WRITE ${OUT}/${TARGET_FUZZER} "#!/bin/sh
# LLVMFuzzerTestOneInput for fuzzer detection.
this_dir=\$(dirname \"\$0\")
chmod +x \$this_dir/$fuzz_basename
${TEST_NAME} --fuzz=${ENTRYPOINT} -- \$@")
file(CHMOD ${OUT}/${TARGET_FUZZER} PERMISSIONS OWNER_EXECUTE)
endforeach()
endfunction()Totally not tested ... You could add more arguments, like env variables you want to pass.
I've been wanting to make use of google/fuzztest which looks like a great tool. There was an effort to get this working with oss-fuzz in #8678. #8678 was closed as completed with #8784. However it seems like #8784 only addressed Bazel specific projects. It'd be great to get this working with cmake, but it's unclear how to do this. Is there anything planned to support this in future?