search

google / oss-fuzz

OSS-Fuzz - continuous fuzzing for open source software.
https://google.github.io/oss-fuzz
Apache License 2.0
10.32k stars 2.2k forks source link

[libsndfile] Coverage build failure #4459

Closed cmeister2 closed 3 years ago

cmeister2 commented 3 years ago

Hi,

I'm trying to work out why libsndfile keeps failing its coverage build:

I tried some diagnosis in 25673 but didn't see anything obvious - timestamps added by ts:

:~/code/github/cmeister2/oss-fuzz$  python infra/helper.py coverage --port 8009 libsndfile 2>&1 | ts
Sep 16 14:09:36 Running sndfile_fuzzer
Sep 16 15:09:36 Error occured while running sndfile_fuzzer:
Sep 16 15:09:36 INFO: Seed: 4206533063
Sep 16 15:09:36 INFO: Loaded 1 modules   (20 inline 8-bit counters): 20 [0x7be955, 0x7be969),
Sep 16 15:09:36 INFO: Loaded 1 PC tables (20 PCs): 20 [0x552f30,0x553070),
Sep 16 15:09:36 MERGE-OUTER: 15663 files, 0 in the initial corpus, 0 processed earlier
Sep 16 15:09:36 MERGE-OUTER: attempt 1
Sep 16 15:09:36 ==27== libFuzzer: run interrupted; exiting
Sep 16 15:09:36 ==27== libFuzzer: run interrupted; exiting
Sep 16 15:09:36 du: cannot access '/out/dumps/sndfile_fuzzer.*.profraw': No such file or directory
Sep 16 15:09:36 error: /out/dumps/*.profdata: No such file or directory
Sep 16 15:09:36 Downloading corpora for libsndfile project to /home/md3/code/github/cmeister2/oss-fuzz/build/corpus/libsndfile
Sep 16 15:09:36 Running: docker run --rm --privileged -i -e FUZZING_ENGINE=libfuzzer -e FUZZING_LANGUAGE=c -e PROJECT=libsndfile -e SANITIZER=coverage -e HTTP_PORT=8009 -e COVERAGE_EXTRA_ARGS= -p 8009:8009 -v /home/md3/code/github/cmeister2/oss-fuzz/build/corpus/libsndfile:/corpus -v /home/md3/code/github/cmeister2/oss-fuzz/build/out/libsndfile:/out -t gcr.io/oss-fuzz-base/base-runner coverage
Sep 16 15:09:36 Failed to generate clang code coverage report.

Is there anything I can do to diagnose this?

inferno-chromium commented 3 years ago

We had some coverage issues in llvm recently - https://github.com/google/oss-fuzz/issues/4348, but they should be fixed now. Are you still seeing this. We can keep it open for a week and if you dont see it again, please close the issue ?

cmeister2 commented 3 years ago

Ok. We were seeing this as of 2 days ago (I note #4348 was closed 13 days ago). https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=25793 succeeded this morning, so let's keep this open for a week and see if any more failures are reported.

cmeister2 commented 3 years ago

@inferno-chromium looks like it's failing again: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=25954

Dor1s commented 3 years ago

Let me see what's going on.

Dor1s commented 3 years ago

Regarding

Sep 16 15:09:36 ==27== libFuzzer: run interrupted; exiting

Did you do anything to interrupt the fuzz target?

Looking at the fuzzer stats, coverage job succeeds sometimes: https://oss-fuzz.com/fuzzer-stats?group_by=by-day&date_start=2020-09-01&date_end=2020-09-24&fuzzer=libFuzzer_libsndfile_sndfile_fuzzer&job=libfuzzer_asan_libsndfile&project=libsndfile

My first guess was that it's super slow and often runs into timeout, but performance report shows some other problems instead, which might also be affecting the coverage report generation: https://oss-fuzz.com/performance-report/libFuzzer_libsndfile_sndfile_fuzzer/libfuzzer_asan_libsndfile/2020-09-24

Dor1s commented 3 years ago

Looks like there are some spontaneous crashes, e.g.:

root@bfe2a1a2c142:/out# ./sndfile_fuzzer -merge=1 ./dummy_corpus_dir_for_sndfile_fuzzer/ ./c   
INFO: Seed: 2620014531
INFO: Loaded 1 modules   (20 inline 8-bit counters): 20 [0x7bd420, 0x7bd434), 
INFO: Loaded 1 PC tables (20 PCs): 20 [0x560798,0x5608d8), 
MERGE-OUTER: 15792 files, 2 in the initial corpus, 0 processed earlier
MERGE-OUTER: attempt 1
INFO: Seed: 2620088037
INFO: Loaded 1 modules   (20 inline 8-bit counters): 20 [0x7bd420, 0x7bd434), 
INFO: Loaded 1 PC tables (20 PCs): 20 [0x560798,0x5608d8), 
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 1048576 bytes
MERGE-INNER: using the control file '/tmp/libFuzzerTemp.Merge37.txt'
MERGE-INNER: 15792 total files; 0 processed earlier; will process 15792 files now
#1  pulse  cov: 6 ft: 6 exec/s: 0 rss: 30Mb
#2  pulse  cov: 7 ft: 7 exec/s: 0 rss: 30Mb
#2  LOADED cov: 7 ft: 7 exec/s: 0 rss: 30Mb
#4  pulse  cov: 7 ft: 7 exec/s: 0 rss: 30Mb
#8  pulse  cov: 7 ft: 7 exec/s: 0 rss: 30Mb
#16 pulse  cov: 7 ft: 7 exec/s: 0 rss: 30Mb
#32 pulse  cov: 9 ft: 9 exec/s: 0 rss: 30Mb
#64 pulse  cov: 12 ft: 16 exec/s: 0 rss: 30Mb
#128    pulse  cov: 12 ft: 21 exec/s: 0 rss: 30Mb
#256    pulse  cov: 13 ft: 22 exec/s: 0 rss: 31Mb
Not a valid error number (666).
Not a valid error number (666).
#512    pulse  cov: 13 ft: 22 exec/s: 0 rss: 31Mb
Not a valid error number (666).
#1024   pulse  cov: 13 ft: 22 exec/s: 0 rss: 31Mb
UndefinedBehaviorSanitizer:DEADLYSIGNAL
==40==ERROR: UndefinedBehaviorSanitizer: SEGV on unknown address 0x0000012d5000 (pc 0x7f7240b712a3 bp 0x7fffc26607c0 sp 0x7fffc2660788 T40)
==40==The signal is caused by a WRITE memory access.
    #0 0x7f7240b712a3  (/lib/x86_64-linux-gnu/libc.so.6+0x14e2a3)
    #1 0x4b01cc in memcpy /usr/include/x86_64-linux-gnu/bits/string3.h:53:10
    #2 0x4b01cc in vfread(void*, long, void*) /src/libsndfile/ossfuzz/sndfile_fuzzer.cc:54:3
    #3 0x4e3569 in psf_fread /src/libsndfile/src/file_io.c:321:10
    #4 0x4c7297 in sds_4byte_read /src/libsndfile/src/sds.c:522:11
    #5 0x4c6916 in sds_init /src/libsndfile/src/sds.c:199:3
    #6 0x4c615e in sds_open /src/libsndfile/src/sds.c:133:15
    #7 0x4b0d7e in psf_open_file /src/libsndfile/src/sndfile.c:3124:13
    #8 0x4b1271 in sf_open_virtual /src/libsndfile/src/sndfile.c:440:9
    #9 0x4aff65 in LLVMFuzzerTestOneInput /src/libsndfile/ossfuzz/sndfile_fuzzer.cc:99:13
    #10 0x441101 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:562:15
    #11 0x44a9aa in fuzzer::Fuzzer::CrashResistantMergeInternalStep(std::__Fuzzer::basic_string<char, std::__Fuzzer::char_traits<char>, std::__Fuzzer::allocator<char> > const&) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerMerge.cpp:231:5
    #12 0x432505 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:874:8
    #13 0x45a4f2 in main /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerMain.cpp:20:10
    #14 0x7f7240a4383f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2083f)
    #15 0x407708 in _start (/out/sndfile_fuzzer+0x407708)

UndefinedBehaviorSanitizer can not provide additional info.
SUMMARY: UndefinedBehaviorSanitizer: SEGV (/lib/x86_64-linux-gnu/libc.so.6+0x14e2a3) 
==40==ABORTING
MS: 0 sndfile_fuzzer: malloc.c:2401: sysmalloc: Assertion `(old_top == initial_top (av) && old_size == 0) || ((unsigned long) (old_size) >= MINSIZE && prev_inuse (old_top) && ((unsigned long) old_end & (pagesize - 1)) == 0)' failed.
==40== ERROR: libFuzzer: deadly signal
    #0 0x4ae6f0 in __sanitizer_print_stack_trace /src/llvm-project/compiler-rt/lib/ubsan/ubsan_diag_standalone.cpp:33:3
    #1 0x459d18 in fuzzer::PrintStackTrace() /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerUtil.cpp:210:5
    #2 0x43fb53 in fuzzer::Fuzzer::CrashCallback() /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:233:3
    #3 0x7f724121c38f  (/lib/x86_64-linux-gnu/libpthread.so.0+0x1138f)
    #4 0x7f7240a58437 in raise (/lib/x86_64-linux-gnu/libc.so.6+0x35437)
    #5 0x7f7240a5a039 in abort (/lib/x86_64-linux-gnu/libc.so.6+0x37039)
    #6 0x7f7240aa02f7  (/lib/x86_64-linux-gnu/libc.so.6+0x7d2f7)
    #7 0x7f7240aa4435  (/lib/x86_64-linux-gnu/libc.so.6+0x81435)
    #8 0x7f7240aa5762  (/lib/x86_64-linux-gnu/libc.so.6+0x82762)
    #9 0x7f7240aa71d3 in malloc (/lib/x86_64-linux-gnu/libc.so.6+0x841d3)
    #10 0x41afe7 in operator new(unsigned long) (/out/sndfile_fuzzer+0x41afe7)
    #11 0x48a7ab in std::__Fuzzer::basic_string<char, std::__Fuzzer::char_traits<char>, std::__Fuzzer::allocator<char> >::push_back(char) (/out/sndfile_fuzzer+0x48a7ab)
    #12 0x42614a in std::__Fuzzer::basic_stringbuf<char, std::__Fuzzer::char_traits<char>, std::__Fuzzer::allocator<char> >::overflow(int) /work/llvm-stage2/projects/compiler-rt/lib/fuzzer/libcxx_fuzzer_x86_64/include/c++/v1/sstream:541:24
    #13 0x46439d in std::__Fuzzer::basic_streambuf<char, std::__Fuzzer::char_traits<char> >::xsputn(char const*, long) (/out/sndfile_fuzzer+0x46439d)
    #14 0x4276e4 in sputn /work/llvm-stage2/projects/compiler-rt/lib/fuzzer/libcxx_fuzzer_x86_64/include/c++/v1/streambuf:229:14
    #15 0x4276e4 in std::__Fuzzer::ostreambuf_iterator<char, std::__Fuzzer::char_traits<char> > std::__Fuzzer::__pad_and_output<char, std::__Fuzzer::char_traits<char> >(std::__Fuzzer::ostreambuf_iterator<char, std::__Fuzzer::char_traits<char> >, char const*, char const*, char const*, std::__Fuzzer::ios_base&, char) /work/llvm-stage2/projects/compiler-rt/lib/fuzzer/libcxx_fuzzer_x86_64/include/c++/v1/locale:1402:26
    #16 0x475964 in std::__Fuzzer::num_put<char, std::__Fuzzer::ostreambuf_iterator<char, std::__Fuzzer::char_traits<char> > >::do_put(std::__Fuzzer::ostreambuf_iterator<char, std::__Fuzzer::char_traits<char> >, std::__Fuzzer::ios_base&, char, unsigned long) const (/out/sndfile_fuzzer+0x475964)
    #17 0x469b6b in std::__Fuzzer::basic_ostream<char, std::__Fuzzer::char_traits<char> >::operator<<(unsigned int) (/out/sndfile_fuzzer+0x469b6b)
    #18 0x45506a in fuzzer::Sha1ToString(unsigned char const*) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerSHA1.cpp:213:57
    #19 0x43ee8e in fuzzer::Fuzzer::DumpCurrentUnit(char const*) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:181:31
    #20 0x43f4fb in DeathCallback /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:193:3
    #21 0x43f4fb in fuzzer::Fuzzer::StaticDeathCallback() /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:173:6
    #22 0x49e1f5 in __sanitizer::Die() /src/llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_termination.cpp:52:5
    #23 0x4a7c2d in __sanitizer::HandleDeadlySignal(void*, void*, unsigned int, void (*)(__sanitizer::SignalContext const&, void const*, __sanitizer::BufferedStackTrace*), void const*) /src/llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_report.cpp:247:3
    #24 0x7f724121c38f  (/lib/x86_64-linux-gnu/libpthread.so.0+0x1138f)
    #25 0x7f7240b712a2  (/lib/x86_64-linux-gnu/libc.so.6+0x14e2a2)
    #26 0x4b01cc in memcpy /usr/include/x86_64-linux-gnu/bits/string3.h:53:10
    #27 0x4b01cc in vfread(void*, long, void*) /src/libsndfile/ossfuzz/sndfile_fuzzer.cc:54:3
    #28 0x4e3569 in psf_fread /src/libsndfile/src/file_io.c:321:10
    #29 0x4c7297 in sds_4byte_read /src/libsndfile/src/sds.c:522:11
    #30 0x4c6916 in sds_init /src/libsndfile/src/sds.c:199:3
    #31 0x4c615e in sds_open /src/libsndfile/src/sds.c:133:15
    #32 0x4b0d7e in psf_open_file /src/libsndfile/src/sndfile.c:3124:13
    #33 0x4b1271 in sf_open_virtual /src/libsndfile/src/sndfile.c:440:9
    #34 0x4aff65 in LLVMFuzzerTestOneInput /src/libsndfile/ossfuzz/sndfile_fuzzer.cc:99:13
    #35 0x441101 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:562:15
    #36 0x44a9aa in fuzzer::Fuzzer::CrashResistantMergeInternalStep(std::__Fuzzer::basic_string<char, std::__Fuzzer::char_traits<char>, std::__Fuzzer::allocator<char> > const&) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerMerge.cpp:231:5
    #37 0x432505 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:874:8
    #38 0x45a4f2 in main /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerMain.cpp:20:10
    #39 0x7f7240a4383f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2083f)
    #40 0x407708 in _start (/out/sndfile_fuzzer+0x407708)

NOTE: libFuzzer has rudimentary signal handlers.
      Combine libFuzzer with AddressSanitizer or similar for better crash reports.
SUMMARY: libFuzzer: deadly signal
MS: 0

It's expected that a fuzz target should not crash on a corpus backup. However, even if it does crash, we still proceed with the coverage job and usually generate at least some report. This case is weird, as fuzz targets seems to hang after the first crash, until I click "ctrl+C" two times, then it proceeds and runs to competion.

You can reproduce this by downloading the latest corpus backup:

$ gsutil cp gs://libsndfile-backup.clusterfuzz-external.appspot.com/corpus/libFuzzer/libsndfile_sndfile_fuzzer/latest.zip libsndfile_sndfile_fuzzer.zip

And then running the fuzz target as follows:

./sndfile_fuzzer -merge=1 ./empty_directory/ ./unpacked_corpus_backup_directory
jonathanmetzman commented 3 years ago

I don't see a failure anymore for this project. Closing.