Some fuzz targets instrumented with honggfuzz seem to crash sometimes

[evverx]

I haven't figured out what's going on yet but it appears some fuzz targets segfault from time to time when they are run with honggfuzz and don't crash when they are run directly. The verifier can't reproduce those crashes either

Crash: saved as '/home/vagrant/systemd/SIGSEGV.PC.7fffe517d7d7.STACK.18f57b0756.CODE.1.ADDR.7fffff7feff8.INSTR.call___0xfffffffffff0d8d9.fuzz'
Sz:4105 Tm:27,835,441us (i/b/h/e/p/c) New:0/0/0/0/0/100, Cur:0/0/0/1/0/2326
Launching verifier for HASH: 18f57b0756 (iteration: 1 out of 5)
[2022-06-08T20:07:05+0000][E][2048098] fuzz_runVerifier():280 Verifier stack mismatch: (original) 18f57b0756 != (new) 0

I'm not sure if it helps but backtraces look like

PID 2046163 - core
TID 2046163:
#0  0x00007fffe517d7b2 dl_iterate_phdr
#1  0x00007fffeeea7239 __wrap_memcmp
#2  0x0000000000432a44 instrumentAddConstMem
#3  0x00007fffeeea72b8 __wrap_memcmp
#4  0x0000000000432ab6 instrumentAddConstMem
#5  0x00007fffeeea72b8 __wrap_memcmp
#6  0x0000000000432a44 instrumentAddConstMem
#7  0x00007fffeeea72b8 __wrap_memcmp
#8  0x0000000000432ad4 instrumentAddConstMem
#9  0x00007fffeeea72b8 __wrap_memcmp
#10 0x0000000000432a44 instrumentAddConstMem
#11 0x00007fffeeea72b8 __wrap_memcmp
...
#116382 0x0000000000432ab6 instrumentAddConstMem
#116383 0x00007fffeeea72b8 __wrap_memcmp
#116384 0x0000000000432ab6 instrumentAddConstMem
#116385 0x00007fffeeea72b8 __wrap_memcmp
#116386 0x0000000000432ad4 instrumentAddConstMem
#116387 0x00007fffeeea72b8 __wrap_memcmp
#116388 0x0000000000432b1e instrumentAddConstMem
#116389 0x00007fffeeea72b8 __wrap_memcmp
#116390 0x000000000043343d instrumentAddConstStrN
#116391 0x0000000000433ee6 __wrap_strncmp
#116392 0x00007fffe57c8f68 startswith
#116393 0x00007fffe567a5a9 documentation_url_is_valid
#116394 0x00007fffeed24013 config_parse_documentation
#116395 0x00007fffe5435569 next_assignment
#116396 0x00007fffe542688f parse_line
#116397 0x00007fffe5424fee config_parse
#116398 0x000000000042f0e7 LLVMFuzzerTestOneInput
#116399 0x000000000042fb13 main
#116400 0x00007fffe5028440 __libc_start_call_main
#116401 0x00007fffe50284f0 __libc_start_main@@GLIBC_2.34
#116402 0x0000000000406785 _start

That particular fuzz target was built with hfuzz-clang/hfuzz-clang++ on Fedora 35 with clang version 13.0.0 (Fedora 13.0.0-3.fc35) with no sanitizers.

[robertswiecki]

Please try with https://github.com/google/honggfuzz/commit/03a377b02525c3658cf9486b975e4dcf9e063833 - it might work, just a guess

[evverx]

Looks like the fuzz target still crashes

PID 2078288 - core
TID 2078288:
#0  0x00007fffe517d7b2     __GI___dl_iterate_phdr - libc.so.6
    /usr/src/debug/glibc-2.34-34.fc35.x86_64/elf/dl-iteratephdr.c:34:1
#1  0x00007fffeeea7299 - 1 __wrap_memcmp - libsystemd-core-251.so
#2  0x0000000000432b16 - 1 instrumentAddConstMem - /home/vagrant/systemd/build/fuzz-unit-file
#3  0x00007fffeeea7318 - 1 __wrap_memcmp - libsystemd-core-251.so
#4  0x0000000000432b7e - 1 instrumentAddConstMem - /home/vagrant/systemd/build/fuzz-unit-file
#5  0x00007fffeeea7318 - 1 __wrap_memcmp - libsystemd-core-251.so
#6  0x0000000000432b16 - 1 instrumentAddConstMem - /home/vagrant/systemd/build/fuzz-unit-file
#7  0x00007fffeeea7318 - 1 __wrap_memcmp - libsystemd-core-251.so
#8  0x0000000000432b16 - 1 instrumentAddConstMem - /home/vagrant/systemd/build/fuzz-unit-file
...
#116371 0x00007fffeeea7318 - 1 __wrap_memcmp - libsystemd-core-251.so
#116372 0x0000000000432b7e - 1 instrumentAddConstMem - /home/vagrant/systemd/build/fuzz-unit-file
#116373 0x00007fffeeea7318 - 1 __wrap_memcmp - libsystemd-core-251.so
#116374 0x0000000000433059 - 1 instrumentAddConstStr - /home/vagrant/systemd/build/fuzz-unit-file
#116375 0x0000000000433849 - 1 __wrap_strcmp - /home/vagrant/systemd/build/fuzz-unit-file
#116376 0x00007fffe57f23f0 - 1 bus_error_name_to_errno - libsystemd-shared-251.so
#116377 0x00007fffe57f27a5 - 1 bus_error_setfv - libsystemd-shared-251.so
#116378 0x00007fffe57f29ba - 1 sd_bus_error_setf - libsystemd-shared-251.so
#116379 0x00007fffeed55b9f - 1 manager_load_unit_prepare - libsystemd-core-251.so
#116380 0x00007fffeed54443 - 1 manager_load_unit - libsystemd-core-251.so
#116381 0x00007fffeee730b9 - 1 unit_add_dependency_by_name - libsystemd-core-251.so
#116382 0x00007fffeed09aa2 - 1 config_parse_unit_deps - libsystemd-core-251.so
#116383 0x00007fffe5435569 - 1 next_assignment - libsystemd-shared-251.so
#116384 0x00007fffe542688f - 1 parse_line - libsystemd-shared-251.so
#116385 0x00007fffe5424fee - 1 config_parse - libsystemd-shared-251.so
#116386 0x000000000042f0e7 - 1 LLVMFuzzerTestOneInput - /home/vagrant/systemd/build/fuzz-unit-file
#116387 0x000000000042fb13 - 1 main - /home/vagrant/systemd/build/fuzz-unit-file
#116388 0x00007fffe5028440 - 1 __libc_start_call_main - libc.so.6
    ../sysdeps/nptl/libc_start_call_main.h:58:16
#116389 0x00007fffe50284f0 - 1 __libc_start_main_impl - libc.so.6
    ../csu/libc-start.c:389:3
#116390 0x0000000000406785 - 1 _start - /home/vagrant/systemd/build/fuzz-unit-file

[robertswiecki]

If you run honggfuzz with -d -l /tmp/log

then you should see something like

[2022-06-09T09:44:23+0200][D][27121] initializeLibcFunctions():93 libc_memcmp=0x7ffff7d744c0, (_memcmp=0xa26f70, memcmp=0xa2c430, __wrap_memcmp=0xa2c430)

in /tmp/log

[evverx]

I can't seem to find initializeLibcFunctions anywhere. I rebuilt honggfuzz with the following patch:

diff --git a/libhfcommon/log.c b/libhfcommon/log.c
index 27b3e444..489156b4 100644
--- a/libhfcommon/log.c
+++ b/libhfcommon/log.c
@@ -50,7 +50,7 @@

 static int             hf_log_fd        = STDERR_FILENO;
 static bool            hf_log_fd_isatty = false;
-enum llevel_t          hf_log_level     = INFO;
+enum llevel_t          hf_log_level     = DEBUG;
 static pthread_mutex_t log_mutex        = PTHREAD_MUTEX_INITIALIZER;

 __attribute__((constructor)) static void log_init(void) {

and compiled a simple C program where main just calls memcpy with hfuzz-clang. When it's run without honggfuzz it shows something like

./a.out
[2022-06-09T13:09:19+0000][D][2104750] initializeInstrument():186 Initializing pid=2104750
[2022-06-09T13:09:19+0000][D][2104750] initializeInstrument():190 The 'HFUZZ_THREAD_NO' envvar is not set
[2022-06-09T13:09:19+0000][D][2104750] __sanitizer_cov_trace_pc_guard_init():588 PC-Guard module initialization: 0x458e08-0x458e0c (count:1) at 1
abc

but with honggfuzz all those logs disappear somehow.

[evverx]

Looks like the logs are shown in persistent mode only.

[2022-06-09T13:59:20+0000][D][2108553] initializeInstrument():186 Initializing pid=2108553
[2022-06-09T13:59:20+0000][D][2108553] initializeLibcFunctions():93 libc_memcmp=0x7ffff7d88180, (_memcmp=0x42f870, memcmp=0x434dc0, __wrap_memcmp=0x434dc0)
[2022-06-09T13:59:20+0000][D][2108553] __sanitizer_cov_trace_pc_guard_init():588 PC-Guard module initialization: 0x459e30-0x459e34 (count:1) at 1
[2022-06-09T13:59:20+0000][D][2108553] fetchIsInputAvailable():90 Current module: ^A_LIBHFUZZ_PERSISTENT_BINARY_SIGNATURE_^Bÿ

I should have probably mentioned that the fuzz target is run in default mode . LLVMFuzzerTestOneInput is probably misleading because it's wrapped into main reading files and passing them to LLVMFuzzerTestOneInput.

[evverx]

Looks like it crashes in persistent mode as well:

[2022-06-09T23:46:00+0000][D][2328799] subproc_New():420 Launched new process, pid=2328882, thread: 0 (concurrency: 1)
[2022-06-09T23:46:00+0000][D][2328882] subproc_New():407 Launching './build/fuzz-unit-file' on file 'PERSISTENT_MODE' (file mode)
[2022-06-09T23:46:00+0000][D][2328882] files_writeBufToFile():86 Written '4' bytes to '/proc/self/oom_score_adj'
[2022-06-09T23:46:00+0000][D][2328799] arch_traceAttach():1055 Attached to PID: 2328882
Persistent mode: Launched new persistent pid=2328882
[2022-06-09T23:46:00+0000][D][2328882] initializeInstrument():186 Initializing pid=2328882
[2022-06-09T23:46:00+0000][D][2328882] initializeLibcFunctions():93 libc_memcmp=0x7fffeeea7250, (_memcmp=0x42f810, memcmp=0x434d60, __wrap_memcmp=0x434d60)
[2022-06-09T23:46:00+0000][D][2328882] __sanitizer_cov_trace_pc_guard_init():588 PC-Guard module initialization: 0x7fffe5af3288-0x7fffe5b45bd0 (count:84562) at 1
[2022-06-09T23:46:00+0000][D][2328882] __sanitizer_cov_trace_pc_guard_init():588 PC-Guard module initialization: 0x7fffeef8c068-0x7fffeefb0854 (count:37371) at 84563
[2022-06-09T23:46:00+0000][D][2328882] __sanitizer_cov_trace_pc_guard_init():588 PC-Guard module initialization: 0x454aa8-0x454b5c (count:45) at 121934
[2022-06-09T23:46:00+0000][D][2328882] fetchIsInputAvailable():90 Current module: ^A_LIBHFUZZ_PERSISTENT_BINARY_SIGNATURE_^Bÿ
Sz:523 Tm:17,022us (i/b/h/e/p/c) New:0/0/0/0/0/120, Cur:0/0/0/1328/0/8289
...
[2022-06-09T23:46:21+0000][D][2328799] arch_checkWait():230 pid=2328882 returned with status: STOPPED with signal: 11 (Segmentation fault)
[2022-06-09T23:46:21+0000][D][2328799] sanitizers_PidForTid():120 Tid 2328882 has Pid 2328882
[2022-06-09T23:46:21+0000][D][2328799] sanitizers_parseReport():139 fopen('/home/vagrant/systemd/HF.sanitizer.log.2328882', 'rb'): No such file or directory
[2022-06-09T23:46:21+0000][D][2328799] arch_traceSaveData():690 Pid: 2328882, signo: 11, errno: 0, code: 1, addr: 0x7fffff7fefe0, pc: 7fffe517d7b2, crashAddr: 7fffff7fefe0 instr: 'mov____%rdi,(%rsp)'
Crash (dup): '/home/vagrant/systemd/SIGSEGV.PC.7fffe517d7b2.STACK.cbf6ad010.CODE.1.ADDR.7fffff7fefe0.INSTR.mov____%rdi,(%rsp).fuzz' already exists, skipping
[2022-06-09T23:46:25+0000][D][2328799] arch_checkWait():230 pid=2328882 returned with status: SIGNALED, signal: 11 (Segmentation fault)
[2022-06-09T23:46:25+0000][W][2328799] arch_checkWait():237 Persistent mode: pid=2328882 exited with status: SIGNALED, signal: 11 (Segmentation fault)
Sz:588 Tm:24,786,273us (i/b/h/e/p/c) New:0/0/0/0/0/213, Cur:0/0/0/615/0/1274

[jonathanmetzman]

I wonder if this is related: https://github.com/google/oss-fuzz/issues/7435

[evverx]

@jonathanmetzman I think that issue is different. As far as I understand OSS-Fuzz doesn't build fuzz targets with hfuzz-clang/hfuzz-clang++ so "-Wl,--wrap=strncmp" (which I suspect causes this particular issue) is never passed there. I'm not 100% sure though.

[robertswiecki]

libhfuzz/memorycmp.o exports e.g. strncmp as a weak symbol, so it can be used w/o --wrap, depending on the static linker used and particulars.

With https://github.com/google/honggfuzz/commit/03a377b02525c3658cf9486b975e4dcf9e063833 I added an additional check, so it makes sure that the memcmp() used inside the library itself, is never the one exported by itself (as a weak symbol), but comparing it to the __wrap_memcmp symbol address.

You're talking about OSS, I suppose we need to upgrade honggfuzz version in oss-fuzz? I haven't done that in a long time, and probably lost the git setup for that. @jonathanmetzman is this something (upgrading honggfuzz to the current HEAD) that you could do given your fluency with oss-code, or would you rather leave it to me?

PS: Updating to whatever current HEAD points to - eg this https://github.com/google/honggfuzz/commit/6efe0eeef8182885ae4b7b1c49b071851bc37b16 - should be fine.

PPS: Ah.. I performed the procedure so long ago, that I've forgotten, that it's now just tag update on the honggfuzz side. Please hold on, I'll try to fix it

PPPS: Ok, done, if this is indeed about oss-fuzz, then this release - https://github.com/google/honggfuzz/releases/tag/oss-fuzz - fetched hopefully automatically by oss-fuzz, should help.

[jonathanmetzman]

I think we always download from: https://github.com/google/honggfuzz/archive/oss-fuzz.tar.gz so your update should be picked up soon. I'll just test things in our CI to make sure it didn't break anything

[jonathanmetzman]

Test: https://github.com/google/oss-fuzz/pull/7878

[evverx]

if this is indeed about oss-fuzz

It's great honggfuzz got updated on OSS-Fuzz :-) Thanks! But as far as I can tell the original issue is still reproducible locally with the tip of the master branch. I switched to -V everywhere to verify crashes though so I just kind of ignore crashes like that.

[jonathanmetzman]

The updated honggfuzz didn't break anything. There were test failures but all due to sourceware.org being down.

[evverx]

@jonathanmetzman speaking of OSS-Fuzz I wonder if it would be possible to switch to hfuzz-clang/hfuzz-clang++ there optionally. The compiler flags honggfuzz passes are helpful when it comes to catching a certain kind of issues like https://github.com/systemd/systemd/issues/23646 even without fuzzing :-) Other than that it would make it easier to reproduce issues locally (where hfuzz-clang is usually used) reliably.

[jonathanmetzman]

I can look the other way if you get it working but I probably wont recommend it. And i don't know if I will make it the default given how it went when AFL did this (it made programs crash at arbitrary/noreproducible times during production fuzzing and not test fuzzing)

[evverx]

i don't know if I will make it the default

I agree the honggfuzz wrappers shouldn't be rolled out by default. It took me a while to get for example systemd and elfutils to even compile so I suspect some projects would just fail to build.

given how it went when AFL did this (it made programs crash at arbitrary/noreproducible times during production fuzzing and not test fuzzing)

I just turned it off at the time as far as I can remember :-) On the bright side all issues I personally ran into were fixed and AFL was brought back.

I can look the other way if you get it working but I probably wont recommend it

Got it

[robertswiecki]

I'll try to repro this today. Shall I simply run it for some time (1h enough?) with compile options/setup taken from oss-fuzz?

[evverx]

Locally the fuzz target crashes in about 5 minutes. It can be reproduced with

git clone --depth=1 https://github.com/systemd/systemd
cd systemd
CC=hfuzz-clang CXX=hfuzz-clang++ meson -Dskip-deps=true build
ninja -C ./build/ fuzz-unit-file
cp test/fuzz/fuzz-unit-file/*.service INPUT/
honggfuzz -i INPUT/ -o OUTPUT -t30 -- ./build/fuzz-unit-file ___FILE___

-Dskip-deps turns off almost all the build dependencies so only meson, ninja, jinja2, libmount-dev and libcap-dev should be installed to get it to compile.

I'm not sure how to reproduce https://github.com/google/oss-fuzz/issues/7435

[robertswiecki]

Changing libc_memcmp to _memcmp here https://github.com/google/honggfuzz/blob/0b47bfdd0df92b332884a83624360ed582eb4aed/libhfuzz/instrument.c#L266 solves it, so it's some kind of circular invocation of memcmp, let me do some debugging then.

[robertswiecki]

The problem was interesting. When two (or more) instrumented by honggfuzz modules (binaries, libraries) end up in one address space, then libc_memcmp will be resolved to the other's module's memcmp (which will in turn call into the original one).

I fixed it temporarily with - https://github.com/google/honggfuzz/commit/a338c5322dea64c468a4419d00daebd635f7ac8b - but I need to think how to do it "for good", as the fix is not pretty.

I updated the oss-fuzz tag too.

[robertswiecki]

PS: I'm not sure how oss-fuzz fuzzes systemd under honggfuzz, but it seems that systemd's fuzzing implements its own main(). This prevents using honggfuzz's main, which in turn disabled the persistent mode. In effect the fuzzing is ~20x as slow as it could be.

If that's the case, maybe we could do something about it. E.g. surrounding systemd's fuzzing main() with e.g.

#ifndef HFND_FUZZING_ENTRY_FUNCTION
int main(...) {

}
#endif  /* HFND_FUZZING_ENTRY_FUNCTION */

or somesuch.

[evverx]

systemd can be fuzzed in a few modes and on OSS-Fuzz LLVMFuzzerTestOneInput isn't wrapped into main. There it's linked with hongfuzz and run in persistent mode. The "main" wrapper is mostly used to run the fuzz targets with corpora accumulated over time under Valgrind or as unit tests or on architectures where ASan doesn't work for various reasons.

[evverx]

I think it would make sense to skip the "main" wrapper when HFND_FUZZING_ENTRY_FUNCTION is defined locally as well. Right now only libFuzzer is supported "natively" by the build script and the other fuzzing engines are slow. Thanks for the pointer!

[robertswiecki]

Thanks for the info. I'll go ahead and close this report, but please re-open if you'd like still to work on that.