Build failure on Debian unstable

[rockdaboot]

Not sure what is going on, clang is 6.0.1-9 and used often for fuzzing other projects (also for OSS-Fuzz).

I naively followed the instructions (../curl was fetched from upstream a minute ago)

git clone http://github.com/curl/curl-fuzzer
cd curl-fuzzer/
./mainline.sh ../curl

....

make[2]: Verzeichnis „/tmp/nghttp2/python“ wird betreten
cython -o nghttp2.c nghttp2.pyx
/usr/bin/python setup.py build
running build
running build_ext
building 'nghttp2' extension
creating build
creating build/temp.linux-x86_64-2.7
clang -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -fsanitize=address -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fPIC -I../lib -I../lib/includes -I../lib/includes -I/usr/include/python2.7 -c nghttp2.c -o build/temp.linux-x86_64-2.7/nghttp2.o
creating build/lib.linux-x86_64-2.7
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-A8UpPM/python2.7-2.7.15=. -fstack-protector-strong -Wformat -Werror=format-security -Wl,-z,relro -fsanitize=address -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION build/temp.linux-x86_64-2.7/nghttp2.o -L../lib/.libs -L../lib -L.. -lnghttp2 -o build/lib.linux-x86_64-2.7/nghttp2.so
/usr/bin/ld: ../lib/.libs/libnghttp2.a(nghttp2_session.o): relocation R_X86_64_32S against symbol `__asan_option_detect_stack_use_after_return' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: ../lib/.libs/libnghttp2.a(nghttp2_submit.o): relocation R_X86_64_32S against symbol `__asan_option_detect_stack_use_after_return' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: ../lib/.libs/libnghttp2.a(nghttp2_helper.o): relocation R_X86_64_32S against symbol `__asan_option_detect_stack_use_after_return' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: ../lib/.libs/libnghttp2.a(nghttp2_hd.o): relocation R_X86_64_32S against symbol `__asan_option_detect_stack_use_after_return' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: ../lib/.libs/libnghttp2.a(nghttp2_hd_huffman.o): relocation R_X86_64_32S against symbol `__asan_option_detect_stack_use_after_return' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: ../lib/.libs/libnghttp2.a(nghttp2_http.o): relocation R_X86_64_32S against `.rodata' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: ../lib/.libs/libnghttp2.a(nghttp2_rcbuf.o): relocation R_X86_64_32S against symbol `__asan_option_detect_stack_use_after_return' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: ../lib/.libs/libnghttp2.a(nghttp2_frame.o): relocation R_X86_64_32S against symbol `__asan_option_detect_stack_use_after_return' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: ../lib/.libs/libnghttp2.a(nghttp2_buf.o): relocation R_X86_64_32S against symbol `__asan_option_detect_stack_use_after_return' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: ../lib/.libs/libnghttp2.a(nghttp2_outbound_item.o): relocation R_X86_64_32S against `.rodata' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: final link failed: nonrepresentable section on output
collect2: error: ld returned 1 exit status
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
make[2]: *** [Makefile:544: all-local] Fehler 1
make[2]: Verzeichnis „/tmp/nghttp2/python“ wird verlassen
make[1]: *** [Makefile:567: all-recursive] Fehler 1
make[1]: Verzeichnis „/tmp/nghttp2“ wird verlassen
make: *** [Makefile:476: all] Fehler 2
+ exit 1
tim@ryzen:~/src/curl-fuzzer$ clang --version
clang version 6.0.1-9 (tags/RELEASE_601/final)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

Possibly something is missing, but not to easily detectable from the above messages. I appreciate any help / ideas :-)

[bagder]

Hm. I've built it with exactly that clang version (6.0.1-9) several times (on debian unstable) and I tried it just now again just to be sure. I just run:

./mainline.sh -ccurl -nnghttp2 -oopenssl

And it works fine for me. I use separate checkouts/builds for all components for the fuzzer.

[rockdaboot]

On a different machine, also latest Debian unstable and also a dev machine, I tried with a fresh clone of curl-fuzzer in /tmp.

...
buildconf: OK
+ /tmp/curl-fuzzer/configure
...
+ make
clang++ -DPACKAGE_NAME=\"curl-fuzzer\" -DPACKAGE_TARNAME=\"curl-fuzzer\" -DPACKAGE_VERSION=\"-\" -DPACKAGE_STRING=\"curl-fuzzer\ -\" -DPACKAGE_BUGREPORT=\"a\ suitable\ curl\ mailing\ list:\ https://curl.haxx.se/mail/\" -DPACKAGE_URL=\"\" -DPACKAGE=\"curl-fuzzer\" -DVERSION=\"-\" -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 -DHAVE_DLFCN_H=1 -DLT_OBJDIR=\".libs/\"    -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -g -I/tmp/curl_install/include -I/tmp/curl_install/utfuzzer  -DFUZZ_PROTOCOLS_ALL -fsanitize=address -stdlib=libstdc++ -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -MT curl_fuzzer-curl_fuzzer.o -MD -MP -MF .deps/curl_fuzzer-curl_fuzzer.Tpo -c -o curl_fuzzer-curl_fuzzer.o `test -f 'curl_fuzzer.cc' || echo './'`curl_fuzzer.cc
mv -f .deps/curl_fuzzer-curl_fuzzer.Tpo .deps/curl_fuzzer-curl_fuzzer.Po
clang++ -DPACKAGE_NAME=\"curl-fuzzer\" -DPACKAGE_TARNAME=\"curl-fuzzer\" -DPACKAGE_VERSION=\"-\" -DPACKAGE_STRING=\"curl-fuzzer\ -\" -DPACKAGE_BUGREPORT=\"a\ suitable\ curl\ mailing\ list:\ https://curl.haxx.se/mail/\" -DPACKAGE_URL=\"\" -DPACKAGE=\"curl-fuzzer\" -DVERSION=\"-\" -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 -DHAVE_DLFCN_H=1 -DLT_OBJDIR=\".libs/\"    -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -g -I/tmp/curl_install/include -I/tmp/curl_install/utfuzzer  -DFUZZ_PROTOCOLS_ALL -fsanitize=address -stdlib=libstdc++ -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -MT curl_fuzzer-curl_fuzzer_tlv.o -MD -MP -MF .deps/curl_fuzzer-curl_fuzzer_tlv.Tpo -c -o curl_fuzzer-curl_fuzzer_tlv.o `test -f 'curl_fuzzer_tlv.cc' || echo './'`curl_fuzzer_tlv.cc
mv -f .deps/curl_fuzzer-curl_fuzzer_tlv.Tpo .deps/curl_fuzzer-curl_fuzzer_tlv.Po
clang++ -DPACKAGE_NAME=\"curl-fuzzer\" -DPACKAGE_TARNAME=\"curl-fuzzer\" -DPACKAGE_VERSION=\"-\" -DPACKAGE_STRING=\"curl-fuzzer\ -\" -DPACKAGE_BUGREPORT=\"a\ suitable\ curl\ mailing\ list:\ https://curl.haxx.se/mail/\" -DPACKAGE_URL=\"\" -DPACKAGE=\"curl-fuzzer\" -DVERSION=\"-\" -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 -DHAVE_DLFCN_H=1 -DLT_OBJDIR=\".libs/\"    -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -g -I/tmp/curl_install/include -I/tmp/curl_install/utfuzzer  -DFUZZ_PROTOCOLS_ALL -fsanitize=address -stdlib=libstdc++ -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -MT curl_fuzzer-curl_fuzzer_callback.o -MD -MP -MF .deps/curl_fuzzer-curl_fuzzer_callback.Tpo -c -o curl_fuzzer-curl_fuzzer_callback.o `test -f 'curl_fuzzer_callback.cc' || echo './'`curl_fuzzer_callback.cc
mv -f .deps/curl_fuzzer-curl_fuzzer_callback.Tpo .deps/curl_fuzzer-curl_fuzzer_callback.Po
/bin/bash ./libtool  --tag=CXX   --mode=link clang++ -g -I/tmp/curl_install/include -I/tmp/curl_install/utfuzzer  -DFUZZ_PROTOCOLS_ALL -fsanitize=address -stdlib=libstdc++ -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION   -o curl_fuzzer curl_fuzzer-curl_fuzzer.o curl_fuzzer-curl_fuzzer_tlv.o curl_fuzzer-curl_fuzzer_callback.o /tmp/curl_install/lib/libcurl.la libstandaloneengine.a  -lpthread -lm
libtool: link: clang++ -g -I/tmp/curl_install/include -I/tmp/curl_install/utfuzzer -DFUZZ_PROTOCOLS_ALL -fsanitize=address -stdlib=libstdc++ -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -o curl_fuzzer curl_fuzzer-curl_fuzzer.o curl_fuzzer-curl_fuzzer_tlv.o curl_fuzzer-curl_fuzzer_callback.o  /tmp/curl_install/lib/libcurl.a -L/tmp/curl_install/lib -L/usr/local/lib /tmp/curl_install/lib/libnghttp2.a /usr/local/lib/libidn2.so -lunistring -lrtmp -lpsl -lssl -lcrypto -lldap -llber -lbrotlidec -lz libstandaloneengine.a -lpthread -lm
clang: error: no such file or directory: 'libstandaloneengine.a'
make: *** [Makefile:753: curl_fuzzer] Error 1
+ exit 4
+ exit 1

[rockdaboot]

Just a guess... my 'make' is an alias to 'make -j$(nproc)'. Is there a problem with parallel builds ? Will check later, have to finish some work now :-|

[cmeister2]

I don't believe there's a problem with parallel builds but obviously there could be something subtle going on. Make's usually pretty good at resolving dependencies in order.

libstandaloneengine.a should be built for you if required; it's defined in the Automake at https://github.com/curl/curl-fuzzer/blob/master/Makefile.am#L61 and this generally works on the travis runners and such.

I'd retry with a simple make; if that doesn't work then attaching the debug output from make -d might be the next step.

[rockdaboot]

In curl-fuzzer/ a make clean && make -j1 yields the same result. Attaching the output from make -j1 -d after a make clean

make.log

[rockdaboot]

For me it looks like a missing dependency. I can see FUZZLIBS = libstandaloneengine.a but FUZZLIBS isn't used in Makefile anywhere as dependency.

[rockdaboot]

This fixes it for me - thanks for for your time. 0001-Add-libstandaloneengine.a-as-dependency.txt

[rockdaboot]

BTW, I now see memory leaks in curl - any interest in reporting those ? Or it that expected behavior ?

[cmeister2]

@rockdaboot: I'm surprised that that patch works, because FUZZLIBS looks like it's a leftover bit of code. The actual define we care about is $(LIB_FUZZING_ENGINE), which is already present on your line.

Could you make a similar make.log file for the working build, possibly?

[rockdaboot]

NP

make2.log

[cmeister2]

So, from the broken log:

libtool: link: clang++ -g -I/tmp/curl_install/include -I/tmp/curl_install/utfuzzer -DFUZZ_PROTOCOLS_ALL -fsanitize=address -stdlib=libstdc++ -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -o curl_fuzzer curl_fuzzer-curl_fuzzer.o curl_fuzzer-curl_fuzzer_tlv.o curl_fuzzer-curl_fuzzer_callback.o /tmp/curl_install/lib/libcurl.a -L/tmp/curl_install/lib -L/usr/local/lib /tmp/curl_install/lib/libnghttp2.a /usr/local/lib/libidn2.so -lunistring -lrtmp -lpsl -lssl -lcrypto -lldap -llber -lbrotlidec -lz libstandaloneengine.a -lpthread -lm

From the working log:

libtool: link: clang++ -g -I/tmp/curl_install/include -I/tmp/curl_install/utfuzzer -DFUZZ_PROTOCOLS_ALL -fsanitize=address -stdlib=libstdc++ -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -o curl_fuzzer curl_fuzzer-curl_fuzzer.o curl_fuzzer-curl_fuzzer_tlv.o curl_fuzzer-curl_fuzzer_callback.o /tmp/curl_install/lib/libcurl.a -L/tmp/curl_install/lib -L/usr/local/lib /tmp/curl_install/lib/libnghttp2.a /usr/local/lib/libidn2.so -lunistring -lrtmp -lpsl -lssl -lcrypto -lldap -llber -lbrotlidec -lz libstandaloneengine.a -lpthread -lm

To my eyes those are identical, so I'm at a loss as to why one of them is working and one of them is not... did you do a clean build for the second one?

[rockdaboot]

Yes I did. I can tell you that the broken build didn't generate libstandaloneengine.a at all. You can see that in make.log, it just wasn't built.

I now revert to the previous commit (hard resetting my changes) and try again with make distclean && ./mainline.sh - and same problem again.

Though this looks good:

COMMON_LDADD = /tmp/curl_install/lib/libcurl.la $(LIB_FUZZING_ENGINE) $(CODE_COVERAGE_LIBS)
# Run e.g. "make all LIB_FUZZING_ENGINE=/path/to/libFuzzer.a"
# OSS-Fuzz will define its own value for LIB_FUZZING_ENGINE.
LIB_FUZZING_ENGINE ?= libstandaloneengine.a

[rockdaboot]

If I replace in Makefile.am

-LIB_FUZZING_ENGINE ?= libstandaloneengine.a
+LIB_FUZZING_ENGINE = libstandaloneengine.a

then autoreconf && make works. Also make distclean && ./mainline.sh then works.

Reverting the above brings back the issue.

Looks like LIB_FUZZING_ENGINE is set outside the of 'make' - but I checked my environment, no such variable.

[rockdaboot]

As a PoC, I changed LIB_FUZZING_ENGINEinto LIB_FUZZING_ENGINE2. Same issue, so nothing from my environment.

[cmeister2]

So the idea of LIB_FUZZING_ENGINE ?= libstandaloneengine.a is that if LIB_FUZZING_ENGINE is defined in the environment, that value is used; otherwise, libstandaloneengine.a is used.

Possibly there's some easy way around this. Perhaps a configuration option could be used instead to set this up.

[rockdaboot]

If not setting LIB_FUZZING_ENGINE works for you but not for me, it looks like a regression in GNU make. Of course there are work-arounds as you say... but still it would be good to track down the issue and ask/report on the make ML. But I am currently out of time for such.

But the way LIB_FUZZING_ENGINE ?= is used in Makefile.am looks absolutely fine. Also, renaming that to show there is no environment issue and then using just = with success tells me that there must be a make issue (make 4.2.1-1.2). I did those changes directly in the generated Makefile to eliminate possible automake issues.

BTW, make libstandaloneengine.a perfectly works. After that, a simple make also works.

[Vancir]

make libstandaloneengine.a also works for me, tested on Ubuntu 20.04.5