search

curl / curl-fuzzer

Quality assurance testing for the curl project
MIT License
82 stars 29 forks source link

Unable to build using mainline.sh with Clang 11 / LLVM 11? #38

Closed geeknik closed 3 years ago

geeknik commented 3 years ago

If I don't touch mainline.sh and just run it, I get to this stage:

/bin/bash ./libtool  --tag=CXX   --mode=link clang++ -g -I/tmp/curl_install/include -I/tmp/curl_install/utfuzzer  -DFUZZ_PROTOCOLS_ALL -fsanitize=address -stdlib=libstdc++ -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION   -o curl_fuzzer curl_fuzzer-curl_fuzzer.o curl_fuzzer-curl_fuzzer_tlv.o curl_fuzzer-curl_fuzzer_callback.o /tmp/curl_install/lib/libcurl.la libstandaloneengine.a  -lpthread -lm
libtool: link: clang++ -g -I/tmp/curl_install/include -I/tmp/curl_install/utfuzzer -DFUZZ_PROTOCOLS_ALL -fsanitize=address -stdlib=libstdc++ -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -o curl_fuzzer curl_fuzzer-curl_fuzzer.o curl_fuzzer-curl_fuzzer_tlv.o curl_fuzzer-curl_fuzzer_callback.o  /tmp/curl_install/lib/libcurl.a -L/tmp/curl_install/lib /tmp/curl_install/lib/libnghttp2.a -lidn2 -lssl -lcrypto -lz libstandaloneengine.a -lpthread -lm
clang: error: no such file or directory: 'libstandaloneengine.a'
make: *** [Makefile:753: curl_fuzzer] Error 1
+ exit 4
+ exit 1

If I update mainline.sh to use -fsanitize=address,fuzzer -fsanitize-coverage=trace-pc-guard, this happens:

/usr/bin/ld: openssl.o: in function `main':
/tmp/openssl/apps/openssl.c:219: multiple definition of `main'; /usr/lib/llvm-11/lib/clang/11.0.0/lib/linux/libclang_rt.fuzzer-x86_64.a(fuzzer.o):(.text.main+0x0): first defined here
/usr/bin/ld: /usr/lib/llvm-11/lib/clang/11.0.0/lib/linux/libclang_rt.fuzzer-x86_64.a(fuzzer.o): in function `main':
(.text.main+0x12): undefined reference to `LLVMFuzzerTestOneInput'
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [../Makefile.shared:164: link_app.] Error 1
make[2]: Leaving directory '/tmp/openssl/apps'
make[1]: *** [Makefile:157: openssl] Error 2
make[1]: Leaving directory '/tmp/openssl/apps'
make: *** [Makefile:293: build_apps] Error 1
+ exit 1

Changing the C/CXX flags to -fsanitize=address,fuzzer-no-link and then passing -fsanitize=address,fuzzer in the LD flags doesn't seem to help:

+ autoreconf -i
+ ./configure --prefix=/tmp/curl_install --disable-shared --enable-static --disable-threads
checking for gcc... clang
checking whether the C compiler works... no
configure: error: in `/tmp/nghttp2':
configure: error: C compiler cannot create executables
See `config.log' for more details
+ exit 1

clang -v

Debian clang version 11.0.0-++20200701115111+4c6683eafc1-1~exp1~20200701095721.865
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/7
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/7.4.0
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/8
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/7
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/7.4.0
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/8
Selected GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/8
Candidate multilib: .;@m64
Selected multilib: .;@m64

dpkg --list | grep clang

ii  clang                            1:11.0-51+0~20200309211112.9~1.gbp214056                    amd64        C, C++ and Objective-C compiler (LLVM based)
ii  clang-11                         1:11~++20200701115111+4c6683eafc1-1~exp1~20200701095721.865 amd64        C, C++ and Objective-C compiler
ii  clang-7                          1:7.0.1-8                                                   amd64        C, C++ and Objective-C compiler
ii  clang-8                          1:8.0.1-3~bpo10+1                                           amd64        C, C++ and Objective-C compiler
ii  clang-tools-11                   1:11~++20200701115111+4c6683eafc1-1~exp1~20200701095721.865 amd64        clang-based tools for C/C++ developments
ii  libclang-11-dev                  1:11~++20200701115111+4c6683eafc1-1~exp1~20200701095721.865 amd64        Clang library - Development package
ii  libclang-8-dev                   1:8.0.1-3~bpo10+1                                           amd64        Clang library - Development package
ii  libclang-common-11-dev           1:11~++20200701115111+4c6683eafc1-1~exp1~20200701095721.865 amd64        Clang library - Common development package
ii  libclang-common-7-dev            1:7.0.1-8                                                   amd64        Clang library - Common development package
ii  libclang-common-8-dev            1:8.0.1-3~bpo10+1                                           amd64        Clang library - Common development package
ii  libclang-cpp11                   1:11~++20200701115111+4c6683eafc1-1~exp1~20200701095721.865 amd64        C++ interface to the Clang library
ii  libclang-cpp11-dev               1:11~++20200701115111+4c6683eafc1-1~exp1~20200701095721.865 amd64        C++ interface to the Clang library
ii  libclang1-11                     1:11~++20200701115111+4c6683eafc1-1~exp1~20200701095721.865 amd64        C interface to the Clang library
ii  libclang1-6.0:amd64              1:6.0.1-10                                                  amd64        C interface to the clang library
ii  libclang1-7                      1:7.0.1-8                                                   amd64        C interface to the Clang library
ii  libclang1-8                      1:8.0.1-3~bpo10+1                                           amd64        C interface to the Clang library

dpkg --list | grep llvm

ii  libllvm11:amd64                  1:11~++20200701115111+4c6683eafc1-1~exp1~20200701095721.865 amd64        Modular compiler and toolchain technologies, runtime library
ii  libllvm6.0:amd64                 1:6.0.1-10                                                  amd64        Modular compiler and toolchain technologies, runtime library
ii  libllvm7:amd64                   1:7.0.1-8                                                   amd64        Modular compiler and toolchain technologies, runtime library
ii  libllvm8:amd64                   1:8.0.1-3~bpo10+1                                           amd64        Modular compiler and toolchain technologies, runtime library
ii  llvm                             1:11.0-51+0~20200309211112.9~1.gbp214056                    amd64        Low-Level Virtual Machine (LLVM)
ii  llvm-11                          1:11~++20200701115111+4c6683eafc1-1~exp1~20200701095721.865 amd64        Modular compiler and toolchain technologies
ii  llvm-11-dev                      1:11~++20200701115111+4c6683eafc1-1~exp1~20200701095721.865 amd64        Modular compiler and toolchain technologies, libraries and headers
ii  llvm-11-runtime                  1:11~++20200701115111+4c6683eafc1-1~exp1~20200701095721.865 amd64        Modular compiler and toolchain technologies, IR interpreter
ii  llvm-11-tools                    1:11~++20200701115111+4c6683eafc1-1~exp1~20200701095721.865 amd64        Modular compiler and toolchain technologies, tools
ii  llvm-7                           1:7.0.1-8                                                   amd64        Modular compiler and toolchain technologies
ii  llvm-7-dev                       1:7.0.1-8                                                   amd64        Modular compiler and toolchain technologies, libraries and headers
ii  llvm-7-runtime                   1:7.0.1-8                                                   amd64        Modular compiler and toolchain technologies, IR interpreter
ii  llvm-8                           1:8.0.1-3~bpo10+1                                           amd64        Modular compiler and toolchain technologies
ii  llvm-8-dev                       1:8.0.1-3~bpo10+1                                           amd64        Modular compiler and toolchain technologies, libraries and headers
ii  llvm-8-doc                       1:8.0.1-3~bpo10+1                                           all          Modular compiler and toolchain technologies, documentation
ii  llvm-8-examples                  1:8.0.1-3~bpo10+1                                           all          Modular compiler and toolchain technologies, examples
ii  llvm-8-runtime                   1:8.0.1-3~bpo10+1                                           amd64        Modular compiler and toolchain technologies, IR interpreter
ii  llvm-8-tools                     1:8.0.1-3~bpo10+1                                           amd64        Modular compiler and toolchain technologies, tools
ii  llvm-dev                         1:11.0-51+0~20200309211112.9~1.gbp214056                    amd64        Low-Level Virtual Machine (LLVM), libraries and headers
ii  llvm-runtime                     1:11.0-51+0~20200309211112.9~1.gbp214056                    amd64        Low-Level Virtual Machine (LLVM), bytecode interpreter

EDIT: I removed all of the clang and llvm artifiacts that weren't from version 11. Didn't change anything.

cmeister2 commented 3 years ago

Intriguing. I'll have to try and find a Docker image with clang 11 on it so I can take a look.

geeknik commented 3 years ago

This repo might help with that. If not, this will work on a Debian or Ubuntu host:

echo "deb http://apt.llvm.org/DISTRO_NAME/ llvm-toolchain-DISTRO_NAME main" >> /etc/apt/sources.list
wget -O - https://apt.llvm.org/llvm-snapshot.gpg.key | apt-key add - 
apt-get update && apt-get upgrade -y
apt-get install -y clang-11 clang-tools-11 libc++1-11 libc++-11-dev \
    libc++abi1-11 libc++abi-11-dev libclang1-11 libclang-11-dev \
    libclang-common-11-dev libclang-cpp11 libclang-cpp11-dev liblld-11 \
    liblld-11-dev liblldb-11 liblldb-11-dev libllvm11 libomp-11-dev \
    libomp5-11 lld-11 lldb-11 llvm-11 llvm-11-dev llvm-11-runtime llvm-11-tools libfuzzer-11-dev
geeknik commented 3 years ago

Same thing happens with Clang 12/LLVM 12. Any thoughts?

geeknik commented 3 years ago

This is no longer an issue I'm having.