RUNPATH seems to be lost when using kcov under Docker

[Rantanen]

I've got a binary, that uses RUNPATH to define the location of a dependent dynamic library.

Running the binary on its own in Docker succeeds. Running the binary under kcov on my laptop succeeds.

Running the binary under kcov in Docker fails at loading the dynamic library with dlopen. Specifying LD_LIBRARY_PATH fixes this.

[Rantanen]

I've repro'd this using docker:

docker run -it --security-opt apparmor=unconfined --security-opt seccomp=unconfined --cap-add SYS_PTRACE rustlang/rust:nightly bash

With the following commands:

apt-get update
apt-get install libcurl4-openssl-dev libelf-dev libdw-dev cmake binutils-dev libiberty-dev -y
wget https://github.com/SimonKagstrom/kcov/archive/master.tar.gz

tar xzf master.tar.gz
cd kcov-master
mkdir build
cd build
cmake ..
make
make install DESTDIR=../../kcov-build
cd ../..
rm -rf kcov-master

git clone https://github.com/Rantanen/intercom
cd intercom
export PROJECT_DIR=${PWD}
mkdir build
cd build
cmake ..
make

cd bin
./cpp-raw
../../../kcov-build/usr/local/bin/kcov --include-pattern=${PROJECT_DIR} --exclude-pattern=${PROJECT_DIR}/test --verify out ./cpp-raw

[Rantanen]

While playing with this some more, the LD_LIBRARY_PATH trick doesn't seem to trace the dynamic library. :|

[Rantanen]

To make this more annoying, the issue seems to be related to how the original binary was compiled as well.

Test compile	kcov	no kcov
Host	OK	OK
Docker	FAIL	OK

If I compile the binary on the host but run kcov in Docker, everything works fine. However a Docker-compiled binary works fine on its own as well.

[SimonKagstrom]

Sorry for the late reply. Thanks a lot for the detailed analysis!

The dependency on the host build sounds very strange, I can't explain that.

That said, kcov does override dlopen: see solib-parser/lib.c, which is loaded using LD_PRELOAD from solib-handler.cc, so it might somehow be related to that.

I have not yet tried to reproduce it myself. Does the kcov compilation output look the same on docker and on the host? I.e., could there be something in the docker environment that messes up the build somehow?

[Rantanen]

Oh dammit. I'm pretty sure I had a response written to the last question, but probably never hit 'Comment'. :|

I remember going through the compilation logs - at least on the cmake output level and remember them being identical.

Here is the one from the docker container that doesn't work. For some reason I'm now unable to run cmake (due to missing TargetArch) on the host machine. Can't remember how I got it compiled the first time.

-- The C compiler identification is GNU 6.3.0
-- The CXX compiler identification is GNU 6.3.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found PkgConfig: /usr/bin/pkg-config (found version "0.29")
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Found LIBBFD: /usr/lib/x86_64-linux-gnu/libbfd.so
-- Target architectures: x86_64
-- Checking for module 'zlib'
--   Found zlib, version 1.2.8
-- Checking for module 'libcurl'
--   Found libcurl, version 7.52.1
-- Found LibElf: /usr/lib/x86_64-linux-gnu/libelf.so
-- Found ElfUtils: /usr/lib/x86_64-linux-gnu/libdw.so
-- Configuring done
-- Generating done
-- Build files have been written to: /kcov-master/build
root@50d828b94230:/kcov-master/build# make
Scanning dependencies of target kcov_system_lib
[  1%] Building CXX object src/CMakeFiles/kcov_system_lib.dir/engines/system-mode-binary-lib.cc.o
[  2%] Building CXX object src/CMakeFiles/kcov_system_lib.dir/utils.cc.o
[  4%] Building CXX object src/CMakeFiles/kcov_system_lib.dir/system-mode/registration.cc.o
[  5%] Linking CXX shared library libkcov_system_lib.so
[  5%] Built target kcov_system_lib
Scanning dependencies of target bash_execve_redirector
[  6%] Building C object src/CMakeFiles/bash_execve_redirector.dir/engines/bash-execve-redirector.c.o
[  8%] Linking C shared library libbash_execve_redirector.so
[  8%] Built target bash_execve_redirector
[  9%] Generating version.c
Scanning dependencies of target kcov-system-daemon
[ 11%] Building CXX object src/CMakeFiles/kcov-system-daemon.dir/configuration.cc.o
[ 12%] Building CXX object src/CMakeFiles/kcov-system-daemon.dir/dummy-solib-handler.cc.o
[ 13%] Building CXX object src/CMakeFiles/kcov-system-daemon.dir/engine-factory.cc.o
[ 15%] Building CXX object src/CMakeFiles/kcov-system-daemon.dir/engines/system-mode-file-format.cc.o
[ 16%] Building CXX object src/CMakeFiles/kcov-system-daemon.dir/engines/ptrace.cc.o
[ 18%] Building CXX object src/CMakeFiles/kcov-system-daemon.dir/filter.cc.o
[ 19%] Building CXX object src/CMakeFiles/kcov-system-daemon.dir/gcov.cc.o
[ 20%] Building CXX object src/CMakeFiles/kcov-system-daemon.dir/main-system-daemon.cc.o
[ 22%] Building CXX object src/CMakeFiles/kcov-system-daemon.dir/parser-manager.cc.o
[ 23%] Building CXX object src/CMakeFiles/kcov-system-daemon.dir/system-mode/file-data.cc.o
[ 25%] Building CXX object src/CMakeFiles/kcov-system-daemon.dir/system-mode/registration.cc.o
[ 26%] Building CXX object src/CMakeFiles/kcov-system-daemon.dir/utils.cc.o
[ 27%] Building C object src/CMakeFiles/kcov-system-daemon.dir/version.c.o
[ 29%] Linking CXX executable kcov-system-daemon
[ 29%] Built target kcov-system-daemon
Scanning dependencies of target kcov_sowrapper
[ 30%] Building C object src/CMakeFiles/kcov_sowrapper.dir/solib-parser/phdr_data.c.o
[ 31%] Building C object src/CMakeFiles/kcov_sowrapper.dir/solib-parser/lib.c.o
[ 33%] Linking C shared library libkcov_sowrapper.so
[ 33%] Built target kcov_sowrapper
[ 34%] Generating library.cc
[ 36%] Generating bash-redirector-library.cc
[ 37%] Generating python-helper.cc
[ 38%] Generating bash-helper.cc
[ 40%] Generating kcov-system-library.cc
[ 41%] Generating html-data-files.cc
Scanning dependencies of target kcov
[ 43%] Building CXX object src/CMakeFiles/kcov.dir/capabilities.cc.o
[ 44%] Building CXX object src/CMakeFiles/kcov.dir/collector.cc.o
[ 45%] Building CXX object src/CMakeFiles/kcov.dir/configuration.cc.o
[ 47%] Building CXX object src/CMakeFiles/kcov.dir/engine-factory.cc.o
[ 48%] Building CXX object src/CMakeFiles/kcov.dir/engines/bash-engine.cc.o
[ 50%] Building CXX object src/CMakeFiles/kcov.dir/engines/system-mode-engine.cc.o
[ 51%] Building CXX object src/CMakeFiles/kcov.dir/engines/system-mode-file-format.cc.o
[ 52%] Building CXX object src/CMakeFiles/kcov.dir/engines/python-engine.cc.o
[ 54%] Building CXX object src/CMakeFiles/kcov.dir/filter.cc.o
[ 55%] Building CXX object src/CMakeFiles/kcov.dir/gcov.cc.o
[ 56%] Building CXX object src/CMakeFiles/kcov.dir/main.cc.o
[ 58%] Building CXX object src/CMakeFiles/kcov.dir/merge-file-parser.cc.o
[ 59%] Building CXX object src/CMakeFiles/kcov.dir/output-handler.cc.o
[ 61%] Building CXX object src/CMakeFiles/kcov.dir/parsers/bfd-disassembler.cc.o
[ 62%] Building CXX object src/CMakeFiles/kcov.dir/parser-manager.cc.o
[ 63%] Building CXX object src/CMakeFiles/kcov.dir/reporter.cc.o
[ 65%] Building CXX object src/CMakeFiles/kcov.dir/source-file-cache.cc.o
[ 66%] Building CXX object src/CMakeFiles/kcov.dir/utils.cc.o
[ 68%] Building CXX object src/CMakeFiles/kcov.dir/writers/cobertura-writer.cc.o
[ 69%] Building CXX object src/CMakeFiles/kcov.dir/writers/json-writer.cc.o
[ 70%] Building CXX object src/CMakeFiles/kcov.dir/writers/coveralls-writer.cc.o
[ 72%] Building CXX object src/CMakeFiles/kcov.dir/writers/html-writer.cc.o
[ 73%] Building CXX object src/CMakeFiles/kcov.dir/writers/sonarqube-xml-writer.cc.o
[ 75%] Building CXX object src/CMakeFiles/kcov.dir/writers/writer-base.cc.o
[ 76%] Building CXX object src/CMakeFiles/kcov.dir/engines/clang-coverage-engine.cc.o
[ 77%] Building CXX object src/CMakeFiles/kcov.dir/engines/ptrace.cc.o
[ 79%] Building CXX object src/CMakeFiles/kcov.dir/engines/kernel-engine.cc.o
[ 80%] Building CXX object src/CMakeFiles/kcov.dir/parsers/elf.cc.o
[ 81%] Building CXX object src/CMakeFiles/kcov.dir/parsers/elf-parser.cc.o
[ 83%] Building CXX object src/CMakeFiles/kcov.dir/parsers/dwarf.cc.o
[ 84%] Building CXX object src/CMakeFiles/kcov.dir/solib-handler.cc.o
[ 86%] Building C object src/CMakeFiles/kcov.dir/solib-parser/phdr_data.c.o
[ 87%] Building CXX object src/CMakeFiles/kcov.dir/system-mode/file-data.cc.o
[ 88%] Building CXX object src/CMakeFiles/kcov.dir/library.cc.o
[ 90%] Building CXX object src/CMakeFiles/kcov.dir/bash-redirector-library.cc.o
[ 91%] Building CXX object src/CMakeFiles/kcov.dir/python-helper.cc.o
[ 93%] Building CXX object src/CMakeFiles/kcov.dir/bash-helper.cc.o
[ 94%] Building CXX object src/CMakeFiles/kcov.dir/kcov-system-library.cc.o
[ 95%] Building CXX object src/CMakeFiles/kcov.dir/html-data-files.cc.o
[ 97%] Building C object src/CMakeFiles/kcov.dir/version.c.o
[ 98%] Linking CXX executable kcov
[100%] Built target kcov