search

envoyproxy / envoy

Cloud-native high-performance edge/middle/service proxy
https://www.envoyproxy.io
Apache License 2.0
25.01k stars 4.82k forks source link

segmentation fault from 1.13.0 binary built from centos docker image #9772

Closed keyolk closed 4 years ago

keyolk commented 4 years ago

I got segmentation fault with recent 1.13.0 binary which built with envoy-build-centos docker image. In case of using envoy-build-ubuntu, it all works well.

Basically centos version also works, but if I trying to use below flags, it happens

Used below command to build the binary

$ ENVOY_DOCKER_BUILD_DIR=/tmp/envoy.build \
IMAGE_NAME=envoyproxy/envoy-build-centos \
IMAGE_ID=latest \
./ci/run_envoy_docker.sh './ci/do_ci.sh bazel.release.server_only

based on this commit

$ git log -1
commit bb7ceff4c3c5bd4555dff28b6e56d27f2f8be0a7
Author: Matt Klein <mklein@lyft.com>
Date:   Mon Jan 20 14:06:02 2020 -0800

    docs: final docs for 1.13.0 (#9745)

    Including reorganizing the configuration overview page.

    Signed-off-by: Matt Klein <mklein@lyft.com>

message I got

build_release/envoy --version
Segmentation fault

from gdb

(gdb) r --version
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
0x00005555561e24b2 in endl<char, std::__1::char_traits<char> > (__os=...) at /opt/llvm/bin/../include/c++/v1/ostream:1002
1002    /opt/llvm/bin/../include/c++/v1/ostream: No such file or directory.
Missing separate debuginfos, use: debuginfo-install glibc-2.17-106.el7_2.1.x86_64
(gdb) bt
#0  0x00005555561e24b2 in endl<char, std::__1::char_traits<char> > (__os=...) at /opt/llvm/bin/../include/c++/v1/ostream:1002
#1  operator<< (this=0x5555578bfb28 <std::__1::cout>, __pf=<optimized out>) at /opt/llvm/bin/../include/c++/v1/ostream:196
#2  TCLAP::StdOutput::version (this=<optimized out>, _cmd=...) at external/com_github_mirror_tclap/include/tclap/StdOutput.h:113
#3  0x00005555561e5585 in TCLAP::VersionVisitor::visit (this=<optimized out>) at external/com_github_mirror_tclap/include/tclap/VersionVisitor.h:73
#4  0x00005555561d3e43 in TCLAP::SwitchArg::processArg (this=0x5555580d23f0, i=0x7fffffffcd40, args=...) at external/com_github_mirror_tclap/include/tclap/SwitchArg.h:231
#5  0x00005555561df0d0 in TCLAP::CmdLine::parse (this=0x7fffffffcdc8, args=...) at external/com_github_mirror_tclap/include/tclap/CmdLine.h:460
#6  0x00005555561d8e31 in Envoy::OptionsImpl::OptionsImpl(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > >, std::__1::function<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > (bool)> const&
, spdlog::level::level_enum) (this=0x555558112358, args=..., hot_restart_version_cb=..., default_log_level=spdlog::level::info) at source/server/options_impl.cc:143
#7  0x00005555561d5e10 in Envoy::OptionsImpl::OptionsImpl(int, char const* const*, std::__1::function<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > (bool)> const&, spdlog::level::level_enum) (this=0x555558112358, argc=<optimized out>, argv=<optimized out>, hot_restart_version_cb=..., default_log_level=spdlog::level::info)
    at source/server/options_impl.cc:36
#8  0x00005555561be69a in Envoy::MainCommon::MainCommon (this=0x555558112000, argc=2, argv=0x7fffffffea18) at source/exe/main_common.cc:139
#9  0x00005555561bd143 in make_unique<Envoy::MainCommon, int&, char**&> (__args=<optimized out>, __args=<optimized out>) at /opt/llvm/bin/../include/c++/v1/memory:3132
#10 main (argc=2, argv=0x7fffffffea18) at source/exe/main.cc:26

my system info

$ uname -r
4.14.5

$ cat /etc/os-release 
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

$ cat /etc/redhat-release 
CentOS Linux release 7.1.1503 (Core)
mattklein123 commented 4 years ago

cc @lizan

lizan commented 4 years ago

I don't have any idea why std::endl can cause SIGSEGV, it could be the toolchain issue though. What's the output of readelf -p .comment <path_to_envoy> of your envoy? @keyolk

pablosole commented 4 years ago

@keyolk it has to do with the STDLIB that it is linked against. Set the envvar ENVOY_STDLIB=libstdc++ in the build stage and it will start working again.

keyolk commented 4 years ago

@lizan here the elf section

$ readelf -p .comment ./envoy 

String dump of section '.comment':
  [     0]  GCC: (GNU) 4.8.5 20150623 (Red Hat 4.8.5-36)
  [    2d]  Linker: LLD 8.0.0
  [    3f]  DynASM 1.4.0
  [    4c]  clang version 8.0.0 (tags/RELEASE_800/final)
  [    79]  GCC: (GNU) 7.3.1 20180303 (Red Hat 7.3.1-5)

@pablosole It works : ) Thanks alot! the env should be set during centos build process

Now I built it with below

IMAGE_NAME=envoyproxy/envoy-build-centos IMAGE_ID=latest ENVOY_STDLIB=libsdtdc++ ./ci/run_envoy_docker.sh './ci/do_ci.sh bazel.release.server_only'

How about set default ENVOY_STDLIB from ci/run_envoy_docker.sh ?

lizan commented 4 years ago

@keyolk I think you're on older image with clang-8, the latest one should be with clang-9. Note with libstdc++ on CentOS 7 you're taking the risk described here too.

keyolk commented 4 years ago

@lizan Thats another point : 0

seems latest tagged image in docker hub not updated yet

$ docker run -ti --rm envoyproxy/envoy-build-centos:latest clang --version
clang version 8.0.0 (tags/RELEASE_800/final)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/llvm/bin

$ docker run -ti envoyproxy/envoy-build-ubuntu:latest 'apt list | grep clang'
/naver/docker/bin/docker: Error response from daemon: Container command 'apt list | grep clang' not found or does not exist..
                    css1668:~$ docker run -ti envoyproxy/envoy-build-ubuntu:latest bash -c 'apt list | gr
'
WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

clang-8/now 1:8.0.1~svn363027-1~exp1~20190611212422.76 amd64 [installed,local]
clang-format-8/now 1:8.0.1~svn363027-1~exp1~20190611212422.76 amd64 [installed,local]
clang-tidy-8/now 1:8.0.1~svn363027-1~exp1~20190611212422.76 amd64 [installed,local]
clang-tools-8/now 1:8.0.1~svn363027-1~exp1~20190611212422.76 amd64 [installed,local]
libclang-common-8-dev/now 1:8.0.1~svn363027-1~exp1~20190611212422.76 amd64 [installed,local]
libclang1-8/now 1:8.0.1~svn363027-1~exp1~20190611212422.76 amd64 [installed,local]

with recent updated tag

$ docker run -ti envoyproxy/envoy-build-centos:a0fbb2626d4658d142a197ec7acdc3b5886ebf78-amd64 bash -c 'clang --version'                                                                                    clang version 9.0.0 (tags/RELEASE_900/final)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/llvm/bin
$ docker run -ti envoyproxy/envoy-build-ubuntu:a0fbb2626d4658d142a197ec7acdc3b5886ebf78-amd64  bash -c 'apt list | grep clang'                                                                         

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

libclang1-3.8/now 1:3.8-2ubuntu4 amd64 [installed,local]
lizan commented 4 years ago

@keyolk can you try envoyproxy/envoy-build-centos:b6c2bae5716a289fd68c45a6e5bc16979dbe69c5? that's the build image tag we build 1.13.0.

keyolk commented 4 years ago

@lizan looks good to me : )

$ IMAGE_NAME=envoyproxy/envoy-build-centos IMAGE_ID=b6c2bae5716a289fd68c45a6e5bc16979dbe69c5 ENVOY_STDLIB=libstdc++ ./ci/run_envoy_docker.sh './ci/do_ci.sh bazel.release.server_only'
$ readelf -p .comment  ./envoy

String dump of section '.comment':
  [     0]  GCC: (GNU) 4.8.5 20150623 (Red Hat 4.8.5-39)
  [    2d]  clang version 9.0.0 (tags/RELEASE_900/final)
  [    5a]  DynASM 1.4.0
  [    67]  Linker: LLD 9.0.0
  [    79]  GCC: (GNU) 7.3.1 20180303 (Red Hat 7.3.1-5)

$ ./envoy --version

./envoy  version: c0ac21349bd0ad37383a9124a2e16cf2a790c63b/1.13.0/Clean/RELEASE/BoringSSL
pablosole commented 4 years ago

@lizan Just built envoy 1.13.0 using the image envoyproxy/envoy-build-centos:b6c2bae5716a289fd68c45a6e5bc16979dbe69c5 and default stdlib (ie. without setting ENVOY_STDLIB=libsdtdc++) and it dies the same way.

Tried it on multiple docker centos containers (centos:7 which is a 7.1, centos:centos7.5.1804, centos:centos8) they all crash in the same way.

Update: running the built envoy binary on the envoy-build-centos container that built it also crashes in the same way as the others.

Output of readelf:

String dump of section '.comment':
  [     0]  GCC: (GNU) 4.8.5 20150623 (Red Hat 4.8.5-39)
  [    2d]  clang version 9.0.0 (tags/RELEASE_900/final)
  [    5a]  DynASM 1.4.0
  [    67]  Linker: LLD 9.0.0
  [    79]  GCC: (GNU) 7.3.1 20180303 (Red Hat 7.3.1-5)

I also did an strace -f but didn't seem very helpful. It dies pretty early during the stdlib boostrapping.

# strace -f ./envoy --help
execve("./envoy", ["./envoy", "--help"], [/* 9 vars */]) = 0
brk(NULL)                               = 0x557b647b6000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff14c809000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/opt/llvm/lib/tls/x86_64/libm.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/opt/llvm/lib/tls/x86_64", 0x7ffe08155c60) = -1 ENOENT (No such file or directory)
open("/opt/llvm/lib/tls/libm.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/opt/llvm/lib/tls", 0x7ffe08155c60) = -1 ENOENT (No such file or directory)
open("/opt/llvm/lib/x86_64/libm.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/opt/llvm/lib/x86_64", 0x7ffe08155c60) = -1 ENOENT (No such file or directory)
open("/opt/llvm/lib/libm.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/opt/llvm/lib", 0x7ffe08155c60)   = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=11488, ...}) = 0
mmap(NULL, 11488, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7ff14c806000
close(3)                                = 0
open("/lib64/libm.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0pS\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1139680, ...}) = 0
mmap(NULL, 3150136, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7ff14c2e7000
mprotect(0x7ff14c3e8000, 2093056, PROT_NONE) = 0
mmap(0x7ff14c5e7000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x100000) = 0x7ff14c5e7000
close(3)                                = 0
open("/lib64/librt.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P\"\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=44448, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff14c805000
mmap(NULL, 2128952, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7ff14c0df000
mprotect(0x7ff14c0e6000, 2093056, PROT_NONE) = 0
mmap(0x7ff14c2e5000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x6000) = 0x7ff14c2e5000
close(3)                                = 0
open("/lib64/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0`\16\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=19776, ...}) = 0
mmap(NULL, 2109744, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7ff14bedb000
mprotect(0x7ff14bedd000, 2097152, PROT_NONE) = 0
mmap(0x7ff14c0dd000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7ff14c0dd000
close(3)                                = 0
open("/lib64/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0m\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=144792, ...}) = 0
mmap(NULL, 2208904, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7ff14bcbf000
mprotect(0x7ff14bcd6000, 2093056, PROT_NONE) = 0
mmap(0x7ff14bed5000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x16000) = 0x7ff14bed5000
mmap(0x7ff14bed7000, 13448, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7ff14bed7000
close(3)                                = 0
open("/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P%\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=2173512, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff14c804000
mmap(NULL, 3981792, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7ff14b8f2000
mprotect(0x7ff14bab5000, 2093056, PROT_NONE) = 0
mmap(0x7ff14bcb4000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1c2000) = 0x7ff14bcb4000
mmap(0x7ff14bcba000, 16864, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7ff14bcba000
close(3)                                = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff14c803000
mmap(NULL, 90112, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff14c7ed000
arch_prctl(ARCH_SET_FS, 0x7ff14c801cc0) = 0
mprotect(0x7ff14bcb4000, 16384, PROT_READ) = 0
mprotect(0x7ff14bed5000, 4096, PROT_READ) = 0
mprotect(0x7ff14c0dd000, 4096, PROT_READ) = 0
mprotect(0x7ff14c2e5000, 4096, PROT_READ) = 0
mprotect(0x7ff14c5e7000, 4096, PROT_READ) = 0
stat("/etc/sysconfig/64bit_strstr_via_64bit_strstr_sse2_unaligned", 0x7ffe081561f0) = -1 ENOENT (No such file or directory)
mprotect(0x557b635af000, 1789952, PROT_READ) = 0
mprotect(0x7ff14c80a000, 4096, PROT_READ) = 0
munmap(0x7ff14c806000, 11488)           = 0
set_tid_address(0x7ff14c801f90)         = 62
set_robust_list(0x7ff14c801fa0, 24)     = 0
rt_sigaction(SIGRTMIN, {0x7ff14bcc57e0, [], SA_RESTORER|SA_SIGINFO, 0x7ff14bcce6d0}, NULL, 8) = 0
rt_sigaction(SIGRT_1, {0x7ff14bcc5870, [], SA_RESTORER|SA_RESTART|SA_SIGINFO, 0x7ff14bcce6d0}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0
gettid()                                = 62
brk(NULL)                               = 0x557b647b6000
futex(0x557b637c8e38, FUTEX_WAKE_PRIVATE, 2147483647) = 0
brk(NULL)                               = 0x557b647b6000
brk(0x557b64fb6000)                     = 0x557b64fb6000
brk(NULL)                               = 0x557b64fb6000
brk(NULL)                               = 0x557b64fb6000
brk(0x557b650b6000)                     = 0x557b650b6000
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
open("/sys/devices/system/cpu/online", O_RDONLY|O_CLOEXEC) = 3
read(3, "0-3\n", 8192)                  = 4
close(3)                                = 0
open("/sys/devices/system/cpu/cpu0/tsc_freq_khz", O_RDONLY) = -1 ENOENT (No such file or directory)
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 284302874}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 285719474}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 285782174}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 286987174}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 287259574}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 287372374}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 288474374}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 288644574}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 289622474}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 289718874}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 289819974}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 290627774}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 290790274}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 292389174}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 292447874}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 292479274}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 292588974}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 293781674}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 293999674}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 294354874}) = 0
nanosleep({0, 1000000}, 0x7ffe081572d8) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 297765374}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 297830774}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 299215674}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 300332474}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 300381974}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 301383174}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 301476474}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 304280974}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 305287074}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 306430674}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 307769274}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 308596274}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 308626674}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 309370674}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 309407574}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 309433974}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 310691374}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 310727874}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 310754774}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 313827174}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 313912274}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 313948774}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 313983574}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 314018174}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 314052774}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 314205274}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 314943274}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 315008174}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 316311274}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 316347974}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 316424574}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 316484274}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 316513474}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 316540674}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 316590574}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 316643174}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 316702174}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 317924174}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 317977474}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 330363074}) = 0
nanosleep({0, 2000000}, 0x7ffe081572d8) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 333236374}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 333269074}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 333324074}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 334996274}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 335491774}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 336669774}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 336741974}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 336784374}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 336838474}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 336894574}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 336938174}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 337028374}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 337182874}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 337286374}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 337334974}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 337377574}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 337499474}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 337554874}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 337610974}) = 0
clock_gettime(CLOCK_MONOTONIC_RAW, {353509, 337692974}) = 0
rt_sigaction(SIGPROF, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGPROF, {0x557b63504420, [], SA_RESTORER|SA_RESTART|SA_SIGINFO, 0x7ff14bcce6d0}, NULL, 8) = 0
futex(0x557b637c4720, FUTEX_WAKE_PRIVATE, 2147483647) = 0
rt_sigprocmask(SIG_BLOCK, [PROF], NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [PROF], NULL, 8) = 0
futex(0x7ffe08157464, FUTEX_WAKE, 1)    = 0
futex(0x7ffe08157464, FUTEX_WAKE_PRIVATE, 1) = 0
mmap(NULL, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7ff14c7e7000
mprotect(0x7ff14c7e7000, 4096, PROT_NONE) = 0
mprotect(0x7ff14c7ec000, 4096, PROT_NONE) = 0
sigaltstack({ss_sp=0x7ff14c7e8000, ss_flags=0, ss_size=16384}, {ss_sp=NULL, ss_flags=SS_DISABLE, ss_size=0}) = 0
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
rt_sigaction(SIGABRT, {0x557b627655a0, [], SA_RESTORER|SA_STACK|SA_NODEFER|SA_RESETHAND|SA_SIGINFO, 0x7ff14bcce6d0}, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGBUS, {0x557b627655a0, [], SA_RESTORER|SA_STACK|SA_NODEFER|SA_RESETHAND|SA_SIGINFO, 0x7ff14bcce6d0}, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGFPE, {0x557b627655a0, [], SA_RESTORER|SA_STACK|SA_NODEFER|SA_RESETHAND|SA_SIGINFO, 0x7ff14bcce6d0}, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGILL, {0x557b627655a0, [], SA_RESTORER|SA_STACK|SA_NODEFER|SA_RESETHAND|SA_SIGINFO, 0x7ff14bcce6d0}, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGSEGV, {0x557b627655a0, [], SA_RESTORER|SA_STACK|SA_NODEFER|SA_RESETHAND|SA_SIGINFO, 0x7ff14bcce6d0}, {SIG_DFL, [], 0}, 8) = 0
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0xffffffffffffffe8} ---
futex(0x7ff14bcbac40, FUTEX_WAKE_PRIVATE, 2147483647) = 0
gettid()                                = 62
gettid()                                = 62
gettid()                                = 62
mmap(NULL, 65536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff14c7d7000
rt_sigprocmask(SIG_SETMASK, ~[RTMIN RT_1], [], 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
mmap(NULL, 65536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff14c7c7000
mmap(NULL, 65536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff14c7b7000
clock_gettime(CLOCK_REALTIME, {1580855616, 213477500}) = 0
gettid()                                = 62
open("/etc/localtime", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
fstat(3, {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff14c808000
read(3, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1\0\0\0\1\0\0\0\0"..., 4096) = 118
lseek(3, -62, SEEK_CUR)                 = 56
read(3, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1\0\0\0\1\0\0\0\0"..., 4096) = 62
close(3)                                = 0
munmap(0x7ff14c808000, 4096)            = 0
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0xffffffffffffffe8} ---
+++ killed by SIGSEGV +++
Segmentation fault

Another weirdness I can say is that if you run it without arguments it will just hang without showing anything on screen and stay there forever. From the strace it seems it is able to move on a bit forward in the code (probably because it didn't need to print anything on screen yet), clone two threads and then the main thread dies with the same SIGSEGV but the children stay alive stuck in some futex loop.

pablosole commented 4 years ago

@lizan some more debugging about this. I wrote a hello world and compiled it inside the envoy bazel workspace with -s to get the building commands and during linking it does something like this:

-o
bazel-out/k8-opt/bin/tools/hello/world
-fuse-ld=/opt/rh/devtoolset-7/root/usr/lib/gcc/x86_64-redhat-linux/7/../../../../bin/ld.gold
-Wl,-no-as-needed
-Wl,-z,relro,-z,now
-B/opt/llvm/bin
-lm
-Wl,--gc-sections
bazel-out/k8-opt/bin/tools/hello/_objs/world/world.o
bazel-out/k8-opt/bin/external/envoy/bazel/foreign_cc/gperftools_build/lib/libtcmalloc_and_profiler.a
-pthread
-lrt
-ldl
-Wl,--hash-style=gnu
-pie
-static-libgcc
-lpthread
-L/opt/llvm/lib
-Wl,-rpath,/opt/llvm/lib
-l:libc++.a
-l:libc++abi.a
-lm

There are 2 -fuse-ld (one using gold and one using ldd). If I comment the line on .bazelrc that forces clang to use ldd (build:clang --linkopt=-fuse-ld=lld) the binary works again.

It sounds the ld.gold is hardcoded somewhere else and it's breaking linking.

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or other activity occurs. Thank you for your contributions.

stale[bot] commented 4 years ago

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted". Thank you for your contributions.