pixie-io / pixie

Instant Kubernetes-Native Application Observability
https://px.dev
Apache License 2.0
5.54k stars 426 forks source link

Go binary version detection results in `std::bad_alloc` exception #1646

Open ddelnano opened 1 year ago

ddelnano commented 1 year ago

We recently revamped how Pixie detects a binary's Go version (#1605). This aligned it with how the go version command implements its functionality as well as addressed other bugs in our previous implementation (as described in the linked PR) .

We received a report of a std::bad_alloc exception that occurs within this new logic, which causes the PEM to crash.

libc++abi: terminating due to uncaught exception of type std::bad_alloc: std::bad_alloc
E20230730 23:53:00.815058 10815 signal_action.cc:63] Caught Aborted, suspect faulting address 0x262c. Trace:
**************************
PC: @     0x7f019b04c472  (unknown)  abort
    @     0x562574f7f976  (unknown)  abort_message
    @     0x562574f6981b  (unknown)  demangling_terminate_handler()
    @     0x562574f7f5f3  (unknown)  std::__terminate()
    @     0x562574f81316  (unknown)  __cxxabiv1::failed_throw()
    @     0x562574f812af  (unknown)  __cxa_throw
    @     0x562574f16384  (unknown)  (anonymous namespace)::handle_oom()
    @     0x562574f868dd  (unknown)  tcmalloc::allocate_full_cpp_throw_oom()
    @     0x56257081c2ca  (unknown)  px::stirling::obj_tools::ElfReader::BinaryByteCode<>()
    @     0x56257081badb  (unknown)  px::stirling::obj_tools::ReadGoString()
    @     0x56257081cdde  (unknown)  px::stirling::obj_tools::ReadGoBuildVersion()
    @     0x562570749126  (unknown)  px::stirling::GoTLSSymAddrs()
    @     0x56257073421f  (unknown)  px::stirling::UProbeManager::UpdateGoTLSSymAddrs()
    @     0x562570738cbf  (unknown)  px::stirling::UProbeManager::AttachGoTLSUProbes()
    @     0x56257073d5aa  (unknown)  px::stirling::UProbeManager::DeployGoUProbes()
    @     0x56257073ecec  (unknown)  px::stirling::UProbeManager::DeployUProbes()
    @     0x562570740d6b  (unknown)  _ZNSt3__114__thread_proxyB6v15006INS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEZN2px8stirling13UProbeManager22RunDeployUProbesThreadERKN4absl12lts_2022062313flat_hash_setINS7_2md4UPIDENSB_    @     0x7f019b0aefd4  (unknown)  (unknown)
    @     0x7f019b12f5bc  (unknown)  (unknown)
**************************

Until this bug is addressed, there are two (soon to be three) ways to remediate this issue:

ddelnano commented 2 months ago

There was a recent report of this in the Pixie slack. For this problematic case, the Go application in question was built with the following configuration:

ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, Go BuildID=zabiqC8wfzfmrjRJ0FDq/NV0zLDU8qJwxHA8tkC-k/2e_IZ0k1YT-EJ5tgrhl9/J1BG3wmTbdYe1tMJkFv3, BuildID[sha1]=f0ea16a8de2ad5701cdea163166fc0ff33014c5e, not stripped

When I run go version -m <binary>, I get this info:
go1.20.3
    path    command-line-arguments
    dep github.com/golang/protobuf  v1.5.3  h1:KhyjKVUg7Usr/dYsdSqoFveMYd5ko72D+zANwlG1mmg=
    dep golang.org/x/net    v0.11.0 h1:Gi2tvZIJyBtO9SDr1q9h5hEQCp/4L2RQ+ar0qjx2oNU=
    dep golang.org/x/sys    v0.9.0  h1:KS/R3tvhPqvJvwcKfnBHJwwthS11LRhmM5D59eEXa0s=
    dep golang.org/x/text   v0.10.0 h1:UpjohKhiEgNc0CSauXmwYftY1+LlaC75SJwh0SgCX58=
    dep google.golang.org/genproto  v0.0.0-20230410155749-daa745c078e1  h1:KpwkzHKEF7B9Zxg18WzOa7djJ+Ha5DzthMyZYQfEn2A=
    dep google.golang.org/grpc  v1.56.1 h1:z0dNfjIl0VpaZ9iSVjA6daGatAYwPGstTjt5vkRMFkQ=
    dep google.golang.org/protobuf  v1.30.0 h1:kPPoIgf3TsEvrm0PFe15JQ+570QVxYzEvvHqChK+cng=
    dep producers   (devel) 
    dep rtp v0.0.0-00010101000000-000000000000
    =>  ../rtp  (devel) 

    dep ztc_interface   v0.0.0-00010101000000-000000000000
    =>  ./ztc_interface (devel) 

    build   -buildmode=exe
    build   -compiler=gc
    build   CGO_ENABLED=1
    build   CGO_CFLAGS=
    build   CGO_CPPFLAGS=
    build   CGO_CXXFLAGS=
    build   CGO_LDFLAGS=
    build   GOARCH=amd64
    build   GOOS=linux
    build   GOAMD64=v1