golang / go

The Go programming language
https://go.dev
BSD 3-Clause "New" or "Revised" License
124.07k stars 17.68k forks source link

x/debug/cmd/viewcore: does not search paths which are marked as deleted #44636

Open ajwerner opened 3 years ago

ajwerner commented 3 years ago

What version of Go are you using (go version)?

$ go version
go version go1.15.5 linux/amd64

Does this issue reproduce with the latest release?

Yes.

When linux dynamically links shared libraries, it seems that sometimes something involving hugetlb pages I don't understand leads to the shared library being mapped in the process to a file marked as deleted. This leads to entries in /proc/$pid/maps tht looks like this:

7fad20209000-7fad2020c000 r--p 00000000 08:01 3478                       /usr/lib/x86_64-linux-gnu/libnss_files-2.31.so (deleted)
7fad2020c000-7fad20213000 r-xp 00003000 08:01 3478                       /usr/lib/x86_64-linux-gnu/libnss_files-2.31.so (deleted)
7fad20213000-7fad20215000 r--p 0000a000 08:01 3478                       /usr/lib/x86_64-linux-gnu/libnss_files-2.31.so (deleted)
7fad20215000-7fad20216000 r--p 0000b000 08:01 3478                       /usr/lib/x86_64-linux-gnu/libnss_files-2.31.so (deleted)
7fad20216000-7fad20217000 rw-p 0000c000 08:01 3478                       /usr/lib/x86_64-linux-gnu/libnss_files-2.31.so (deleted)
7fad20217000-7fad20322000 rw-p 00000000 00:00 0 
7fad20322000-7fad20347000 r--p 00000000 08:01 3470                       /usr/lib/x86_64-linux-gnu/libc-2.31.so (deleted)
7fad20347000-7fad204bf000 r-xp 00025000 08:01 3470                       /usr/lib/x86_64-linux-gnu/libc-2.31.so (deleted)
7fad204bf000-7fad20509000 r--p 0019d000 08:01 3470                       /usr/lib/x86_64-linux-gnu/libc-2.31.so (deleted)
7fad20509000-7fad2050a000 ---p 001e7000 08:01 3470                       /usr/lib/x86_64-linux-gnu/libc-2.31.so (deleted)
7fad2050a000-7fad2050d000 r--p 001e7000 08:01 3470                       /usr/lib/x86_64-linux-gnu/libc-2.31.so (deleted)
7fad2050d000-7fad20510000 rw-p 001ea000 08:01 3470                       /usr/lib/x86_64-linux-gnu/libc-2.31.so (deleted)
7fad20510000-7fad20514000 rw-p 00000000 00:00 0 
7fad20514000-7fad20523000 r--p 00000000 08:01 3472                       /usr/lib/x86_64-linux-gnu/libm-2.31.so (deleted)
7fad20523000-7fad205ca000 r-xp 0000f000 08:01 3472                       /usr/lib/x86_64-linux-gnu/libm-2.31.so (deleted)
7fad205ca000-7fad20661000 r--p 000b6000 08:01 3472                       /usr/lib/x86_64-linux-gnu/libm-2.31.so (deleted)
7fad20661000-7fad20662000 r--p 0014c000 08:01 3472                       /usr/lib/x86_64-linux-gnu/libm-2.31.so (deleted)
7fad20662000-7fad20663000 rw-p 0014d000 08:01 3472                       /usr/lib/x86_64-linux-gnu/libm-2.31.so (deleted)
7fad20663000-7fad20667000 r--p 00000000 08:01 3484                       /usr/lib/x86_64-linux-gnu/libresolv-2.31.so (deleted)
7fad20667000-7fad20677000 r-xp 00004000 08:01 3484                       /usr/lib/x86_64-linux-gnu/libresolv-2.31.so (deleted)
7fad20677000-7fad2067a000 r--p 00014000 08:01 3484                       /usr/lib/x86_64-linux-gnu/libresolv-2.31.so (deleted)
7fad2067a000-7fad2067b000 ---p 00017000 08:01 3484                       /usr/lib/x86_64-linux-gnu/libresolv-2.31.so (deleted)
7fad2067b000-7fad2067c000 r--p 00017000 08:01 3484                       /usr/lib/x86_64-linux-gnu/libresolv-2.31.so (deleted)
7fad2067c000-7fad2067d000 rw-p 00018000 08:01 3484                       /usr/lib/x86_64-linux-gnu/libresolv-2.31.so (deleted)
7fad2067d000-7fad2067f000 rw-p 00000000 00:00 0 
7fad2067f000-7fad20680000 r--p 00000000 08:01 3471                       /usr/lib/x86_64-linux-gnu/libdl-2.31.so (deleted)
7fad20680000-7fad20682000 r-xp 00001000 08:01 3471                       /usr/lib/x86_64-linux-gnu/libdl-2.31.so (deleted)
7fad20682000-7fad20683000 r--p 00003000 08:01 3471                       /usr/lib/x86_64-linux-gnu/libdl-2.31.so (deleted)
7fad20683000-7fad20684000 r--p 00003000 08:01 3471                       /usr/lib/x86_64-linux-gnu/libdl-2.31.so (deleted)
7fad20684000-7fad20685000 rw-p 00004000 08:01 3471                       /usr/lib/x86_64-linux-gnu/libdl-2.31.so (deleted)
7fad20685000-7fad2068c000 r--p 00000000 08:01 3483                       /usr/lib/x86_64-linux-gnu/libpthread-2.31.so (deleted)
7fad2068c000-7fad2069d000 r-xp 00007000 08:01 3483                       /usr/lib/x86_64-linux-gnu/libpthread-2.31.so (deleted)
7fad2069d000-7fad206a2000 r--p 00018000 08:01 3483                       /usr/lib/x86_64-linux-gnu/libpthread-2.31.so (deleted)
7fad206a2000-7fad206a3000 r--p 0001c000 08:01 3483                       /usr/lib/x86_64-linux-gnu/libpthread-2.31.so (deleted)
7fad206a3000-7fad206a4000 rw-p 0001d000 08:01 3483                       /usr/lib/x86_64-linux-gnu/libpthread-2.31.so (deleted)
7fad206a4000-7fad206a8000 rw-p 00000000 00:00 0 
7fad206a8000-7fad206ab000 r--p 00000000 08:01 3485                       /usr/lib/x86_64-linux-gnu/librt-2.31.so (deleted)
7fad206ab000-7fad206af000 r-xp 00003000 08:01 3485                       /usr/lib/x86_64-linux-gnu/librt-2.31.so (deleted)
7fad206af000-7fad206b0000 r--p 00007000 08:01 3485                       /usr/lib/x86_64-linux-gnu/librt-2.31.so (deleted)
7fad206b0000-7fad206b1000 ---p 00008000 08:01 3485                       /usr/lib/x86_64-linux-gnu/librt-2.31.so (deleted)
7fad206b1000-7fad206b2000 r--p 00008000 08:01 3485                       /usr/lib/x86_64-linux-gnu/librt-2.31.so (deleted)
7fad206b2000-7fad206b3000 rw-p 00009000 08:01 3485                       /usr/lib/x86_64-linux-gnu/librt-2.31.so (deleted)
7fad206b3000-7fad206b5000 rw-p 00000000 00:00 0 
7fad206bb000-7fad206bc000 r--p 00000000 08:01 3466                       /usr/lib/x86_64-linux-gnu/ld-2.31.so (deleted)
7fad206bc000-7fad206df000 r-xp 00001000 08:01 3466                       /usr/lib/x86_64-linux-gnu/ld-2.31.so (deleted)
7fad206df000-7fad206e7000 r--p 00024000 08:01 3466                       /usr/lib/x86_64-linux-gnu/ld-2.31.so (deleted)
7fad206e7000-7fad206e8000 rw-p 00000000 00:00 0 
7fad206e8000-7fad206e9000 r--p 0002c000 08:01 3466                       /usr/lib/x86_64-linux-gnu/ld-2.31.so (deleted)
7fad206e9000-7fad206ea000 rw-p 0002d000 08:01 3466                       /usr/lib/x86_64-linux-gnu/ld-2.31.so (deleted)

I think this has something to do with the link process copying the so to the hugetlb fs and then deleting it. In fact, using older versions of gdb and linux, the path with the (deleted) annotation doesn't seem to correspond to the path to the actual lib. I can live with using more modern tools than 16.04 so I'm not going to make noise about that.

What operating system and processor architecture are you using (go env)?

go env Output
$ go env

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/ajwerner/.cache/go-build"
GOENV="/home/ajwerner/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/ajwerner/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/ajwerner"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/ajwerner/src/golang.org/x/debug/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build465746507=/tmp/go-build -gno-record-gcc-switches"

What did you do?

ubuntu@ajwerner-test-0001:~$ ./viewcore-old /mnt/data1/cores/.4556 --exe ./cockroach overview

What did you expect to see?

No errors.

What did you see instead?

WARNING: Missing data for addresses [7fad20209000 7fad2020c000] because of failure to open /usr/lib/x86_64-linux-gnu/libnss_files-2.31.so (deleted): no such file or directory. Assuming all zero.
WARNING: Missing data for addresses [7fad2020c000 7fad20213000] because of failure to open /usr/lib/x86_64-linux-gnu/libnss_files-2.31.so (deleted): no such file or directory. Assuming all zero.
WARNING: Missing data for addresses [7fad20213000 7fad20215000] because of failure to open /usr/lib/x86_64-linux-gnu/libnss_files-2.31.so (deleted): no such file or directory. Assuming all zero.
WARNING: Missing data for addresses [7fad20215000 7fad20216000] because of failure to open /usr/lib/x86_64-linux-gnu/libnss_files-2.31.so (deleted): no such file or directory. Assuming all zero.
WARNING: Missing data for addresses [7fad20216000 7fad20217000] because of failure to open /usr/lib/x86_64-linux-gnu/libnss_files-2.31.so (deleted): no such file or directory. Assuming all zero.
WARNING: Missing data for addresses [7fad20322000 7fad20347000] because of failure to open /usr/lib/x86_64-linux-gnu/libc-2.31.so (deleted): no such file or directory. Assuming all zero.
WARNING: Missing data for addresses [7fad20347000 7fad204bf000] because of failure to open /usr/lib/x86_64-linux-gnu/libc-2.31.so (deleted): no such file or directory. Assuming all zero.
WARNING: Missing data for addresses [7fad204bf000 7fad20509000] because of failure to open /usr/lib/x86_64-linux-gnu/libc-2.31.so (deleted): no such file or directory. Assuming all zero.
WARNING: Missing data for addresses [7fad20509000 7fad2050a000] because of failure to open /usr/lib/x86_64-linux-gnu/libc-2.31.so (deleted): no such file or directory. Assuming all zero.
WARNING: Missing data for addresses [7fad2050a000 7fad2050d000] because of failure to open /usr/lib/x86_64-linux-gnu/libc-2.31.so (deleted): no such file or directory. Assuming all zero.
WARNING: Missing data for addresses [7fad2050d000 7fad20510000] because of failure to open /usr/lib/x86_64-linux-gnu/libc-2.31.so (deleted): no such file or directory. Assuming all zero.
WARNING: Missing data for addresses [7fad20514000 7fad20523000] because of failure to open /usr/lib/x86_64-linux-gnu/libm-2.31.so (deleted): no such file or directory. Assuming all zero.
WARNING: Missing data for addresses [7fad20523000 7fad205ca000] because of failure to open /usr/lib/x86_64-linux-gnu/libm-2.31.so (deleted): no such file or directory. Assuming all zero.
WARNING: Missing data for addresses [7fad205ca000 7fad20661000] because of failure to open /usr/lib/x86_64-linux-gnu/libm-2.31.so (deleted): no such file or directory. Assuming all zero.
WARNING: Missing data for addresses [7fad20661000 7fad20662000] because of failure to open /usr/lib/x86_64-linux-gnu/libm-2.31.so (deleted): no such file or directory. Assuming all zero.
WARNING: Missing data for addresses [7fad20662000 7fad20663000] because of failure to open /usr/lib/x86_64-linux-gnu/libm-2.31.so (deleted): no such file or directory. Assuming all zero.
WARNING: Missing data for addresses [7fad20663000 7fad20667000] because of failure to open /usr/lib/x86_64-linux-gnu/libresolv-2.31.so (deleted): no such file or directory. Assuming all zero.
WARNING: Missing data for addresses [7fad20667000 7fad20677000] because of failure to open /usr/lib/x86_64-linux-gnu/libresolv-2.31.so (deleted): no such file or directory. Assuming all zero.
WARNING: Missing data for addresses [7fad20677000 7fad2067a000] because of failure to open /usr/lib/x86_64-linux-gnu/libresolv-2.31.so (deleted): no such file or directory. Assuming all zero.
WARNING: Missing data for addresses [7fad2067a000 7fad2067b000] because of failure to open /usr/lib/x86_64-linux-gnu/libresolv-2.31.so (deleted): no such file or directory. Assuming all zero.
WARNING: Missing data for addresses [7fad2067b000 7fad2067c000] because of failure to open /usr/lib/x86_64-linux-gnu/libresolv-2.31.so (deleted): no such file or directory. Assuming all zero.
WARNING: Missing data for addresses [7fad2067c000 7fad2067d000] because of failure to open /usr/lib/x86_64-linux-gnu/libresolv-2.31.so (deleted): no such file or directory. Assuming all zero.
WARNING: Missing data for addresses [7fad2067f000 7fad20680000] because of failure to open /usr/lib/x86_64-linux-gnu/libdl-2.31.so (deleted): no such file or directory. Assuming all zero.
WARNING: Missing data for addresses [7fad20680000 7fad20682000] because of failure to open /usr/lib/x86_64-linux-gnu/libdl-2.31.so (deleted): no such file or directory. Assuming all zero.
WARNING: Missing data for addresses [7fad20682000 7fad20683000] because of failure to open /usr/lib/x86_64-linux-gnu/libdl-2.31.so (deleted): no such file or directory. Assuming all zero.
WARNING: Missing data for addresses [7fad20683000 7fad20684000] because of failure to open /usr/lib/x86_64-linux-gnu/libdl-2.31.so (deleted): no such file or directory. Assuming all zero.
WARNING: Missing data for addresses [7fad20684000 7fad20685000] because of failure to open /usr/lib/x86_64-linux-gnu/libdl-2.31.so (deleted): no such file or directory. Assuming all zero.
WARNING: Missing data for addresses [7fad20685000 7fad2068c000] because of failure to open /usr/lib/x86_64-linux-gnu/libpthread-2.31.so (deleted): no such file or directory. Assuming all zero.
WARNING: Missing data for addresses [7fad2068c000 7fad2069d000] because of failure to open /usr/lib/x86_64-linux-gnu/libpthread-2.31.so (deleted): no such file or directory. Assuming all zero.
WARNING: Missing data for addresses [7fad2069d000 7fad206a2000] because of failure to open /usr/lib/x86_64-linux-gnu/libpthread-2.31.so (deleted): no such file or directory. Assuming all zero.
WARNING: Missing data for addresses [7fad206a2000 7fad206a3000] because of failure to open /usr/lib/x86_64-linux-gnu/libpthread-2.31.so (deleted): no such file or directory. Assuming all zero.
WARNING: Missing data for addresses [7fad206a3000 7fad206a4000] because of failure to open /usr/lib/x86_64-linux-gnu/libpthread-2.31.so (deleted): no such file or directory. Assuming all zero.
WARNING: Missing data for addresses [7fad206a8000 7fad206ab000] because of failure to open /usr/lib/x86_64-linux-gnu/librt-2.31.so (deleted): no such file or directory. Assuming all zero.
WARNING: Missing data for addresses [7fad206ab000 7fad206af000] because of failure to open /usr/lib/x86_64-linux-gnu/librt-2.31.so (deleted): no such file or directory. Assuming all zero.
WARNING: Missing data for addresses [7fad206af000 7fad206b0000] because of failure to open /usr/lib/x86_64-linux-gnu/librt-2.31.so (deleted): no such file or directory. Assuming all zero.
WARNING: Missing data for addresses [7fad206b0000 7fad206b1000] because of failure to open /usr/lib/x86_64-linux-gnu/librt-2.31.so (deleted): no such file or directory. Assuming all zero.
WARNING: Missing data for addresses [7fad206b1000 7fad206b2000] because of failure to open /usr/lib/x86_64-linux-gnu/librt-2.31.so (deleted): no such file or directory. Assuming all zero.
WARNING: Missing data for addresses [7fad206b2000 7fad206b3000] because of failure to open /usr/lib/x86_64-linux-gnu/librt-2.31.so (deleted): no such file or directory. Assuming all zero.
WARNING: Missing data for addresses [7fad206bb000 7fad206bc000] because of failure to open /usr/lib/x86_64-linux-gnu/ld-2.31.so (deleted): no such file or directory. Assuming all zero.
WARNING: Missing data for addresses [7fad206bc000 7fad206df000] because of failure to open /usr/lib/x86_64-linux-gnu/ld-2.31.so (deleted): no such file or directory. Assuming all zero.
WARNING: Missing data for addresses [7fad206df000 7fad206e7000] because of failure to open /usr/lib/x86_64-linux-gnu/ld-2.31.so (deleted): no such file or directory. Assuming all zero.
WARNING: Missing data for addresses [7fad206e8000 7fad206e9000] because of failure to open /usr/lib/x86_64-linux-gnu/ld-2.31.so (deleted): no such file or directory. Assuming all zero.
WARNING: Missing data for addresses [7fad206e9000 7fad206ea000] because of failure to open /usr/lib/x86_64-linux-gnu/ld-2.31.so (deleted): no such file or directory. Assuming all zero.
arch    amd64
runtime go1.15.6
memory  928.9 MB
ajwerner commented 3 years ago

This patch seems to fix it but I wouldn't really know how we'd want to test it:

diff --git a/internal/core/process.go b/internal/core/process.go
index d4da13e..944380e 100644
--- a/internal/core/process.go
+++ b/internal/core/process.go
@@ -512,14 +512,23 @@ func (p *Process) openMappedFile(fname string, m *Mapping) (*os.File, error) {
                isMainExe = true
        }

-       if !isMainExe {
-               backing.f, backing.err = os.Open(filepath.Join(p.base, fname))
-       } else { // keep main executable in p.mainExecName
-               p.mainExecName = fname
-               if p.exe != nil {
-                       backing.f, backing.err = p.exe, nil
-               } else {
-                       backing.f, backing.err = os.Open(filepath.Join(p.base, fname))
+       candidates := []string{fname}
+       if strings.HasSuffix(fname, "(deleted)") {
+               candidates = append(candidates, strings.TrimSuffix(fname, " (deleted)"))
+       }
+       for _, n := range candidates {
+               if !isMainExe {
+                       backing.f, backing.err = os.Open(filepath.Join(p.base, n))
+               } else { // keep main executable in p.mainExecName
+                       p.mainExecName = fname
+                       if p.exe != nil {
+                               backing.f, backing.err = p.exe, nil
+                       } else {
+                               backing.f, backing.err = os.Open(filepath.Join(p.base, fname))
+                       }
+               }
+               if backing.err == nil {
+                       break
                }
        }