odigos-io / opentelemetry-go-instrumentation

OpenTelemetry auto-instrumentation for Go applications
Apache License 2.0
292 stars 44 forks source link

Code segment in process memory mapping not always writable #44

Closed ocampeau closed 1 year ago

ocampeau commented 1 year ago

Expected Behavior

I have a Go process running on my machine, which is a web server using net/http. The executable path is /home/ocampeau/hello

When I run the following command, I expect the tool to emit traces when an HTTP request is received by my process

sudo OTEL_TARGET_EXE=/home/olivier/hello ./kv-go-instrumentation

Actual Behavior

Instead, I get a panic

panic: cant find keyval map

Steps to Reproduce the Problem

  1. Build an http server in Go
  2. Compile the current version of opentelemetry-go-instrumentation
  3. Run sudo OTEL_TARGET_EXE=PATH_TO_EXECUTABLE ./kv-go-instrumentation

Additional Info

My guess is that there is a bug with the permission check of the memory mapping of the process.

When I inspect /proc/PID/maps, I see this:

00400000-00612000 r-xp 00000000 103:02 17174730                          /home/olivier/hello
00612000-0080d000 r--p 00212000 103:02 17174730                            /home/olivier/hello
0080d000-0084a000 rw-p 0040d000 103:02 17174730                          /home/olivier/hello
0084a000-00883000 rw-p 00000000 00:00 0 
0248f000-024b0000 rw-p 00000000 00:00 0                                           [heap]
c000000000-c000400000 rw-p 00000000 00:00 0 
c000400000-c004000000 ---p 00000000 00:00 0 
7f45f4000000-7f45f4021000 rw-p 00000000 00:00 0 
7f45f4021000-7f45f8000000 ---p 00000000 00:00 0 
7f45f8000000-7f45f8021000 rw-p 00000000 00:00 0 
7f45f8021000-7f45fc000000 ---p 00000000 00:00 0 
7f45fc000000-7f45fc021000 rw-p 00000000 00:00 0 
7f45fc021000-7f4600000000 ---p 00000000 00:00 0 
7f4600000000-7f4600021000 rw-p 00000000 00:00 0 
...

It seems that the code segment referenced in the memory mapping of the process has READ and EXECUTABLE permissions, but not WRITE. In this case, the start address of the code segment would be 00400000 and the end address would be 00612000.

When I update the following code to remove the check on the write permissions, the tool is indeed working as expected:

func (a *processAnalyzer) findKeyvalMmap(pid int) (uintptr, uintptr) {
    fs, err := procfs.NewProc(pid)
    if err != nil {
        panic(err)
    }

    maps, err := fs.ProcMaps()
    if err != nil {
        panic(err)
    }

    for _, m := range maps {
        if m.Perms != nil && m.Perms.Read && m.Perms.Execute { // I removed the check on write permission here
            log.Logger.Info("found addr of keyval map", "addr", m.StartAddr)
            return m.StartAddr, m.EndAddr
        }
    }
    panic(errors.New("cant find keyval map"))
yfractal commented 1 year ago

will this issue https://github.com/keyval-dev/opentelemetry-go-instrumentation/issues/36 could help you?

it needs /odigos-launcher/launch to create some memory area for writing.

ocampeau commented 1 year ago

Oh my yes thank you! Sorry for not looking that up before. I will close this issue.

yfractal commented 1 year ago

no worry 😄