TestIssue871 failure with undo backend

$ go run scripts/make.go test -v -s proc -r TestIssue871 -b undo
=== RUN   TestIssue871
--- FAIL: TestIssue871 (1.85s)
    support.go:246: enabling recording for github.com/undoio/delve/pkg/proc_test.TestIssue871
    proc_test.go:81: recording
    proc_test.go:83: replaying
    proc_test.go:3003: variable a not found
    proc_test.go:3007: variable b not found

See https://github.com/go-delve/delve/issues/871 for the original issue.

Analysis

The test program looks like this:

func main() {
    a := [3]int{1, 2, 3}
    b := &a
    runtime.Breakpoint()
    fmt.Println(b, *b) // set breakpoint here
}

When run with the default backend, the program breaks on the call to fmt.Println in main:

(dlv) continue
> main.main() ./issue871.go:12 (PC: 0x4aa44d)
     7: 
     8: func main() {
     9:     a := [3]int{1, 2, 3}
    10:     b := &a
    11:     runtime.Breakpoint()
=>  12:     fmt.Println(b, *b) // set breakpoint here
    13: }

and so the locals are a and b as expected by the test:

(dlv) locals
a = [3]int [...]
b = (*[3]int)(0xc0000be000)

but when run with the undo backend, the program breaks inside the runtime.Breakpoint function:

(dlv) continue
> runtime.breakpoint() /home/grees/goroot/src/runtime/asm_amd64.s:240 (PC: 0x458bd1)
Warning: debugging optimized function
0000000000023066
   235: DATA    runtime·mainPC+0(SB)/8,$runtime·main(SB)
   236: GLOBL   runtime·mainPC(SB),RODATA,$8
   237: 
   238: TEXT runtime·breakpoint(SB),NOSPLIT,$0-0
   239:     BYTE    $0xcc
=> 240:     RET
   241: 
   242: TEXT runtime·asminit(SB),NOSPLIT,$0-0
   243:     // No per-thread init.
   244:     RET
   245:

and so the locals are not available.

Note that the behaviour when you try to "stepout" from the breakpoint is not very satisfactory:

(dlv) stepout
> runtime.breakpoint() /home/grees/goroot/src/runtime/asm_amd64.s:240 (PC: 0x458bd1)
Warning: debugging optimized function
0000000000023066
    breakpoint hit during stepout, continuing...
> runtime.breakpoint() /home/grees/goroot/src/runtime/asm_amd64.s:240 (PC: 0x458bd1)
Warning: debugging optimized function
0000000000023066
    breakpoint hit during stepout, continuing...
> runtime.breakpoint() /home/grees/goroot/src/runtime/asm_amd64.s:240 (PC: 0x458bd1)
Warning: debugging optimized function
0000000000023066
    breakpoint hit during stepout, continuing...

and so on.

Delve has logic for getting out of runtime.breakpoint and back into code. See pkg/proc/proc.go:

case loc.Fn.Name == "runtime.breakpoint":
        // Single-step current thread until we exit runtime.breakpoint and
        // runtime.Breakpoint.
        // On go < 1.8 it was sufficient to single-step twice on go1.8 a change
        // to the compiler requires 4 steps.
        if err := stepInstructionOut(dbp, curthread, "runtime.breakpoint", "runtime.Breakpoint"); err != nil {
                return err
        }
        return conditionErrors(threads)

However, with the Undo backend we never reach this point because it is guarded by:

// runtime.Breakpoint, manual stop or debugCallV1-related stop
recorded, _ := dbp.Recorded()
if recorded {
        return conditionErrors(threads)
}

and in the Undo backend, Recorded is implemented like this in pkg/proc/gdbserial/gdbserver.go:

// Recorded returns whether or not we are debugging
// a recorded "traced" program.
func (p *Process) Recorded() (bool, string) {
    return p.tracedir != "", p.tracedir
}

and Undo always has a trace directory configured:

(udb) p p.tracedir
$2 = 0xc0000bc800 "/tmp/undo225061744"

Unfortunately, we can't just revised Recorded to return false, because if we do then Continue gets stuck inside stepInstructionOut. The problem is that the Undo backend resumes on the breakpoint instruction and so when it steps again, it hits the breakpoint again and doesn't make progress.

Possibly this is a bug in Delve? I don't see that the other gdbserial backends will do any better.

undoio / delve

TestIssue871 failure with undo backend #8

Analysis