Breakpointing/JIT

[Keno]

I've encountered the following issue. Suppose we set a breakpoint in JITted code. We will look at the value that is there, write the breakpoint byte and continue. Later, we come back and restore the byte that we read. However, in the mean time the JIT could have come along and modified the code, causing the written back value to no longer be the value that was actually supposed to be there. Of course this is also an issue for plain old gdb, but it's somewhat more problematic for rr, because in reverse mode, it can easily happen that rr goes back to before the jitted code was there, but still wants to set a breakpoint (vs. regular debugging where such an operation makes little sense since we don't know what code will be there a priori).

Methods that I can think of by which JITs allocate modify executable memory:

1. /proc/self/mem/process_vm_writev.
2. Dual mapping with one RX and one RW page.
3. RWX pages

For 1., I think a simple check in the relevant code path to update any breakpoint would be sufficient. 2&3 are more tricky. We'd probably have to detect this situation in the address space code and do the appropriate dance to disable writing/executing in lock step (i.e. when executing writing is disabled, but we catch that trap, flip the permissions, and do the same thing vice versa once we get back to executing). This could be sped up by using hardware breakpoints, though of course those are a limited resource.

Thoughts?

[rocallahan]

I agree that supporting 1 would be pretty easy so I think it makes sense to do that if we know that some application benefits.

I'm not excited about adding a bunch of complexity to support 2 and 3. I might change my mind if someone submitted a PR, but I think it would be complicated. At the back of mind is the fact that I'm not excited about gdb as a long-term solution and want to minimize the amount of my time I spend dealing with it. (The new stuff I'm working on doesn't have this issue.)

I do think it's probably worth reworking rr's internal breakpoints so they use a hardware breakpoint if a free one is available. Then, if the user is careful to use hardware breakpoints for JITted code, and only uses one or two, everything should work.

[Keno]

I'm fine with only handling 1, since it's easy and sufficient for my use case, though I know that lots of JITs use one of the other two methods.

At the back of mind is the fact that I'm not excited about gdb as a long-term solution and want to minimize the amount of my time I spend dealing with it. (The new stuff I'm working on doesn't have this issue.)

No argument there. I have mentioned before that I don't actually use GDB as a frontend (unless reporting rr issues). Nevertheless, the number of hardware breakpoints is quite limited, and I don't think I would be comfortable telling people that if they use JIT they're limited to one or two breakpoints.

[Keno]

Also cc @khuey, since I know at least some parts of Chromium use approach 2

[rocallahan]

Nevertheless, the number of hardware breakpoints is quite limited, and I don't think I would be comfortable telling people that if they use JIT they're limited to one or two breakpoints.

Well, that only applies to breakpoints in JITted code. And I feel like people debugging JITted code in rr with reverse execution are already getting a pretty good deal :-).

I guess I'm open to someone trying to implement the W/X flipping approach to see how hard it is. BTW I hope JITs don't ever try to read the code they've written, since there is no such thing as R/X flipping on Intel (at least until pkeys ships).