Integrating a gdbserver and record/replay functionality into wasmtime

[DrRuhe]

Feature

• let wasmtime expose a gdbserver
• Also allow record/replay out-of-the-box to support reversible debugging.

Implementation

I have some ideas on how this might be doable, but I am not familiar enough with wasmtime to make an accurate assessment.

• adding a gdbserver could be done via gdbstub
• recording would require storing the results from wasi calls and reinstating the runtime state on replay.

I would be interested in implementing this, however, due to the complexity of such a feature paired with the deep understanding required for successful implementation, I would need guidance/mentoring.

Benefit

• the current debugging has the wasmtime runtime "in the way" of the debugger.
• recording in the wasm-runtime level allows for optimizations not possible when using rr record wasmtime run ...
• adding reversible debugging capability to wasmtime would also indirectly add the capability to every language that can compile to wasm

Alternatives

Improve the rr-records-wasmtime-running-wasm story directly.

[cfallin]

@DrRuhe thanks for writing this up and registering interest -- we've definitely talked about this idea internally before, though I can't find any related issues on file at the moment; now we do!

Personally I think this would be a hugely compelling use-case for Wasm's determinism, and very useful. The main trick is in what to record. In particular, one needs to understand the semantics of hostcalls and their memory side-effects. Consider the WASI fd_read call: it provides pointers in which the runtime is to place the data it reads; a recording for later replay would need to record that memory update as well, and replay it at the right time.

(A lot of this thinking was done in the context of a "multiprocess wasmtime", which is something else a few of us have discussed: if the hostcall implementation lives in a separate process than the guest code, one has all of the same problems in capturing the precise memory reads/writes, in that case to marshal across the boundary.)

There are at least three ways of doing this that I can think of:

• Require annotations on imports, and disallow direct access to an exported memory otherwise. The annotations would need to be in some sort of DSL: "writes memory at ptr arg0 up to length arg1", with more complex variants for e.g. iovecs.
• Use some sort of virtual-memory tricks to capture a hostcall's updates to guest memory. (mprotect the whole memory read-only on a call out to another module or the host; any pagefault from a write allows the page to be written but then copies the whole page into the recording trace before return to guest.) Would work, but very slow.
• Use another mechanism for calls across modules / into the host that doesn't require raw access to shared memory from handwritten code.

The last option is the most compelling given the future use of the component model: the IDL (WIT) describes calls with pure value semantics (pass this buffer in, return that buffer) and this would allow us to accurately capture everything in the recording without any additional code in the hostcall implementation. For that reason, it probably makes sense to wait until e.g. WASI is imported via component-level APIs, and then build this recording feature for modules that don't export their memory otherwise.

(Completely separate discussion: true parallelism and the resulting nondeterminism. Probably we'd have to disallow wasi-threads in modules being recorded, or else use something like token-passing on synchronization edges (like this work) for full determinism.)

[cfallin]

Ah, and actually there's a fourth way that @tekknolagi came up with, and told me about: one could do a Wasm-to-Wasm transform that rewrites a module to have two memories for every original memory (or at least the exported ones); every store writes to both the original and "shadow", while every load loads from both and compares. If they differ, emit a recording-trace record and then update the shadow. (Basically we're tracking what state the replay would have, and emitting diffs from it.) This allows for arbitrary external updates to happen to the memory, at the cost of 2x memory overhead and slowdown on every load/store. I do very much like its generality and the fact it would work without runtime involvement, though. @tekknolagi may or may not be building this at some point, I'm not sure; Max, please do let us know if you do!

[tekknolagi]

This is not currently under development---there are some other projects I am working on right now---but may happen later! I'll add you to the list of interested parties :P

[tekknolagi]

Also sorry to be nitpicky, but I did not come up with it. I don't remember whose idea it was in the email thread, but not mine

[fitzgen]

FYI, there is a forthcoming RFC for Wasmtime's debugging story and how incremental milestones along the way from where we are now to where we want to end up eventually. It will touch on core dumps, live debugging, record and replay, debugging protocols and servers, and all that stuff. Will try and remember to cross post here when it is ready, but it probably doesn't hurt to watch the RFCs repo if you're particularly interested.

[fitzgen]

There is also the SIG-debugging group, if you are interested in participating in the design and implementation of Wasmtime's debugging story.

https://github.com/bytecodealliance/meetings/tree/main/SIG-Debugging

[DrRuhe]

Thanks for the pointers! I'll try to make it to the Debugging SIG Meeting.