Executable buffer relocations

[adyanth]

I am trying to build a REPL+JIT compiler using this. It works for the most part, but I ran into an issue as described below.

The flow is: Rust execute JIT buffer: dynasm!(ops; ... call into rust runtime) -> runtime: dynasm!(ops; new functions here); ops.alter(... call site -> jmp new_call_site);

The runtime function modifies the original ops Assembler to include new functions and alters one instruction in the original buffer.

Doing this works fine when the functions are small. But a simple example of when I have larger functions being compiled like this, even before the ops.alter stage, in the debugger, I can no longer print the memory location of the assembly that called the rust runtime. Looks like the executable buffer is being reallocated, thus invalidating my return pointer from Rust back to assembly causing a segfault.

Is there a way to not have the ops executable buffer relocate, or is there a way to extend the buffer in place so as to keep all the instruction pointers valid? (This is assuming my analysis of the issue is correct)

Edit: Basic info I missed providing: Versions:

dynasm = "2.0.0"
dynasmrt = "2.0.0"

Calling style:

ops.commit().unwrap();

let jitted_fn: extern "C" fn() -> i64 = {
      let reader = ops.reader();
      let buf = reader.lock();
      unsafe { mem::transmute(buf.ptr(start)) }
};
jitted_fn()

PS: Thanks for such a stable and wonderful implementation of dynasm!

[CensoredUsername]

The basic Assembler's I wrote for dynasm-rt don't natively support any kind of buffer modification while executing the assembled code. That's not to say that doing such a thing is impossible, but the runtime implementations provided are meant to be simple first.

As you've noticed, the default Assembler might move the underlying code buffer if the previously allocated buffer turns out to not be big enough. There's unfortunately no nice portable way to grow it in place, and a fixed size buffer has several other drawbacks, so this is why it uses a resizing buffer. It operates on the idea that it isn't accessed in a re-entrant manner. The lock you're supposed to hold while executing the assembled code even explicitly precludes this, so you're likely already circumventing this in some way. Unfortunately this kind of behaviour is more complex than what I could provide as a ready-to-use solution.

If you want to do re-entrant assembling you'll really have to bring your own runtime implementation (or at least, add a bit on top of the basic Asssembler). One way to deal with this is to rewrite the call stack to fix up the return pointers as the buffer is moved. You can query the assembler for the current buffer location/length before and after commit to detect if a move has happened, and if it has you can walk the callstack and fix up any return addresses in the old range. Of course, for this to work your assembled code needs to actually emit sane frame pointers so you can actually do this call stack walk. And of course, this is calling convention dependent.

Alternatively, you can bring your own assembling runtime. Either just one with a fixed buffer size, some nonportable way of contiguously expanding the buffer, or something even fancier. There's many solutions here, each with their own intricacies and drawbacks. If you need some pointers for that feel free to ask, but be aware that re-entrant JIT-ting is a complex beast by its definition ;)

[adyanth]

I only hold the lock for the duration to get the pointer to the buffer. While executing, I no longer have it locked.

Glad to know that indeed was the issue. I can definitely fix the call stack myself, I could not find any way to get the raw address to the Assembler/Executable buffer before, but let me look more in depth. I do have frames set up (for GC), so there should not be a problem of walking the stack.

I know it is complex, wanted to see how far I can push it by myself. Thanks for your pointers, will surely pop back in here if I hit a wall I cannot scale :)

[CensoredUsername]

You can get the raw address right now by just casting a pointer to the begin of the jitted buffer to a usize. That said, that requires getting a lock and stuff which really isn't necessary as Assembler actually just has that address available internally so at some point in the future I'll likely just add a method to access that directly. Good luck with your endeavours!

[adyanth]

I cannot believe that actually fixed it! Rewrote my assembly's stack pointers, as well as the return pointer for rust and now it works beautifully with relocations.

Thank you!

[CensoredUsername]

You're welcome!