[discussion] jmp or call to trampoline

[wangbj]

when we jump into our temperorily trampoline, we have two choices use either jmpq (e9)or callq(e8). It seems rr and liteinst both use jmpq, but we're using callq.

There're two reasons I can think of why jmpq is used:

1) when patching a sequence like:

syscall    ; (2B)
retq         ; (1B)
nop         ; (xxB)

with callq, we have a double return issue - though it can be solved, because we just have to pop the return address saved by our generated callq, but it looks more confusing than jmpq solution

2) stack alignment

callq pushes return address (8B) on to stack, meaning we must do stack alignment by ourself -- before calling any high level functions - it is fine by our trampoline, because it is entirely assembly, and does not use SSE instruction who requires stack alignment.

We do have our own benefits by using callq, because we don't have to worry about return address, so we have much less interim trampoline to generate, because we can reuse them, we can also pre-populate our interim trampolines, rather than generate them on the fly, this makes our code base smaller, and easier to implement.

[rrnewton]

I really like the "simulated double return" solution, to allow us to continue to use CALLQ (and thereby reuse trampolines across multiple call sites).

For the alignment, it sounds like we:

1. add some dynamically computed amount to the stack in the trampoline's code (aligning)
2. set up and do the function call to the instrumentation func
3. subtract the same amount after returning

Is that right? So that's essentially the (variable sized) "stack frame" associated with the trampoline itself.

[wangbj]

Yes that has been done: https://github.com/iu-parfunc/systrace/blob/bac4b39fdf21921cc30542b1a74a567b68d05436/src/trampoline.S#L60

I hope @devietti can review the code :)

[devietti]

Looks good to me, though I am far from an expert here!