ABI for `syscall` and `call` instructions

The current ABI that we use for making syscalls and calls is quite confusing (I now I came up with it at some point, but after not thinking about it for a while, it is difficult even for me to figure out what's the right place to pad the stack etc.). So, I'm thinking we should adapt some simple rules - even if they are slightly less efficient.

The core issue that creates complexity is that after a syscall or a call from the callee's standpoint, the stack is only 16 elements deep, but the caller may have other data on the stack. So, for example, the below procedure when invoked via exec would just drop 4 elements from the stack, but if call-ed or syscall-ed, it would also "shift-in" 4 elements at the end of the stack.

export.foo.1
    dropw
end

To illustrate:

# => [X0, X1, X2, X3 | part not visible to the callee -> X4, X5, ...]
dropw
# => [X1, X2, X3, ZERO | part not visible to the callee -> X4, X5, ...]

This creates an awkward situation when we need to compensate for this shifting in by doing something like:

export.foo.1
    dropw movupw.3
    # => [ZERO, X1, X2, X3 | part not visible to the callee -> X4, X5, ...]
end

Basically, the called procedure is responsible for maintaining the integrity of the stack for the caller.

Instead, I'm thinking we should have the following rule:

When invoking a procedure that is supposed to be call-ed or syscall-ed say that the elements at the top of the stack which are not return values should be assumed to be garbage upon return.

For example:

# Input stack: [param1, param2, parm3, param4, GARBAGE, GARBAGE, GARBAGE]
# Output stack: [return1, return2, garbage, garbage, GARBAGE, GARBAGE, GARBAGE]
export.foo.1

So, the caller will be responsible for padding the top of the stack with extra zeros (or w/e other values), and the callee would be responsible for making sure that the stack is exactly 16 elements deep upon return without worrying about maintaining the values on the top of the stack.

This would result in some more push/drop operations for the caller - but would probably be slightly more efficient for the callee.

Another side-effect is that we may not be able to execute the same procedure using call/syscall and exec. For syscalls that's not an issue, but for calls it may be. But I think this is probably fine as we should move away from allowing the same procedure to be both call-able and exec-able (and hopefully, soon we'll have annotations in MASM to support this).

cc @bitwalker @greenhat

0xPolygonMiden / miden-base

ABI for `syscall` and `call` instructions #685