Calling tail callable functions in non-tail position.

[ElvishJerricco]

With most platforms' C ABI, the caller is responsible for popping the arguments to the callee after it returns. This helps to support getting the varargs ABI for free, among other things. With most tail calling conventions, when calling a function that may perform a tail call, the callee is usually the one responsible for popping its own arguments, so that it may reuse the stack frame for its successor. The tail called function will return by popping its own arguments and jumping to the return address. Thus, the calling convention for calling a function that may perform a tail call is somewhat different than the typical C calling convention. In fact, I believe that the typical tail calling convention is such that any function which may perform a tail call is also tail callable itself. i.e. If a function either performs a tail call, or is tail callable, it must pop its own arguments and jump to the return address. This is somewhat nice in that we only need one signature modifier; there is no distinction between "performs a tail call" and "is tail callable."

So a caller cannot perform a normal call on a callee that is tail callable, even though that caller is not doing a tail call. If it is the case that a function which performs a tail call uses the same calling convention as a tail call, then we can't simply create a normally callable wrapper that performs the tail call. Essentially, we must support tail call instructions in non tail positions. This isn't problematic, overall. The caller can simply push a custom return address before pushing args and jumping to the tail-callable function. This is similar to the typical C calling convention, except that it omits the step where it pops the arguments, as the callee will have done that itself.

But this raises the question: Should this functionality be a separate instruction? i.e. should there be a push_call instruction for calling tail callable instructions in non-tail positions? Or should the call instruction simply infer that it should use this behavior from the function signature, which indicates the calling convention?

I lean towards the former, that there should be a new instruction. It seems easier on hosts if they don't have to look up extra information while compiling call instructions. But I can imagine how bloating the number of instructions we have could be seen as problematic.

[fgmccabe]

I believe that the analysis above is not quite accurate; in particular

With most tail calling conventions, when calling a function that may perform a tail call, the callee is usually the one responsible for popping its own arguments, so that it may reuse the stack frame for its successor.

The reason is that, in languages that support full TCO, the callee does not know that it is being called in a tail position.

So, what actually has to happen is that the caller overwrites its own frame on the way to invoking the tail call.

I may be missing something here; but that is always how I have implemented it (>10 times)

[ElvishJerricco]

Sure, but overwriting its own frame is functionally identical to popping its own argument then pushing the arguments to the successor. Regardless, this means a function that may perform a tail call must be fully aware of its frame, and either it or its successor (inductively) is responsible for popping before returning to the original non-tail caller.

And you're right, the callee cannot know that it is being tail called, thus the ABI for calling a tail-callable function has to be identical in both tail and non-tail call cases.

[rossberg]

IIUC, this was more or less a duplicate of #1. The CG discussed various options in the past and ultimately resolved to what's currently specced in the proposal.