Better outlining when software tracing

[ptersilie]

Currently, the trace builder has a complicated outlining logic to decide which blocks need processing and which are being ignored (i.e. outlined). Here's an idea to simplify this by moving the outlining logic into the tracing part so that the trace builder part never sees blocks that need to be outlined:

bb1:
   yk_record_block(1)
   g_outlining = true // this is a thread_local
   call extern_func(callback_func) // and external function with callback into mappable code
   g_outlining = false

bb2:
  yk_record_block(2)
  ...

and

fn yk_record_block(bid) {
    if g_outlining == false {
        return;
    }
}

Basically, at AOT compile time we toggle a global thread_local every time we call and return from an external function. Inside the basic block recording function, we then only record blocks when we are not outlining. This way the software tracer will only record blocks we care about, and we could rip out the whole outlining logic from the trace builder (once/if we remove the hardware tracer), making the trace builder much simpler.

[ptersilie]

@Pavel-Durov This isn't something we need to act on immediately and likely only relevant once we decide to remove the hardware tracer completely. I just raised this so we don't forget.

[ltratt]

Maybe what we need is a counter tracing: u64. It defaults to 0. When we start tracing, we increment it. If we start outlining, we increment it again, and we finish outlining we decrement it. This means that we only record blocks iff tracing == 1: any other value means either "we're not tracing at all" or "we're in (possibly nested) outlining".

[ptersilie]

Due to the performance hit I would only implement this once we have a tracing interpreter and then we don't need the "we are not tracing" case. I don't think nested outlining is an issue.

Although, recursion is still a worry as we still need outlining logic for that, unless we can find a better approach for that too.

[ltratt]

I think nested outlining is an issue because you need to differentiate "I've finished executing an outlined function and I need to restart tracing" from "I've finished executing an outlined function but there are other outlined functions on the stack so I don't need to restart trcaing".

[ptersilie]

Oh I see. Yes you're right!