Implement a transactions log for co-located actors

[aagapi]

At the moment, all actors, whether deployed on the same or different RTSs communicate via distributed transactions that include their input queues. Specifically, as the "destination" actor processes the message, he creates and commits a transaction that contains internal state updates, possibly enqueues upstream to other actors, and consumption of the input message.

For actors co-located on the same RTS, we have the opportunity to delay committing their transactions until either: 1) The program flow requires some actor not co-located with them to participate, thus we need to reach out to it over the distributed queue; or 2) Some time expires (we thus ensure that eventually the work from this RTS node is committed to the distributed DB, thus we ensure that upon loosing the node to a failure, we do not have to re-do the work).

Doing this will allow us to bulk together many transactions from co-located actors, considerably increasing our throughput. This is especially true if actors that communicate most to each other are co-located together on the same RTS by the placement algorithm.

Implement the mechanism above, including the required transactions log.

[plajjan]

I think the simpler case and natural first step is to bulk together the updates from multiple continuations of the same actor up to a certain time or until there are no more continuations.

[aagapi]

Yes, that is a separate optimization in my view, although it c'd make use of the same transactions log.

[plajjan]

I'm saying it's the natural first step because it doesn't need a new transaction log. It's just a bit of code reorg.

Now I'm really eager to try and implement that kind of batching ;)

[aagapi]

Well, I argue simply batching together multiple updates for the same actor present in its input queue would still incur batching several "transactions" together, i.e. effects from executing those continuations by that same actor in the RTS. So I am suggesting since we need to do the same thing for this PR (which w'd batch together txns among different but co-located actors), that we use the same transaction log for this. I think it w'd make for the cleanest code. I will do a PR and we can decide upon review if it looks clean enough?

[plajjan]

@aagapi for the change I'm thinking about we don't need a transaction log at all, but yeah, let's write some code :)