Currently tablet pipes are accepted by the "system" actor, which causes pipe servers to start in the System thread pool. However, these system pipes mostly just unpack and forward events to the "user" actor in the User thread pool. This forwarding is mostly unnecessary and spends a lot of cpu time jumping between threads. Ideally we would want events to arrive directly at the user actor, without any need for extra wrappers. Ideally we would want to bypass extra forwarding on the client side as well, but it may be more challenging.
I think I know how to remove these forwarding layers by introducing actor aliases (where the same IActor instance has multiple addresses). Pipe servers would then register a new address for the "user" tablet actor, and all events would arrive there directly at the target actor without any need for additional layers. Tablet's default handler would need a special handler for pipe server messages, but it shouldn't be a big deal. A happy coincidence would be ev->Recipient pointing to the server address, which would help shards identify which shards this message belongs. Using an alias would also help reject stray messages after the pipe has been closed.
Some challenges:
We would need to buffer and forward pipe establishing attempts from the system to the user actor, this is local to the process, so no compatibility concerns.
We would need some actor to handle session subscriptions, but we already have the executor, where this pipe handling would go.
While executor would handle pipes we would need to register aliases for the tablet actor, not the other way around.
We would need to negotiate with the client whether server supports direct forwarding. This way when both the client and the server support it pipes wouldn't need to wrap the message into TEvSend/TEvPush.
We would need to synthesize TEvServerConnected/TEvServerDisconnected without changing the order, this would be possible using PushFront in lock-free intrusive mailboxes.
Tests use ev->GetRecipientRewrite() to detect requests arriving at the tablet actor. We would probably need the actor system to "auto-rewrite" events to the primary address when encountering aliases.
Pipes currently have support for SeqNo of the last forwarded events, how do we support it without extending events?
Currently tablet pipes are accepted by the "system" actor, which causes pipe servers to start in the System thread pool. However, these system pipes mostly just unpack and forward events to the "user" actor in the User thread pool. This forwarding is mostly unnecessary and spends a lot of cpu time jumping between threads. Ideally we would want events to arrive directly at the user actor, without any need for extra wrappers. Ideally we would want to bypass extra forwarding on the client side as well, but it may be more challenging.
I think I know how to remove these forwarding layers by introducing actor aliases (where the same IActor instance has multiple addresses). Pipe servers would then register a new address for the "user" tablet actor, and all events would arrive there directly at the target actor without any need for additional layers. Tablet's default handler would need a special handler for pipe server messages, but it shouldn't be a big deal. A happy coincidence would be
ev->Recipientpointing to the server address, which would help shards identify which shards this message belongs. Using an alias would also help reject stray messages after the pipe has been closed.Some challenges:
ev->GetRecipientRewrite()to detect requests arriving at the tablet actor. We would probably need the actor system to "auto-rewrite" events to the primary address when encountering aliases.