Open petuhovskiy opened 11 months ago
One advantage of one TCP connection per tenant is, though, that we get
I haven't kept up with how good gRPC is in either of those areas.
Let's perhaps turn this into a short RFC to discuss there?
RFC with implementation details make sense, I can create it later when/if I'll start working on this task. For now it should be ok to have a small conversation here.
observability through standard tools
What standard tools do you have in mind? netstat
can show port and destination safekeeper address, and we can count number of active sockets, but it's hard associate specific tenant/timeline with a socket.
fairness between tenant connections through TCP
HTTP/2 streams are fair by default, and we also can specify weights if needed. I wouldn't expect any fairness differences from existing setup.
Motivation
Currently we use a standard postgres way to download WAL –
START_REPLICATION
described in the postgres docs https://www.postgresql.org/docs/current/protocol-replication.htmlThe main problem with it is scalability, because each connection uses a separate TCP connection. TCP connections are not cheap to establish and maintain, and number of concurrent ps<->sk TCP connections is limited by the ports count.
The less improtant problem is protocol extensibility, which can also be improved, but may be ommitted in the first iteration since there is no bottleneck in it.
We already saw issues with network overloading, which manifested in DNS resolution failures. These issues were reproduced around pageserver/safekeeper restarts in a presence of 10k+ active timelines.
DoD
TCP connections between pageservers and safekeepers should be multiplexed, and there should be
O(1)
real network connections between specific pageserver and safekeeper.Implementation ideas
https://neondb.slack.com/archives/C039YKBRZB4/p1683125758173469?thread_ts=1683097406.177109&cid=C039YKBRZB4
Use gRPC. It has HTTP/2 to multiplex many streams within a single TCP connection. It also provides convenient protocol extensibility.
I also think if we will have a gRPC connection between every safekeeper and pageserver, we can use it to bypass broker in delivering timeline updates from safekeepers to pageserver.
Tasks
Other related tasks and Epics