search

neondatabase / neon

Neon: Serverless Postgres. We separated storage and compute to offer autoscaling, code-like database branching, and scale to zero.
https://neon.tech
Apache License 2.0
15.1k stars 439 forks source link

docs: GetPage@LSN Compute Request Documentation #7293

Open ericjest opened 7 months ago

ericjest commented 7 months ago

Like in #6887, I've read over neon/docs/pageserver-processing-getpage.md. Based on the info in #6887 and by taking a look at the Neon pageserver source code, I understand the network (i.e. TCP server) flow on the pageserver side and how the pageserver constructs a GetPage response. However I'm having trouble understanding how the TCP client (the Compute/postgres node) makes a network request. I can't seem to find any source code in the Neon compute_tools directory.

Can any documentation be added to give a brief explanation of how the compute node communicates with the pageserver (i.e. when making a GetPage@LSN request)? I'm interested in the networking details/stack in particular but any explanation would help a lot. Thank you!

hlinnaka commented 5 months ago

Hi! The client for that is in the neon Postgres extension, see https://github.com/neondatabase/neon/blob/dc2ab4407f8b9636a6e570818154f21fde14b9ce/pgxn/neon/pagestore_smgr.c#L2374. It's a bit complicated because the client maintains a queue of optimistically issued "prefetch" requests, and needs to deal with network errors and retries. The lower level function that sends one request is in https://github.com/neondatabase/neon/blob/dc2ab4407f8b9636a6e570818154f21fde14b9ce/pgxn/neon/libpagestore.c#L719