pageserver: optimize heatmap downloads with S3 conditional GET

neondatabase / neon

Neon: Serverless Postgres. We separated storage and compute to offer autoscaling, code-like database branching, and scale to zero.

https://neon.tech

Apache License 2.0

14.49k stars 421 forks source link

pageserver: optimize heatmap downloads with S3 conditional GET #6199

Open jcsp opened 9 months ago

jcsp commented 9 months ago

In #6123, periodic download of heatmaps is added for secondary mode locations.

While downloading the heatmap is not bandwidth intensive, we can avoid transferring any body at all if the heatmap has not changed: https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html (If-None-Match or If-Modified-Since)

Currently, the S3Bucket type doesn't have support for setting such headers during a GET.

jcsp commented 5 months ago

The download code now stores the etag (last_etag) and checks it before downloading body: all that remains is to augment the remote storage client's GET path with the ability to set the conditional download header.