search

neondatabase / neon

Neon: Serverless Postgres. We separated storage and compute to offer autoscaling, code-like database branching, and scale to zero.
https://neon.tech
Apache License 2.0
13.7k stars 385 forks source link

support split image layers during create_image_layers #7796

Open skyzh opened 2 months ago

skyzh commented 2 months ago

Currently, the partition algorithm ensures a single create_image_layers job is not too large, and each time it only creates a single image layer, regardless of the size. This does not matter for relational data, but needs to be fixed for metadata keys.

skyzh commented 1 month ago

This is only a problem with metadata keys, but if metadata gets too large (i.e., >= 4GB), basebackup will consume a lot of memory. Therefore, this is not a priority to fix it now.

skyzh commented 2 weeks ago

turned out also be a good idea for image layer compression b/c we don't know the size of each layer in advance (cc @arpad-m)