pageserver: handle aux file keys in imports

neondatabase / neon

Neon: Serverless Postgres. We separated storage and compute to offer autoscaling, code-like database branching, and scale to zero.

https://neon.tech

Apache License 2.0

14.28k stars 408 forks source link

pageserver: handle aux file keys in imports #8202

Closed skyzh closed 2 months ago

skyzh commented 2 months ago

currently it does not work with the page server import functionality https://neondb.slack.com/archives/C033RQ5SPDH/p1719553893686989?thread_ts=1719403793.586119&cid=C033RQ5SPDH

We need some logic to auto-detect if there exists v2 aux file keys during the import. If yes, we should put last_aux_file_policy = v2 in the index_part file; or forcing v2 for all imported tenants.

jcsp commented 2 months ago

or forcing v2 for all imported tenants.

That sounds fine to me. The import API isn't used in prod under normal circumstances, and if it doesn't work for people with weird beta-feature stuff in their databases that's fine.

knizhnik commented 2 months ago

Please notice that right now import_file just ignores all AUX files (replication slots,...). So if we are importing data from tar, then this files are missed. It was fixed in #6560

jcsp commented 2 months ago

We can also consider removing this API, as it is not used in production. Last time we discussed removing it there was concern that it was used in some support cases by the compute team.