Epic: timeline detach ancestor

[koivunej]

Implement the timeline_detach API endpoint.

Motivation

6888

DoD

Implementation ideas

### Tasks
- [ ] https://github.com/neondatabase/neon/pull/7228
- [ ] https://github.com/neondatabase/neon/pull/7285
- [ ] https://github.com/neondatabase/neon/pull/7422
- [ ] https://github.com/neondatabase/neon/pull/7456
- [ ] https://github.com/neondatabase/neon/pull/7639
- [ ] https://github.com/neondatabase/neon/pull/7706
- [ ] https://github.com/neondatabase/neon/pull/7779
- [ ] https://github.com/neondatabase/neon/pull/7813
- [ ] https://github.com/neondatabase/neon/pull/7650
- [ ] https://github.com/neondatabase/neon/pull/7833
- [ ] #6888
- [ ] https://github.com/neondatabase/neon/issues/7830
- [ ] https://github.com/neondatabase/neon/pull/8229
- [ ] drop tenant before entering restart: https://github.com/neondatabase/neon/pull/8354#discussion_r1677687592
- [ ] https://github.com/neondatabase/neon/pull/8332
- [ ] https://github.com/neondatabase/neon/pull/8353
- [ ] https://github.com/neondatabase/neon/pull/8354
- [ ] https://github.com/neondatabase/neon/pull/8430
- [x] failpoint testing between the 3 different completion points
- [x] storcon: should just pass through the ApiError from pageserver
- [ ] --- Cutline for considering project functionally complete ---
- [ ] Persist the ancestry of timelines after detach, to enable future WAL recovery; fixme comment with full example added in https://github.com/neondatabase/neon/pull/8354#discussion_r1677681420
- [x] should L1s be written with the actual highest key? probably not, because we wouldn't know the highest key except only after completing the rewrite. Resolution: they should not be.
- [ ] pageserver config: default concurrency items from options
- [ ] unify reset_tenant and complete detaching
- [ ] there seems to always be one rewritten layer, but test asserts a case where that does not happen; investigate, is the straddling check wrong or test assertion does not work?
- [ ] TODO: discuss fate of WAL recovery for these tenants

Other related tasks and Epics

[koivunej]

Next steps:

• hoping to start working on this week
• other team members will read RFC this week
• @arpad-m will look into interactions with DR (initdb, specifically)

[koivunej]

Last week I did not get to start this, however, not everyone had a chance to look at the PR either. We did have one call about the RFC, summarized here: https://github.com/neondatabase/neon/pull/6888#issuecomment-1981093123

This week:

• hopefully start it after the RFC is merged

[koivunej]

This week:

• hopefully we can close the pending RFC discussion:
  • differences in approach with control plane team (waiting for their suggestion)
  • decide on WAL DR being made possible or not
    • recording the lineage to index_part.json would be an ever growing list
• merge the PR
• start the work

[koivunej]

Starting implementation this week even if RFC discussion still is waiting for alternate proposals.

[koivunej]

Per discussion:

• we avoid implementing the "truncation" of in-memory layers by flushing ancestor to disk
• we can avoid implementing the head-object for an optimization completely by "periodically" uploading a new index_part.json, which will limit the amount of possible copied but not yet visible to say magical number 20 layers

[koivunej]

Update for the week:

• implementation is making good progress
• currently unimplemented:
  • ancestor blocking (we don't want multiple detach_ancestor at the same time)
  • persistent timeline "is detaching" recording (gc inhibition)

After discussing with John:

• instead of online timeline detach, we should:
  • avoid new locks on the read path
  • require that branch has had a compute started and produced some WAL so that we have a previous record lsn
  • assume maintenance mode
  • shutdown and restart the whole tenant

So, reworking the implementation.

[koivunej]

Reworking the implementation has been slow, but some:

The maintenance mode requirement can be relaxed to maintenance mode, which is only required for page_service connections because we can "rewrite" the ancestor+ancestor_lsn in RemoteTimelineClient. If we accept that the API endpoint leaves the in-memory state of Timelines as out-of-sync, then it is possible for the control plane to just reset the tenant after the operation.

Another detached ancestor case was discovered: If the "new main" gc_cutoff has progressed beyond the ancestor_lsn, then there is no real connection between the two timelines. This also means we must not reparent any timelines but only do the metadata change. So far, I've been thinking of this case as "diverged".

The uncertainty the "diverged" case brings to which the selection of timelines that were reparented means that control plane will need to query the ancestry relationships always after invoking the operation.

[koivunej]

Plan for the week:

• manual testing
• merge #7456
• stretch goals:
  • solve the compaction problem
  • hard-link copied layers
  • do copy prefix lsn and remote copying at the same time

Compaction problem:

Right now, if ancestor receives writes while the operation is ongoing, they might trigger a compaction. If there is a shutdown, then on retry we would end up with an union of the layers which might be a bad thing. Easy solutions include not allowing copying lsn prefix of L0 layers but does not scale to next compaction algorithm, only the current legacy.

[koivunej]

Plan for the week:

• merge a simplified version which does not have the compaction problem, same PR: #7456
• manual testing (first done)
• merge the RFC with the learnings -- no way around compaction problems for more restartable

[koivunej]

Plan for the week:

• continue with follow-ups from #7456 after finally getting it merged
• merge the RFC which got de-prioritized from the review

[koivunej]

Plan for the week:

1. finish up the openapi descriptions, adapt the code to work like described, document delta
2. detach two ancestors for one specific prod tenant after deploy
3. complete persistent gc+compaction blocking
   • pageserver crash (failpoint) tests

[koivunej]

Last week only "detach two ancestor for one specific prod tenant after deploy" was done for other work. Minor follow-up from that was completed:

• 7779

Not started:

• fix the lineage to always copy the detached ancestors list of detached ancestors
  • keeping only the latest is not enough for future WAL DR
  • however on delete the ancestor WAL is deleted, possibly with initdb, so we are not really supporting even a possibility of WAL DR
• does the compaction which is "induced" by detaches change the branch output? It obviously should not, but this was unforeseen, so it should be tested.

What remains:

• finish up the openapi descriptions
• complete persistent gc+compaction blocking

New for this week:

• another prod tenant (internal slack thread)

[koivunej]

Last week got trampled by testing and troubleshooting.

This week:

• complete persistent gc+compaction blocking (incl. failpoint testing)

[koivunej]

Similarly with the previous week.

This week:

• complete persistent gc+compaction blocking (incl. failpoint testing)

[koivunej]

Plan remains the same for the next few weeks when I will be on vacation.

Before that, production usage of timeline ancestor detach is required to retry with reset_tenant endpoint in case #7830 happens. Using it in for sharded tenants is also not advisable, because the endpoint is not yet idempotent, but it could be done manually.

[koivunej]

This week plan remains the same, but #7830 looks like an easy first issue disregarding testing.

[koivunej]

This week:

• complete #8229 (merge)
• complete #8430 splitting it if necessary, incl. locking in the reparented timelines