storage controller: rolling restart hooks

[jcsp]

Hooks to enable some external orchestrator to ensure that attachments are drained from a node before restarting it, and hint the controller to move attachments back to the node after the restart.

[VladLazar]

This week:

• work on POC and RFC

[VladLazar]

Last week:

• RFC: https://github.com/neondatabase/neon/pull/7704
• POC: implements 90% of the RFC https://github.com/neondatabase/neon/pull/7682

This week:

• Get feedback
• Work on POC to make it prod ready

[VladLazar]

Last week (2024-06-10):

• Merged a prereq pr
• Tidied up https://github.com/neondatabase/neon/pull/8014
• Worked on stabilising impl for scale test - passing now

This week (2024-06-17):

• Merge https://github.com/neondatabase/neon/pull/8014
• Work on Ansible side

[VladLazar]

There's a number of small PR's currently up for review. See the topological dependency sort below:

https://github.com/neondatabase/neon/pull/8029

https://github.com/neondatabase/neon/pull/8061 ---------------
                                                              |-------> https://github.com/neondatabase/neon/pull/8119
https://github.com/neondatabase/neon/pull/8099 ----------------

[VladLazar]

Last week (2024-06-17):

• Merged the main PR with drain and fill support
• Spent a lot of time on fine-tuning shard placement and scale testing
• Implemented Ansible side: https://github.com/neondatabase/aws/pull/1511

This week (2024-06-24):

• Test in staging
• Merge https://github.com/neondatabase/aws/pull/1511
• Open PR for scale test

[VladLazar]

Last week (2024-06-24)

• Investigated storcon segfaults => downgraded openssl
• Multiple staging tests
• Merged ansible side of things https://github.com/neondatabase/aws/pull/1511

This week (2024-07-01)

• Do graceful storcon cluster restart in ap-southeast-1 as part of the platform release

[VladLazar]

Last week (2024-07-01)

• Do graceful storcon cluster restart in ap-southeast-1 as part of the platform release - all went fine
• Do a bigger region this week. I'm storage on call, so I'll pick one and do it

[VladLazar]

Last week (2024-07-08)

• Did graceful storcon cluster restart up to and including eu-central-1
• eu-central-1:
  • Storcon went in a busy loop and was killed by k8s for not responding to /status. Fixed here
  • Restart proceeded without fill and the background optimizations gradually balanced the cluster

Next week (2024-07-15)

• Verify that cplane fix https://github.com/neondatabase/cloud/pull/15403 is released first. These rolling restarts issue many reconciles and makes the bug more likely.
• Do graceful restarts in ap-* and one us region.

[VladLazar]

Last week (2022-07-22):

• Storage release did graceful storcon cluster restart in all regions (- ap-southeast-2)

This week (2022-07-29):

• Set up an alert for ERROR logs coming from the Python script
• Enable by default?

[VladLazar]

2024-08-06:

• Added alerts: Storage Controller Cluster Restart Logs (Prod|Staging)
• Will monitor during deployments for the next couple of weeks, but closing the issue as complete