storcon: improve drain and fill shard placement

[VladLazar]

Problem

While adapting the storage controller scale test to do graceful rolling restarts via drain and fill, I noticed that secondaries are also being rescheduled, which, in turn, caused the storage controller to optimise attachments.

Summary of changes

• Introduce a transactional looking rescheduling primitive (i.e. "try to schedule to this secondary, but leave everything as is if you can't")
• Use it for the drain and fill stages to avoid calling into Scheduler::schedule and having secondaries move around.

Checklist before requesting a review

• [ ] I have performed a self-review of my code.
• [ ] If it is a core feature, I have added thorough tests.
• [ ] Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• [ ] If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• [ ] Do not forget to reformat commit message to not include the above checklist

[github-actions[bot]]

2910 tests run: 2793 passed, 0 failed, 117 skipped (full report)

---

Code coverage* (full report)

• functions: 32.4% (6856 of 21131 functions)
• lines: 49.9% (53406 of 107092 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
de1f108776d6757412628d82efc3829405d063ce at 2024-06-22T14:28:56.293Z :recycle:}

[VladLazar]

test_graceful_cluster_restart is failing because this pr is not based on: https://github.com/neondatabase/neon/pull/8061