Less surprising behavior for tiered storage topics when remote hasn't been constructed

[andrwng]

Today, when you change tiered storage configs (e.g. to start using tiered storage), the underlying cloud_storage::remote() doesn't get constructed until the process is restarted. It'd be nice if the remote could be rebuilt at runtime, but this is a tricky task, considering all the places the remote leaks into different abstractions with the expectation that it is constructed once at startup.

Some surprises that come out of this:

• Tiered storage topics will tell RPK that they have tiered storage enabled, but they will be unable to upload anything.
• This can result in disk filling up because space management will not trim anything for local-only topics.

We should refine the behavior of the span in between setting tiered storage configs and the next restart. A simple strawman proposal is to at least reject topic creation. It doesn't solve everything though because topics that have already been created can have equally surprising behavior when toggling tiered storage cluster configs.

Some other, spicier options to consider:

• Nuke everything from within Redpanda rather than relying on external services to restart Redpanda. On a config change that includes something that needs-restart, perhaps we should stop/shutdown and rebuild most of the application. This seems a bit risky to do automatically, but it would avoid this problem.
• Don't allow changing tiered storage via cluster configs at all, and force users to use bootstrap yaml. If needed, maybe we'd need to make toggling tiered storage into a more explicit, heavy-weight operation that makes users confront the effects of the toggling (maybe this would be RPK driven and show topics that would be affected by toggling).

JIRA Link: CORE-2911

[hcoyote]

Wonder if we should bubble up to the health status the fact that the cluster config is in a state requiring a restart. Right now it's buried in some api the user wouldn't normally access. Or treat it as a metric state that they can alert on (similar to what we do now with some disk alert functions).

[piyushredpanda]

@hcoyote : Please file a separate ticket for your good suggestion there -- and that'll go to @aanthony-rp and team.

Let's keep this ticket for the original issue that Andrew filed.