cloud_storage: robust handling of enabling/disable tiered storage

[jcsp]

Currently, toggling remote.write on a topic has quite literal effects: e.g. when you stop doing uploads, it'll just stop. Existing data stays in S3. When you switch it on again, if the local log was truncated in the meantime, you end up with a weird state in S3 with a gap in the data range.

Summarizing a list of suggestions for lifecycle operations (from @Lazin):

• disabling SI temporarily (prevent retention from deleting local data until SI is re-enabled, maybe the command should take a duration)
• disabling SI permanently (delete all data in the cloud)
• enable SI for existing partition (upload what's available locally)

JIRA Link: CORE-1035

[mmedenjak]

Follow-up - https://app.incident.io/incidents/82

[jcsp]

Related:

• 7916

• 7918

Those tickets are for debug-oriented APIs, but a lot of the wiring will be the same as something more unified.

This ticket staying open to track creating a full supported API that we would plumb into e.g. Console UI.