Move scylla config creation into the operator

[tnozicka]

Is this a bug report or feature request?

• Feature Request

What should the feature do: Scylla config should be built in the operator, then represented as a CM/Secret and mounted into the container. Same goes for the args. Currently it is build in a sidecar which has several disadvantages:

• the sidecar has to run exactly the same image as the operator which is impossible to enforce and breaks with rolling tags
• if any remote call fails it cases a restart
• we can't trigger rolling restarts on changes
• wastes live api call, the operator already has caches with data that needs to be substituted
• is build again for every replica, instead of the operator doing it once
• if config building fails we end up with 1 replica down, while the operator would find out sooner and would stop before trying the first replica
• it can't be revisioned
• needs extra RBAC
• it doesn't honor the Three laws of controllers (ownerRefs, adoption, release)
• it needs additional arguments to know e.g. the manager-agent secret location, while the operator already has all of the prerequisites and caches
• we can't prevent a rollout with invalid config

[scylla-operator-bot[bot]]

The Scylla Operator project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 30d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out

/lifecycle stale

[tnozicka]

/remove-lifecycle stale /triage accepted