Support running on non-default cluster DNS settings

[sbernauer]

Reported in https://github.com/orgs/stackabletech/discussions/35

Currently we hard-code svc.cluster.local in a lot of places. This is bad, as some user installations have a non-default installation by doing stuff such as https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/, e.g. --cluster-domain=<default-local-domain>. We need users to be able to configure the Service DNS suffix.

Possible solutions:

1. Add an cli flag (or better: env var) to the product operators which overrides the svc.cluster.local default.
2. More ideal: Somehow let operators detect the DNS suffix of the k8s cluster and use that
3. Maybe listener-operator can help us here
4. [...]

[fhennig]

maybe a bit of a tangential question, but

Add an cli flag to the product operators which overrides the svc.cluster.local default.

How is the customer supposed to use CLI flags? since the Operator is in the container

[sbernauer]

Good question! I though there is some sort of configuration that is passed to our operators but I can't find it, so it might be wrong.

Anyway, in case we can not automatically detect it the new setting must go into the values.yaml of the operators, the helm-chart can than e.g. set an env var or cli flag on our operators

[nightkr]

values.yaml into env var is largely the way to go for those config things IMO. We use it for secret-op's kubelet dir already.

For this specifically, it looks like we could auto-detect it from the resolvconf, but there should definitely be an option to override it as well.

[NickLarsenNZ]

Just pointing out that the overridable K8s DNS suffix is cluster.local (as opposed to svc.cluster.local) Services would still be foo.bar.svc.$SUFFIX.