Using an existing / external Consul cluster

[hamishforbes]

I already have Consul deployed and working in my environment and would like to reuse the existing cluster.
There doesn't seem to be an easy way to override the consul hostname though.

I might be missing something but there also seem to be places were hardcoded config is overriding values from $._config which is a bit confusing and tricky to deal with.

For example if i set the consul hostname for the distributor in the place I would expect to:

{
  _config+:: {
    distributorConfig+: {
      'distributor.ring.consul.hostname': 'my-consul.service',
    }
  }
}

It is overriden in https://github.com/grafana/cortex-jsonnet/blob/1.9.0/cortex/distributor.libsonnet#L37 and I have to set those values in $.distributor_args+:: instead / as well.
This appears to be the case for ruler and compactor too.

On a semi-related note, the distributor HA Tracker is also hardcoded to use etcd when it could use Consul as well.
Is there a reason for preferring etcd here? (e.g. performance issues in large production environments for this use case)

[pracucci]

What we typically do is to override CLI flags in <service>_args (eg. distributor_args+:: { ... }) and not distributorConfig, as you already pointed out.

On a semi-related note, the distributor HA Tracker is also hardcoded to use etcd when it could use Consul as well. Is there a reason for preferring etcd here?

@gouthamve Could follow up on this. If I remember correctly, there were some perfs / scalability issues.

[hamishforbes]

Ok well at least i'm not just doing something stupid!

It does feel a bit odd to have the default configs in the libs be higher priority than the values from _config though.

For this particular issue (consul hostname) would a change abstracting that out to a top level config var be worthwhile?
e.g. $._config.consul.{host,port}