Open nahsi opened 2 years ago
It is highly possible that this is misconfiguration on my side. Although I spent a lot of time trying to understand what is wrong with no avail.
Hi @nahsi,
ui_config: enabled: true metrics_provider: "prometheus" metrics_proxy: base_url: "https://victoria-metrics.service.consul" dashboard_url_templates: service: !unsafe "https://grafana.service.consul/d/{{Service.Meta.dashboard}}"
This configuration structure is only valid when providing the UI configuration to the Consul agent via HCL or JSON–the latter of which can be provided using the server.extraConfig
or client.extraConfig
config options.
Since you're configuring this as YAML, I assume that you're trying to configure this for a Consul cluster that is deployed on Kubernetes. If so, you can use a slightly altered configuration that will be correctly deployed by the Helm chart. Try enabling the metrics proxy by adding the following snippet to your Helm values file.
ui:
metrics:
enabled: true
provider: prometheus
baseURL: https://victoria-metrics.service.consul
dashboardURLTemplates:
service: !unsafe "https://grafana.service.consul/d/{{Service.Meta.dashboard}}"
These configuration parameters are documented in the Helm chart's docs at https://www.consul.io/docs/k8s/helm#v-ui-metrics.
I noticed that https://www.consul.io/docs/connect/observability/ui-visualization does not correctly document how to configure these settings when using the Helm chart. I'll create a PR to address this.
Hi @blake, hi @johncowen
I'm not running Consul in K8S, running it on bare metal.
It's just I'm configuring Consul with Ansible and store configuration in Ansible inventory in yaml, then copy to host with to_nice_json
filter:
- name: create consul config
tags: config
copy:
content: "{{ consul_config | to_nice_json }}"
dest: "{{ consul_dirs.main.path }}/consul.json"
owner: "{{ consul_user }}"
group: "{{ consul_group }}"
mode: 0640
validate: "consul validate -config-format=json %s"
notify: restart consul
Sorry for the confusion, I'm so very used to dealing with Consul/Nomad/Vault configs in yaml that I assume everyone do the same.
This is how the resulting json looks like on host:
"ui_config": {
"dashboard_url_templates": {
"service": "https://grafana.service.consul/d/{{Service.Meta.dashboard}}"
},
"enabled": true,
"metrics_provider": "prometheus",
"metrics_proxy": {
"base_url": "https://victoria-metrics.service.consul"
}
In the documentation it was pretty clear to me that configuration of Consul in K8S and outside uses different options.
@blake sorry, can you reopen this ticket?
My apologies. It closed when the PR was merged, but before we had resolved this conversation.
Have you restarting the agent after making this configuration change to enable the metrics proxy? If so, are you still seeing the 404 errors?
@blake yes I did, it did't help.
I will provide more info and do some testing soon, please don't close this issue :)
This may be related, a strange thing is it seems Consul is not updating the "Host:" header when making the query, so my load-balancer gets the prom-queries it sends them straight back to consul and returning a 404.
@tristanmorgan I ran into this one too and found this PR: https://github.com/hashicorp/consul/pull/13071
tl;dr: you can now use the add_headers
configuration option to force set the Host header, that fixed the issue for me.
Overview of the Issue
Cannot setup metrics in topology view. Configuration:
When I enable debug logs and click on generated queries from the log it works fine. For example:
Only unavailable from the consul ui.
Reproduction Steps
Describe the solution you'd like
Metrics to be displayed in topology UI
Consul Version
1.11.3
Browser and Operating system details
chromium, gentoo, amd64 metrics database is VictoriaMetrics
Screengrabs / Web Inspector logs