Open suslikas opened 9 months ago
Thanks @suslikas . It sounds like we need to expose an option to set X-Scope-OrgId
HTTP header on the Prometheus Remote Write sink. This seems like a feature request rather than a bug so I'll update the labels.
From my point of view is a bug, because in docs wrote:
If set, a header named X-Scope-OrgID is added to outgoing requests with the value of this setting.
Not forget, without this header auth for multi-tenant prometheus not work, because auth request contain user+pass+tenant
But as you wish :)
Aha, I stand corrected, thanks @suslikas . I see the intent is for tenant_id
to control that. It does seem to be implemented so there may be some bug.
Fast solution with Haproxy
sinks:
out_websec_mimir:
type: prometheus_remote_write
inputs: [ "log_to_metric" ]
endpoint: http://websec-logwrap-general-haproxy.websec-logwrap-general.svc.cluster.local/api/v1/push
listen mimir-wrapper
bind 0.0.0.0:80
mode http
balance roundrobin
option httplog
http-request add-header X-Scope-OrgId "websec-logwrap"
http-request add-header X-Haproxy-Wrappwer "yes"
http-request add-header Authorization "Basic ZG...A=="
server mimir-aws-lb-1 xxx.us-east-1.elb.amazonaws.com:443 ssl verify none check
Same problem, mimir
and vector
involved. +1 on this
Just adding that we're not experiencing this problem (with 0.35 or 0.36):
[sinks.prw]
type = "prometheus_remote_write"
inputs = [ "..." ]
endpoint = "http://.../api/v1/push"
tenant_id = "tid"
healthcheck.enabled = false
Perhaps something with the auth headers is overriding the X-Scope-OrgId
header
A note for the community
Problem
prometheus_remote_write miss tenant_id (X-Scope-OrgId header) during basic auth. As result not work health check and data can't be sended to Prometheus (Grafana Mimir)
Configuration
Version
Kubernetes, 0.35.0-debian
Debug Output
Example Data
No response
Additional Context
Auth with curl work fine
References
No response