Issues writing to Grafana Mimir

[adberger]

Hi there, promising project. Unfortunately I have a problem with the configuration.

Prometheus configuration (deployed via https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack):

prometheus:
  prometheusSpec:
    remoteWrite:
     # doesn't work
     - name: cortex-tenant
       url: http://cortex-tenant.default.svc:8080/push
       writeRelabelConfigs:
         - targetLabel: namespace
           replacement: tenant-a
     # works
     - name: direct
       url: http://mimir-nginx.mimir-test.svc:80/api/v1/push
       writeRelabelConfigs:
          - targetLabel: namespace
            replacement: tenant-a

Cortex-Tenant configuration:

listen: 0.0.0.0:8080
listen_pprof: 0.0.0.0:7008
target: http://mimir-nginx.mimir-test.svc:80/api/v1/push
log_level: debug
timeout: 10s
timeout_shutdown: 10s
concurrency: 1000
metadata: false
log_response_errors: true

tenant:
  label: namespace
  label_remove: false
  header: X-Scope-OrgID
  default: cortex-tenant-default
  accept_all: false

In Mimir and also in cortex-tenant I see the following errors popping up:

Mimir:

10.244.0.87 - - [24/May/2023:10:58:40 +0000]  405 "GET /api/v1/push HTTP/1.1" 0 "-" "cortex-tenant" "-"
10.244.0.87 - - [24/May/2023:10:58:40 +0000]  405 "GET /api/v1/push HTTP/1.1" 0 "-" "cortex-tenant" "-"
10.244.0.87 - - [24/May/2023:10:58:40 +0000]  405 "GET /api/v1/push HTTP/1.1" 0 "-" "cortex-tenant" "-"
10.244.0.87 - - [24/May/2023:10:58:41 +0000]  405 "GET /api/v1/push HTTP/1.1" 0 "-" "cortex-tenant" "-"

cortext-tenant:

time="2023-05-24T11:11:56Z" level=error msg="proc: src=10.244.0.86:54214 req_id=30cb3077-f5bd-4861-80fa-e485a56995f4 HTTP code 405 ()"
time="2023-05-24T11:11:56Z" level=error msg="proc: src=10.244.0.86:54214 req_id=90bcdf68-4c83-45ad-a68b-d23d3e4c6f53 HTTP code 405 ()"
time="2023-05-24T11:11:56Z" level=error msg="proc: src=10.244.0.86:54214 req_id=1d746aa7-fcf7-4b6b-9715-9adbf6313453 HTTP code 405 ()"
time="2023-05-24T11:12:01Z" level=error msg="proc: src=10.244.0.86:54214 req_id=c126f0be-0350-4afc-82e4-08497bbe1690 HTTP code 405 ()"

HTTP method GET doesn't seem right (see https://grafana.com/docs/mimir/latest/references/http-api/#remote-write).

Maybe you can help me?

[blind-oracle]

Hmm I guess something changed either on Cortex/Mimir side or in FastHTTP lib. Fixed it by explicitly setting method to POST in v1.9.1, please try

[adberger]

Wow, really fast feedback. Appreciate :)

It works now!