promxy to prometheus "remote_read" calls using "query" / "query_range" instead

[BertHartm]

I've been trying to test out the remote_read functionality, but I'm seeing that promxy is not actually making the remote read calls, and is instead using the query / query_range calls when I run queries via the promxy UI.

I have the following promxy config:

global:
  scrape_interval: 1m
  scrape_timeout: 10s
  evaluation_interval: 30s
promxy:
  server_groups:
  - remote_read: true
    remote_read_path: api/v1/read
    http_client:
      dial_timeout: 200ms
      follow_redirects: false
      enable_http2: false
    scheme: http
    labels: {}
    path_prefix: ""
    query_params: {}
    anti_affinity: 10s
    ignore_error: false
    relative_time_range: null
    absolute_time_range: null
    label_filter: null
    static_configs:
    - targets:
      - localhost:9090
tls_server_config:
  cert_file: ""
  key_file: ""
  client_auth_type: ""
  client_ca_file: ""
  cipher_suites: []
  curve_preferences: []
  min_version: 0
  max_version: 0
  prefer_server_cipher_suites: false

most of that is default, but remote_read: true was explicitly set in my config file.

I'm running wireshark to see the actual request sent to prometheus from promxy, and it's a POST /api/v1/query_range instead of the expected POST /api/v1/read.

Frame 239: 313 bytes on wire (2504 bits), 313 bytes captured (2504 bits) on interface lo0, id 0
...
Hypertext Transfer Protocol
HTML Form URL Encoded: application/x-www-form-urlencoded
    Form item: "end" = "1700857449.92"
    Form item: "query" = "up"
    Form item: "start" = "1700853849.92"
    Form item: "step" = "14"

Additionally, the promxy metrics seem to support this with:

# HELP prometheus_remote_storage_read_request_duration_seconds Histogram of the latency for remote read requests.
# TYPE prometheus_remote_storage_read_request_duration_seconds histogram
prometheus_remote_storage_read_request_duration_seconds_bucket{remote_name="foo",url="http://localhost:9090/api/v1/read",le="0.005"} 0
prometheus_remote_storage_read_request_duration_seconds_bucket{remote_name="foo",url="http://localhost:9090/api/v1/read",le="0.01"} 0
prometheus_remote_storage_read_request_duration_seconds_bucket{remote_name="foo",url="http://localhost:9090/api/v1/read",le="0.025"} 0
prometheus_remote_storage_read_request_duration_seconds_bucket{remote_name="foo",url="http://localhost:9090/api/v1/read",le="0.05"} 0
prometheus_remote_storage_read_request_duration_seconds_bucket{remote_name="foo",url="http://localhost:9090/api/v1/read",le="0.1"} 0
prometheus_remote_storage_read_request_duration_seconds_bucket{remote_name="foo",url="http://localhost:9090/api/v1/read",le="0.25"} 0
prometheus_remote_storage_read_request_duration_seconds_bucket{remote_name="foo",url="http://localhost:9090/api/v1/read",le="0.5"} 0
prometheus_remote_storage_read_request_duration_seconds_bucket{remote_name="foo",url="http://localhost:9090/api/v1/read",le="1"} 0
prometheus_remote_storage_read_request_duration_seconds_bucket{remote_name="foo",url="http://localhost:9090/api/v1/read",le="2.5"} 0
prometheus_remote_storage_read_request_duration_seconds_bucket{remote_name="foo",url="http://localhost:9090/api/v1/read",le="5"} 0
prometheus_remote_storage_read_request_duration_seconds_bucket{remote_name="foo",url="http://localhost:9090/api/v1/read",le="10"} 0
prometheus_remote_storage_read_request_duration_seconds_bucket{remote_name="foo",url="http://localhost:9090/api/v1/read",le="25"} 0
prometheus_remote_storage_read_request_duration_seconds_bucket{remote_name="foo",url="http://localhost:9090/api/v1/read",le="60"} 0
prometheus_remote_storage_read_request_duration_seconds_bucket{remote_name="foo",url="http://localhost:9090/api/v1/read",le="+Inf"} 0
prometheus_remote_storage_read_request_duration_seconds_sum{remote_name="foo",url="http://localhost:9090/api/v1/read"} 0
prometheus_remote_storage_read_request_duration_seconds_count{remote_name="foo",url="http://localhost:9090/api/v1/read"} 0
# HELP prometheus_remote_storage_remote_read_queries The number of in-flight remote read queries.
# TYPE prometheus_remote_storage_remote_read_queries gauge
prometheus_remote_storage_remote_read_queries{remote_name="foo",url="http://localhost:9090/api/v1/read"} 0

...

# HELP server_group_request_duration_seconds Summary of calls to servergroup instances
# TYPE server_group_request_duration_seconds summary
server_group_request_duration_seconds_sum{call="query_range",host="localhost:9090",status="success"} 0.014273459
server_group_request_duration_seconds_count{call="query_range",host="localhost:9090",status="success"} 1

showing that it created the remote connection, but never called it and called query_range instead.

I'm building off master:

promxy, version  (branch: , revision: 300c73943dd83efb396f4aadb7f24cd4303a0155)
  build user:
  build date:
  go version:       go1.21.4
  platform:         darwin/arm64
  tags:             netgo,builtinassets

[BertHartm]

I tested v0.0.84, v0.0.70, and v0.0.60 as well, and this isn't new, so I'm not sure if it's somehow a misconfiguration / misunderstanding on my end, or a bug that's been around a while.

[jacksontj]

This is a bit of a complicated topic -- which I attempted to cover in the comments for the option https://github.com/jacksontj/promxy/blob/master/pkg/servergroup/config.go#L39-L56

Let me attempt to explain here in a bit more detail -- hopefully we can clarify the confusion here and get the docs updated.

The main feature within promxy that makes it performant is this concept of a NodeReplacer which replaces complex queries with smaller queries we can farm out to downstream prom API endpoints and re-aggregate. A simple example of this is sum(foo) -- instead of querying all data for foo then doing the sum in promxy we can simply ask each downstream sum(foo) then re-sum that in promxy.

This may seem like an aside; but this is really the key function here. So in addition to making this performant this also means that the majority of queries are not raw data fetches -- instead they are smaller promql queries. So, promxy will actually only send a raw data fetch downstream when it is absolutely necessary -- such as a bare matrix selector (e.g. foo[1h]). In the case of a raw data fetch we can chose to use remote_read; but in all other cases (where we send a promql query) we cannot.

So now you may ask "if its so niche, why does this feature even exist?" -- to which I say what a great question! There are 2 reasons for this the initial PoC implementation of promxy did all queries using raw fetches (node replacer was added soon after). During that initial PoC though I was reminded that prometheus treats NaNs a bit ... weird. Specifically it has a concept of a StaleNan which indicates that a series was not scrape-able anymore -- but unfortunately the promql interface consumes those StaleNans so using the regular query endpoint we can't determine if the series ended or if there is a gap. RemoteRead does provide the values for StaleNan instead of consuming them -- so generally speaking its a more faithful representation of the correct values -- but this edge case is very uncommon (you have to have a query that can't be nodereplaced and has some interactions between Lookback delta due to missing series). I did actually consider requiring RemoteRead (for simplicity) but some implementations don't support RemoteRead (i.e. VictoriaMetrics) and at that point the code/tests already existed -- so I have left it in.

So with that (probably way too much) context hopefully you can understand the feature a bit more and the complexity that comes with it. If you have any suggestions on how to better communicate this complexity in the comments/docs I am definitely more than happy to talk through those :)

[BertHartm]

Thanks, that does explain more, and I had been trying to work around the node replacer in this test. I think my largest mis-understanding was around when remote_read would take effect. My assumption was that it would happen in most (all) cases, but it seems a much more limited subset than I had expected, though now that I re-read the config I do see that it's explicitly stated, so I'm not sure how I'd go about wording it better.

[jacksontj]

Fair enough -- it is a somewhat nuanced thing. Well, if nothing else if someone is confused they will hopefully find this issue/question and get the answer from there!

Sounds like we're all set with this so I'll go ahead and mark this closed; if you have anything else (or have more questions on this) feel free to re-open or create a new one!