search

influxdata / influxdb

Scalable datastore for metrics, events, and real-time analytics
https://influxdata.com
Apache License 2.0
28.8k stars 3.55k forks source link

Handle ReadHints pushed down by prometheus remote read #17641

Open foobar opened 4 years ago

foobar commented 4 years ago

Proposal: Make use of pushed-down ReadHints to speed up prometheus remote read query

Current behavior: InfluxDB 1.x has support for prometheus remote read/write, but historically to avoid dependency it copied protobuf definition(from https://github.com/influxdata/influxdb/pull/8784). The remote-read pb message contains label matcher and time range and that is used to filter influxdb points. But the current prometheus version has added more fields to the pb message, including ReadHints. The read hints may be used by remote storage to do more filtering and return less data. Also the new-added streamed response type can mitigate memory usage-for now it needs to hold the whole result set in memory.

Desired behavior: InfluxDB may make use of the ReadHint and handles prometheus query locally.

Alternatives considered: n/a

Use case:

For users who use InfluxDB as prometheus remote storage, the pushed-down hints will avoid large data transformation.

kannan-rafay commented 2 years ago

Hi , I believe ReadHints wasn't implemented by influx yet , correct ? can you confirm please , as is detailed in the issue will the prometheus pull all the dataset as part of response from influx after applying matching labels (ignoring group by clauses that can be part of ReadHints ) for further processing , please clarify . regards Kannan