How to use with Graphite storage schemas?

[Knud3]

It seems that Mimir through Graphite write proxy saves datapoint every one second no matter what.

I know I can set minimal interval for query options in Grafana, but that is suboptimal and it wont work if panel have metrics with different schemas.

I can see in nginx logs that carbon-relay-ng is trying to POST schemas every 6 hours, but what to use as endpoint? If any...

I am using grafana/carbon-relay-ng:1.4.4 deployment in k8s with Graphite write proxy (build with latest a2c232a commit) and grafana/mimir:2.10.0.

carbon-relay-ng.ini

[[route]]
key = 'carbon-default'
type = 'sendAllMatch'
schemasFile = '/etc/carbon/storage-schemas.conf'
aggregationFile = '/etc/carbon/storage-aggregation.conf'
spool = true
spoolbuf = 200000
spoolmaxbytesperfile = '5000MiB'
destinations = [
  'go-carbon-tcp:2003'
]

[[route]]
key = 'grafanaNet'
type = 'grafanaNet'
addr = 'http://graphite-proxy-writes:8000/graphite/metrics'
apikey = 'anonymous'
schemasFile = '/etc/carbon/storage-schemas.conf'
aggregationFile = '/etc/carbon/storage-aggregation.conf'
concurrency = 1000
bufSize = 100000000

[instrumentation]
graphite_addr = "localhost:2003"
graphite_interval = 10000

storage-schemas.conf

[carbon-relay-ng]
pattern = ^carbon-relay-ng\.*
retentions = 10s:14d,1h:10y

Mimir proxy is started with: /usr/bin/graphite-proxy-writes -auth.enable=false -write-endpoint http://distributor:8080/api/v1/push

Example query

Carbon: alias(averageSeries(carbon-relay-ng.stats.*.memory.gc.last_duration.gauge64), 'Duration', false)

Mimir: avg by(__n006__) (graphite_untagged{__n000__="carbon-relay-ng", __n001__="stats", __n003__="memory", __n004__="gc", __n005__="last_duration", __n006__="gauge64"})

[Knud3]

I have to test this further, but this might actually be working...

I found querying with last_over_time(foo[$__interval]) seems to do its trick, but now I have another problem to solve https://github.com/grafana/mimir/discussions/6483

[npazosmendez]

Hi @Knud3 ,

It seems that Mimir through Graphite write proxy saves datapoint every one second no matter what.

The Graphite write proxy writes to mimir the exact timestamps it receives (from carbon-relay-ng in your case), and mimir stores only the exact timestamps it receives from the proxy. The thing is in PromQL, every step will show the most recent sample as long as it is not older than 5 minutes, so not all points you are seeing correspond to a sample stored. See docs: https://prometheus.io/docs/prometheus/latest/querying/basics/#staleness

On the question of storage/aggregation schemas: unfortunately, as you noticed, they are not supported in the Graphite write proxy. It is designed to store the data into Mimir as it comes without aggregating, and then the desired aggregations are to be handled during query time.

[Knud3]

I close this for now. Discussion about data points are continued here: https://github.com/grafana/mimir/discussions/6674