Configuration question

[lukipro]

Hi guys,

I am having troubles configuring the remote storage adapter. Or perhaps I am just missing something obvious.

To give an example I picked a random metric: alertmanager_http_response_size_bytes_bucket If I execute this query in Prometheus I get 96 results:

However in Graphite there are only two results being stored:

What may be the reason for this? I do not see any errors in the remote-storage-adaptor logs nor does Prometheus report any issues with remote writes.

Appreciate your help!

[geobeau]

Hello, sorry for the delay.

I don't see any obvious issues. It could come from the parsing of the le label or some kind of collision of multiple Prometheus metrics resulting in the same Graphite path.

On the admin UI of your Graphite Remote Adapter you should see a simulation tab: you can write the prometheus metrics that you have and then check the Graphite path to see if it's what you expect.

If it doesn't help you solve this issue, share here your current configuration and the set of prometheus metrics so we can try to reproduce the issue

[melchiormoulin]

Hello can you provide your prometheus and graphite remote adapter configuration ?

[lukipro]

Hello,

Thanks for your feedback. Turned out that I just had to wait for quite a while in order for those metrics to appear in Graphite. My Prometheus scrapes about 200 000 metrics. Storage schema configuration is following: 10s:6h,1m:6d,10m:1800d. Every metric requires about 3 MB, so Graphite had to create over 600 GB of data. I am not sure if it is correct or not but it took more than two days.

On this graph you may see that until about 18:00 on 2/17 metrics were created all the time. Afterwards it was way less frequent.

Here are my configs:

• remote adapter

  web:
  listen_address: "0.0.0.0:9201"
  telemetry_path: "/metrics"
  write:
  timeout: 5m
  read:
  timeout: 5m
  delay: 1h
  ignore_error: true
  graphite:
  default_prefix: prometheus.
  enable_tags: false
  openmetrics: false
  read:
  url: http://graphite.graphite.svc:8080
  write:
  carbon_address: graphite.graphite.svc:2003

• Prometheus

  
  remote_write:

• url: http://remote-storage-adapter-app.default:9201/write remote_timeout: 30s queue_config: capacity: 500 max_shards: 2000 min_shards: 1 max_samples_per_send: 100 batch_send_deadline: 5s min_backoff: 30ms max_backoff: 100m

I will appreciate your comments on this configuration.

[geobeau]

The remote adapter config looks similar to what we currently use. For Prometheus, it depends on your workload and if Graphite is able to handle the throughput.

The graphite metric creation rate seems quite low but I'm not used to vanilla, we use a customised version at Criteo.

[hdost]

Seems to have answered the question.