search

jfindley / newrelic_exporter

NewRelic exporter for prometheus
BSD 2-Clause "Simplified" License
29 stars 20 forks source link

Issues with applications with large numbers of metrics #7

Closed stuart-warren closed 8 years ago

stuart-warren commented 8 years ago

We seem to have at least one application that has over 1000 pages of metric names :( https://api.newrelic.com/v2/applications/%appid%/metrics.json?page=1240

To make this worse, it takes about 6 seconds to fetch each page of metric names... about 2 hours just to get the names, let alone the metrics :'(

Took quite a while to work out where this was getting 'stuck', but everything blocks until these requests finish.

Obviously ideally we wouldn't have so many, but is it possible/sensible to asynchronously update the Exporter.metrics map? Maybe a pool of goroutines fetching an application each.

stuart-warren commented 8 years ago

Nevermind, obviously thinking about it a request should probably give the 'current' values rather than stale cached ones.

Maybe a timeout per application would make more sense.

stuart-warren commented 8 years ago

Went with an API client request timeout in the end...