search

grafana / carbon-relay-ng

Fast carbon relay+aggregator with admin interfaces for making changes online - production ready
Other
468 stars 151 forks source link

Missing error notification #110

Open kruxErinWillingham opened 8 years ago

kruxErinWillingham commented 8 years ago

While testing carbon-relay-ng i ran into a problem where I wasn't receiving any error notifications nor any notice of dropped metrics. In my environment i was sending 1.5 million metrics per minute through carbon-relay-ng while performing around 15 complex aggregations. Example aggregation: 'addAgg sum ^([^.]+)\.timers\.(.*pdx.*)\.(.+)\.([^.]+)$ $1.timers.$2._totals.pdx.$4.sum 10 20',

before bumping this line up to 1000000, https://github.com/graphite-ng/carbon-relay-ng/blob/master/conn.go#L15 almost no aggregations were happening, when they were, only partial data was actually getting aggregated. After the change I also saw no errors or dropped metrics, however my data was being received and aggregated as expected. The app has been running stable for a few days with the new value.

Is there any way to have the app alert or report when it gets into such a state? Is there any way to move the conn_in_buffer into a config file variable?

Dieterbe commented 8 years ago

Is there any way to move the conn_in_buffer into a config file variable?

yes, change the code (and submit a PR ) :-)

other than that, sorry no time to help with this.