The SNMPCollector is currently focused and oriented to send metrics to InfluxDB, using internally specific functions of InfluxDB client to manage and send those metrics.
This PR tries to break with this dependency on metric internal management adopting the current telegraf metric definitions.
With this abstracion, a new engine is created in order to send those generic metrics to specific backends in a centralized way.
The new metric sender engine also adopts the current telegraf buffer. The metrics are now stored on the buffer instead of being written each time that a measurement produces a small set of them.
The proposed engine is changed and it is based on the current telegraf way: a ticker is defined and the metrics are flushed on several batches, reducing drastically the number of request that are being done and with the change to control the current buffer size and make operations on top the output.
Since all the metrics are stored on a top-level buffer, the send process is being done by the backend implementation. This PR redefines the current influxdb to be a backend that only performs the connection, write close.
The internal selfmon metrics -measurement, runtime, outdb stats- are refactored to follow the same generic metric definition, using, as always, the 'default' backend defined, now, as an Output.
With this behaviour, a new Kafka Backend has been created to be able to write metrics to several brokers using JSON output format. The code is copied/modified from the current telegraf.output Kafka
The SNMPCollector OutDB now is related with an Output instead of InfluxDB Server. All the management of Outputs and new Kafka Server can be done on the UI and via API. The output relation with the backend is, right now 1:1, and a Device can have only 1 output attached
BREAKING CHANGE: the metrics are now stored on internal buffer and written to the final backend based on time. The write process is being done each FlushInterval and splitted into several request based on the buffer length and the MetricsBatchSize.
BREAKING CHANGE: the InfluxDB UI component is being moved to a new section and the relation with the SNMPDevice is break and changed to an Output. The field BufferSize is being removed from API request and new DB initializtion, but mantained on old configurations to perform a migration process
The SNMPCollector is currently focused and oriented to send metrics to InfluxDB, using internally specific functions of InfluxDB client to manage and send those metrics.
This PR tries to break with this dependency on metric internal management adopting the current telegraf metric definitions. With this abstracion, a new engine is created in order to send those generic metrics to specific backends in a centralized way.
The new metric sender engine also adopts the current telegraf buffer. The metrics are now stored on the buffer instead of being written each time that a measurement produces a small set of them.
The proposed engine is changed and it is based on the current telegraf way: a ticker is defined and the metrics are flushed on several batches, reducing drastically the number of request that are being done and with the change to control the current buffer size and make operations on top the output.
Since all the metrics are stored on a top-level buffer, the send process is being done by the backend implementation. This PR redefines the current influxdb to be a backend that only performs the connection, write close.
The internal selfmon metrics -measurement, runtime, outdb stats- are refactored to follow the same generic metric definition, using, as always, the 'default' backend defined, now, as an Output.
With this behaviour, a new Kafka Backend has been created to be able to write metrics to several brokers using JSON output format. The code is copied/modified from the current telegraf.output Kafka
The SNMPCollector OutDB now is related with an Output instead of InfluxDB Server. All the management of Outputs and new Kafka Server can be done on the UI and via API. The output relation with the backend is, right now 1:1, and a Device can have only 1 output attached
BREAKING CHANGE: the metrics are now stored on internal buffer and written to the final backend based on time. The write process is being done each FlushInterval and splitted into several request based on the buffer length and the MetricsBatchSize.
BREAKING CHANGE: the InfluxDB UI component is being moved to a new section and the relation with the SNMPDevice is break and changed to an Output. The field BufferSize is being removed from API request and new DB initializtion, but mantained on old configurations to perform a migration process
fix #343