Support InfluxDB line protocol over UDP

[wei-hai]

it would be convenient to export springboot production metrics inside endpoint /metrics to influxdb via line protocol, in the properties file it could support the following variables:

influxdb.host=localhost
influxdb.port=8086
influxdb.protocol=tcp or udp
influxdb.database=springboot
influxdb.measurement=counter
influxdb.interval=60s

influxdb line protocol: https://docs.influxdata.com/influxdb/v1.4/write_protocols/line_protocol_tutorial/

[jkschneider]

@wei-hai micrometer-registry-influx currently writes metrics to InfluxDB via the line protocol. You can alternatively get metrics to Influx via Telegraf StatsD with micrometer-registry-statsd.

However, our current implementation only supports shipping line protocol over HTTP as documented at the bottom of that page you linked to.

It's never been clear to me that you could ship line protocol over UDP, but can't say I've tried it. It's worth a shot.

[wei-hai]

@jkschneider Telegraf has plugin socket_listener which can handle UDP packet and forward it to influxdb via HTTP. In our use case we send line protocol metric via UDP packet and let Telegraf handle it.

https://github.com/influxdata/telegraf/tree/master/plugins/inputs/socket_listener

[jkschneider]

@wei-hai Ah, thanks. That makes it a lot clearer! We should support this.

[chenatu]

Is UDP supported yet? In which version?

[shakuzen]

@chenatu this issue is assigned to milestone 1.x which means we would like to get it in some release in major version 1 but it hasn't been assigned to any specific release yet. If we had a complete pull request for it, it may help get it into a release sooner.

[shakuzen]

Looking ahead to InfluxDB 2, I found this in their README (emphasis on UDP is mine):

What is NOT planned?

• Direct support by InfluxDB for CollectD, StatsD, Graphite, or UDP. ACTION REQUIRED: Leverage Telegraf 1.9+ along with the InfluxDB v2.0 output plugin to translate these protocols/formats.

[cmallwitz]

My primary motivation to take a look at UDP was the fact that the current HTTP InfluxDB micrometer backend is allocating too much (more than zero) memory on the heap when sending data - so this is a bit of a bummer...

[shakuzen]

My primary motivation to take a look at UDP was the fact that the current HTTP InfluxDB micrometer backend is allocating too much (more than zero) memory on the heap when sending data

I'm not sure sending the same thing via UDP is going to be significantly different heap usage (of course what is significant is relative). We should probably focus on identifying efficiency improvements in sending data via HTTP rather than supporting UDP for that reason alone.

So I'm trying to understand the motivation of users requesting support for shipping Influx line protocol over UDP as opposed to HTTP. If it is to send it to Telegraf, Telegraf can take Influx line protocol over HTTP as well, so that should already be supported.

[cmallwitz]

Well, I was hoping the UDP implementation would be any better than the TCP one...

Background: we had a simple use case - start up a JVM, start GC data collection with metrics being send to Influx once a second, and sleep forever. What you will find is that micrometer + influx client will allocate memory on heap constantly (but not leaking) triggering GCs. In fact for us it was allocating more memory than our application (which wasn't just sleeping) - creating more noise than we wanted to measure..

As a result, we rewrote some of the GC metrics handling code and implemented our own HTTP Influx injector (bypassing all JDK HTTP client code).

[jonatan-ivanov]

Well, I was hoping the UDP implementation would be any better than the TCP one...

It depends. :) If you are ok with losing data, UDP is more lightweight but you might lose some of the data you recorded. E.g.: I worked with a team years ago who implemented this (Influx with UDP), they lost 6-8% of their data during publishing.

I don't think TCP-UDP will mean a big difference in GC cycles either but you can easily test this with Telegraf as Tommy mentioned above. Please let us know the difference you see if you try this out. If it is not bit I would recommend playing with your heap and GC settings.

[shakuzen]

@cmallwitz Thank you for the context. That's helpful.

As a result, we rewrote some of the GC metrics handling code and implemented our own HTTP Influx injector (bypassing all JDK HTTP client code).

Do you mean you wrote code to replace Micrometer's JvmGcMetrics implementation and a custom HttpSender implementation that uses UDP with the Micrometer InfluxMeterRegistry? Or do you mean you wrote separate code to use instead of Micrometer? Improving the out-of-the-box performance of Micrometer is a persistent goal of ours, so if you have any specific insight on that, please feel free to share that with us (perhaps in a separate issue).

[cmallwitz]

We slightly modified the Micrometer's JvmGcMetrics and JvmMemory to support a "multi value" measurement (single record send to influx for GC and Memory metrics. We created our own simplistic HTTP sender because all the HTTP clients out there allocate memory on every server interaction: a buffer here, a URL object there, ... We created our own Registry to serialise Influx Line protocol without any allocations.

I think the only thing left from Micrometer are various Meter implementations e.g. for distributions

In general, people seem to be too fond of the Java Stream libraries but they are rubbish when you are in a low latency environment and don't want to allocate any memory at all (beyond whatever is allocated at start-up or warm-up phase)

As I said before, for us the goal was zero heap allocations and not so much performance itself. Example: create a process that enables the three Jvm Metrics, injects data from them into Influx and sleeps otherwise. We wanted that to run forever without it ever allocating a single byte of memory and eventually triggering a minor GC just because of Micrometer code.

[marcingrzejszczak]

Is there anything we should do in this issue or can we consider this as done?

[cmallwitz]

we moved on - this could be closed.

[marcingrzejszczak]

Thank you @cmallwitz !