µ/log publisher for InfluxDB

[BrunoBonacci]

A µ/log publisher for InfluxDB.

Preferred Approach:

• use https://github.com/influxdata/influxdb-java

Guidelines to write a publisher:

• Please read How to write custom publishers
• Get yourself familiar with µ/log internals
• Try to use the least amount possible of dependencies, in order to reduce the clashes for the library users.
• When possible provide a docker-compose.yaml file for local testing
• For AWS API, please use Localstack for testing
• When possible, please provide one or more integration tests to verify the integration.
• Include support for user-defined transformation
• Test libraries we use in µ/log are: Midje and test.check
• Ask questions if anything is not clear.

[misterzirillo]

Hey Bruno,

A µ/log publisher for Prometheus.

Is this a typo?

Also it is easy enough to write InfluxDB data points without the java library. In my travels I have written a spec and a mapping function to produce InfluxDB line protocol:

https://gitlab.com/misterzirillo/plant-parenthood/-/blob/master/src/pp/device/metrics.cljs

Would some version of that code be a suitable implementation for this project?

[BrunoBonacci]

Hi @misterzirillo,

thanks for spotting the typo, probably a copy/paste issue.

Thanks also for pointing out the line protocol, it is easy enough that we don't need a Java wrapper for it. So definitely your approach could be suitable.

The only issue is that InfluxDB is designed to store measurements/metrics (not events), therefore it will be required to turn the events intro metrics before publishing them.

We have done something similar for Prometheus:

https://github.com/BrunoBonacci/mulog/blob/master/mulog-prometheus/src/com/brunobonacci/mulog/publishers/prometheus/metrics.clj

However, the InfluxDB model is closer to μ/log model, so it should be easier. Probably, only we have to do is separate numerical pairs from string pairs and turn the numerical into fields and the string pairs into tags.

For example an event such as:

{:mulog/event-name :db/store-records
 :mulog/timestamp 1587504242983,
 :mulog/trace-id #mulog/flake "4VTF9QBbnef57vxVy-b4uKzh7dG7r7y4",
 :mulog/root-trace #mulog/flake "4VTF9QBbnef57vxVy-b4uKzh7dG7r7y4",
 :mulog/duration 254402837,
 :mulog/namespace "your-ns",
 :mulog/outcome :ok,
 :app-name "mulog-demo",
 :env "local",
 :version "0.1.0"
 :num-items 123}

could be stored as follow:

:mulog/event-name -> measurementName
:mulog/timestamp  -> timestamp
:mulog/duration   -> field
:num-items 123    -> field
:mulog/namespace  -> label
:mulog/outcome    -> label
:app-name         -> label
:env              -> label
:version          -> label

I'm not sure whether there is any value on storing the trace-id into InfluxDB (what's your thoughts on this?) but it could be added as an extra label.

[misterzirillo]

Probably, only we have to do is separate numerical pairs from string pairs and turn the numerical into fields and the string pairs into tags.

This is my intuition as well. I think with that basic rule we a pretty solid out-of-the-box behavior for the InfluxDB publisher.

I'm not sure whether there is any value on storing the trace-id into InfluxDB (what's your thoughts on this?) but it could be added as an extra label.

It depends on how strict μ/log is about recording the whole event. There are performance reasons that an InfuxDB user might want to store a subset of the event data or change which data are stored as tags/fields. How much control do you think the publisher should allow the user over the transformation?

[BrunoBonacci]

It depends on how strict μ/log is about recording the whole event. There are performance reasons that an InfuxDB user might want to store a subset of the event data or change which data are stored as tags/fields. How much control do you think the publisher should allow the user over the transformation?

On second thought, the trace-id could be useful to avoid duplicates in case of publisher failures. In regards to the control, the users have the ability to manipulate events at will before the publishing, so I would say that the users have full control on the shape of the data published.

[BrunoBonacci]

This issue seems stale and not sure anyone requires InfluxDB. So closing issue.