Adding tags to the InfluxDB data model

[amotl]

@TheOneWhoKnocks96 asked over at #13:

I already have data in my InfluxDB, but without tags. Where in the Kotori's code, could i add tags like longitude and latitude?

Thanks for keeping responding. Best Regards.

[RuiPinto96]

Hello again!!

Is it possible to add tags through here? : https://github.com/daq-tools/kotori/blob/master/kotori/io/protocol/influx.py#L45

Best Regards.

[amotl]

Dear @TheOneWhoKnocks96,

after starting this yesterday, it seems I got distracted somehow. Sorry.

The link you are pointing out looks like it is coming from the InfluxDB query machinery, i.e. suitable for requesting data from InfluxDB -- it's the read path again!

expression = Query('*').from_(measurement).where(time__gte=time_begin, time__lte=time_end, **tags)

With kind regards, Andreas.

[RuiPinto96]

The link you are pointing out looks like it is coming from the InfluxDB query machinery, i.e. suitable for requesting data from InfluxDB -- it's the read path again!

Ah okay, my bad.

after starting this yesterday, it seems I got distracted somehow. Sorry.

This means that the feature of adding tags to the InfluxDB is not implemented yet?

Best Regards.

[amotl]

Dear @TheOneWhoKnocks96,

Adding tags to the InfluxDB database

This means that the feature of adding tags to the InfluxDB database is not implemented yet?

It well is - kind of. You might want to amend some code at kotori.daq.storage.influx.format_chunk according to your needs. The whole file is about the data acquisition path where you will get an insight how data is actually written into the InfluxDB database.

Inside this method, you will find references to chunk["tags"], which is a dictionary where you can stuff tags into.

Example

There's already a section for transforming the geohash telemetry field to an InfluxDB tag:

# Extract geohash from data. Finally, thanks Rich!
# TODO: Also precompute geohash with 3-4 different zoomlevels and add them as tags
if "geohash" in data:
    chunk["tags"]["geohash"] = data["geohash"]
    del data['geohash']

which we are using for ingesting data from luftdaten.info through https://github.com/hiveeyes/luftdatenpumpe.

Hope this helps.

With kind regards, Andreas.

[RuiPinto96]

So i can just add my tags here? -> https://github.com/daq-tools/kotori/blob/0.22.7/kotori/daq/storage/influx.py#L145

something like this?:

chunk = {
            "measurement": meta['measurement'],
            "tags": {
                "host": "server01",
                "region": "europe",
             },
        }

"""
        if "gateway" in meta:
            chunk["tags"]["gateway"] = meta["gateway"]
        if "node" in meta:
            chunk["tags"]["node"]    = meta["node"]
"""

And then this after?:

if "gateway" in meta:
            chunk["tags"]["gateway"] = data["gateway"]
            del data['gateway']

I don't know if this is correct, hope you could help. Thanks for the help so far anyway @amotl

Best Regards

[amotl]

Dear @Ozymandias96,

correct, you can fill key/value pairs into chunk["tags"] anywhere within this method. Whether you statically insert them right there using a hacked version of influx.py or if you want to derive the tag fields from the data ingress payload itself like done with the fields geohash, location, location_id, location_name, sensor_id and sensor_type is really up to you.

We could also think about introducing a mechanism to declare a set of static tags per channel by defining them inside the Kotori configuration file.

Please let us know if you see other mechanisms for injecting tags into the record Kotori is about to insert into InfluxDB here.

With kind regards, Andreas.

[RuiPinto96]

I can't find and edit the influx.py script in my machine. How do i derive the tag fields from the data ingress payload?

Thanks for the help Best Regards

[amotl]

Dear @Ozymandias96,

sorry for the delay.

Installation location

I can't find and edit the influx.py script in my machine.

When installing Kotori from the Debian package, it will unfold at /opt/kotori/lib/python2.7/site-packages/kotori.

Derive tag fields

How do i derive the tag fields from the data ingress payload?

And then this after?:

if "gateway" in meta:
chunk["tags"]["gateway"] = data["gateway"]
del data['gateway']

Right, I've outlined an example within https://github.com/daq-tools/kotori/issues/14#issuecomment-521815250.

Example

There's already a section for transforming the geohash telemetry field to an InfluxDB tag:

# Extract geohash from data. Finally, thanks Rich!
# TODO: Also precompute geohash with 3-4 different zoomlevels and add them as tags
if "geohash" in data:
    chunk["tags"]["geohash"] = data["geohash"]
    del data['geohash']

With kind regards, Andreas.