search

influxdata / telegraf

Agent for collecting, processing, aggregating, and writing metrics, logs, and other arbitrary data.
https://influxdata.com/telegraf
MIT License
14.65k stars 5.59k forks source link

field/tag as “variables” for the “files” parameter of File Output Plugin #16045

Open dk9uv opened 2 weeks ago

dk9uv commented 2 weeks ago

Use Case

I would like to store metrics in multiple files. E.g. LoRaWAN devices have unique device numbers. Metrics of each device shall be stored in a file with device number as file name. There is no list of device numbers availabe. File names shall be generated automatically based on a tag or field.

Expected behavior

In files = ["stdout","conf/metrics.out"] it shall be possible to use variables (tag or field values).

Actual behavior

Only constant values are allowed in files=[ ]

Additional info

No response

srebhan commented 1 week ago

@dk9uv the remotefile output plugin can do this and it has support for the local filesystem. Is this what you are looking for?

dk9uv commented 1 week ago

Hi,

thanks very much for the help. I am still struggling with configuration.

In Telegraf metrics look like this (Influxx format): eui-2cf7f12032308837,device_id=eui-2cf7f12032308837,host=DESKTOP-RLVP129,topic=v3/test-e5@ttn/devices/eui-2cf7f12032308837/up temperature=21.2 1730026545173756400

Measurement: eui-2cf7f12032308837 Tag: device_id=eui-2cf7f12032308837 Field: temperature=21.2

Background information: Later I will have a single measurement "Weatherstation" and many decives with different device_id. I would like to have separate files for each device_id.

When using:

[[outputs.remotefile]] remote = "local:" files = ['{{.Name}}']

the file eui-2cf7f12032308837 is automatically generated

When using

[[outputs.remotefile]] remote = "local:" files = ['{{.Tag "device_id"}}']

Telegraf throws an error:

2024-10-27T11:16:16Z E! [outputs.remotefile] Cannot create filename "{{.Tag \"device_id\"}}" for metric &{eui-2cf7f12032308837 map[device_id:eui-2cf7f12032308837 host:DESKTOP-RLVP129 topic:v3/test-e5@ttn/devices/eui-2cf7f12032308837/up] map[temperature:21.5] 1730027766875933600 0xc0000d3080}: template: {{.Tag "device_id"}}:1:2: executing "{{.Tag \"device_id\"}}" at <.Tag>: can't evaluate field Tag in type *metric.trackingMetric
eui-2cf7f12043308981,device_id=eui-2cf7f12043308981,host=DESKTOP-RLVP129,topic=v3/test-e5@ttn/devices/eui-2cf7f12043308981/up temperature=15.21 1730027769834887500

Any ideas what could be wrong?

Best regards

Chris