search

influxdata / telegraf

Agent for collecting, processing, aggregating, and writing metrics, logs, and other arbitrary data.
https://influxdata.com/telegraf
MIT License
14.5k stars 5.55k forks source link

inputs.modbus assign tags as a key/value array to each fields #12141

Closed lfarkas closed 1 year ago

lfarkas commented 1 year ago

Use Case

As described in this issue #12091 it'd be useful for many users if in the modbus input plugin we can assign tags for each fields. just to copy the example from that issue:

 [[inputs.modbus.request]]
   slave_id = 9
   byte_order = "ABCD"
   register = "holding"
   fields = [
     { address=32016, measurement="Voltage", tags=[resource="PV1"], name="ActivePower", type="INT16", scale=0.1, output="FLOAT64" },
     { address=32018, measurement="Voltage", tags=[resource="PV2"], name="ActivePower", type="INT16", scale=0.1, output="FLOAT64" },
     { address=32020, measurement="Voltage", tags=[resource="PV3"], name="ActivePower", type="INT16", scale=0.1, output="FLOAT64" },
     { address=32022, measurement="Voltage", tags=[resource="PV1"], name="RectivePower", type="INT16", scale=0.1, output="FLOAT64" },
     { address=32024, measurement="Voltage", tags=[resource="PV2"], name="RectivePower", type="INT16", scale=0.1, output="FLOAT64" },
     { address=32026, measurement="Voltage", tags=[resource="PV3"], name="RectivePower", type="INT16", scale=0.1, output="FLOAT64" },
   ]

Expected behavior

in that case we can separately tag all fields in stead of put the into different measurement (since currently we can put them into different measurement but can't add different tags.)

Actual behavior

In theory you could do this already. But I'm not quite sure if telegraf does here separate calls are the same as above. since if it's create 3 different read request then it's a huge performance bottleneck. modbus is a very slow communication protocol. we need optimize as much as possible to be able to get out the desired performance. if it;s create 3 (or in a real more complicated example a few dozen) of requests then it's not a possible solution.


[[inputs.modbus.request]]
   slave_id = 9
   byte_order = "ABCD"
   register = "holding"
   fields = [
     { address=32016, measurement="Voltage", name="ActivePower", type="INT16", scale=0.1, output="FLOAT64" },
     { address=32022, measurement="Voltage", name="RectivePower", type="INT16", scale=0.1, output="FLOAT64" },
   ]
   [inputs.modbus.request.tags]
     resource = "PV1"

[[inputs.modbus.request]]
   slave_id = 9
   byte_order = "ABCD"
   register = "holding"
   fields = [
     { address=32018, measurement="Voltage", name="ActivePower", type="INT16", scale=0.1, output="FLOAT64" },
     { address=32024, measurement="Voltage", name="RectivePower", type="INT16", scale=0.1, output="FLOAT64" },
   ]
   [inputs.modbus.request.tags]
     resource = "PV2"

[[inputs.modbus.request]]
   slave_id = 9
   byte_order = "ABCD"
   register = "holding"
   fields = [
     { address=32020, measurement="Voltage", name="ActivePower", type="INT16", scale=0.1, output="FLOAT64" },
     { address=32026, measurement="Voltage", name="RectivePower", type="INT16", scale=0.1, output="FLOAT64" },
   ]
   [inputs.modbus.request.tags]
     resource = "PV3"

Additional info

I don't know whether it's a big request since i don't know the codebase. but for someone who knows it probably not a big deal:-) @srebhan ?

reimda commented 1 year ago

Hi @lfarkas, as a workaround since the modbus plugin doesn't have a way to add tags per modbus field, you could use a processor to add the tags afterward. The starlark processor lets you write a script to do specific tasks. You can use it to check the address and add a tag.

There's an example script that does a similar job. It checks the value of a field and clears fields: https://github.com/influxdata/telegraf/blob/master/plugins/processors/starlark/testdata/value_filter.star. In your case you would want to add a tag instead of clearing fields.

lfarkas commented 1 year ago

unfortunately it's not possible in this case. as you can see different memory address represent different values. just think about the above case is the 3 phase electricity measurement. for each phase you'd like to add a tag, but if you don't distinguish at read time later you can't add that tag. and another problems is that we can't read all this value in different round. so we can have to add it ad read stage or put them into different measurement.

gcsontos commented 1 year ago

Hi @lfarkas, as a workaround since the modbus plugin doesn't have a way to add tags per modbus field, you could use a processor to add the tags afterward. The starlark processor lets you write a script to do specific tasks. You can use it to check the address and add a tag.

There's an example script that does a similar job. It checks the value of a field and clears fields: https://github.com/influxdata/telegraf/blob/master/plugins/processors/starlark/testdata/value_filter.star. In your case you would want to add a tag instead of clearing fields.

Hi,

I have been searching for the same solution as @lfarkas. I would like to put phase tags (like L1, L2, L3) per address (only if needed), and I don't want to separete the requests because of that. I am not familiar with the starlark processor. Are you sure we can use it for that? There isn't any "address" field in the output of the modbus plugin.

Thanks

lfarkas commented 1 year ago

@sspaink do you think it can be implemented in the near future or we've to follow @SokoFromNZ's hack and put it into different measurements?

SokoFromNZ commented 1 year ago

@lfarkas thanks for this FR. Its exactly how I expected it to find in telegraf already :) Hope this gets implemented soon

srebhan commented 1 year ago

@lfarkas and @SokoFromNZ I think we might hit TOML limitations with this as you cannot define an array like this. Not even sure if you can define a map inside this structure. Currently, the solution would be to either tag the phase in the measurement, i.e. have one metric per phase or to add the phase to the field name and do the conversion to tag in a processor.

The reason why I'm a bit hesitant to add this feature is that you create a new series when tagging single fields and this can lead to high cardinality on the output side (imagine lot of fields, each one being its own series).

SokoFromNZ commented 1 year ago

@srebhan Thanks for your input here.

If TOML cannot do such things we are out of luck anyhow. But giving multiple tags to a single field/address is especially in my case (reading data of a photovoltaic power inverter) basically a must.

Ideally I would like to do something like this so I can easily do separate queries later on the input/output of the inverter (_measurement AC or DC), the type (_field Voltage, Current, Power), or the phase (1, 2 or 3):

[[inputs.modbus.request]]
   slave_id = 9
   byte_order = "ABCD"
   register = "holding"
   fields = [
       { address=..., measurement="DC", tags=[phase="1"], name="Voltage", ... },
       { address=..., measurement="DC", tags=[phase="1"], name="Current", ... },
       { address=..., measurement="DC", tags=[phase="1"], name="Power", ... },
       { address=..., measurement="DC", tags=[phase="2"], name="Voltage", ... },
       { address=..., measurement="DC", tags=[phase="2"], name="Current", ... },
       { address=..., measurement="DC", tags=[phase="2"], name="Power", ... },
       { address=..., measurement="DC", tags=[phase="3"], name="Voltage", ... },
       { address=..., measurement="DC", tags=[phase="3"], name="Current", ... },
       { address=..., measurement="DC", tags=[phase="3"], name="Power", ... },
       { address=..., measurement="AC", tags=[phase="1"], name="Voltage", ... },
       { address=..., measurement="AC", tags=[phase="1"], name="Current", ... },
       { address=..., measurement="AC", tags=[phase="1"], name="Power", ... },
       { address=..., measurement="AC", tags=[phase="2"], name="Voltage", ... },
       { address=..., measurement="AC", tags=[phase="2"], name="Current", ... },
       { address=..., measurement="AC", tags=[phase="2"], name="Power", ... },
       { address=..., measurement="AC", tags=[phase="3"], name="Voltage", ... },
       { address=..., measurement="AC", tags=[phase="3"], name="Current", ... },
       { address=..., measurement="AC", tags=[phase="3"], name="Power", ... },
   ]

So this would be a big improvement compared to the Actual-Behaviour in the opening post (not only for readability).

By "high cardinality on the output side" you mean inside of telegraf or in the influx-bucket later? Using a processor after the collection (as you suggested as a current solution) would create the same cardinality, wouldn't it?

PS: Would the type of definition as shown in Actual-Behaviour in the opening post lead to a single request to the modbus device as same as this nicely readable solution would do?

lfarkas commented 1 year ago

Let we see this current example. In case of electricity there are dozen things which are phase specific and another dozen which are not. But it'd be logical to keep all of them in one measurement. Here is a concrete example how do you keep it? In a dozen of measurements? And then what's happening when you like to aggregate eg. 15 minutes min max mean value to the influx cloud? You create 3 times dozen of measurements? Imho this case tags would be much more easier for the user point of view but I don't know about cardinality problems... 

input_registers = [ 
{ name = "L1 Phase Voltage (V)", byte_order = "ABCD", data_type = "UINT32", scale=0.01, address = [1,2]}, 
{ name = "L2 Phase Voltage (V)", byte_order = "ABCD", data_type = "UINT32", scale=0.01, address = [3,4]}, 
{ name = "L3 Phase Voltage (V)", byte_order = "ABCD", data_type = "UINT32", scale=0.01, address = [5,6]}, 
{ name = "L1 Current (A)", byte_order = "ABCD", data_type = "UINT32", scale=0.0001, address = [7,8]}, 
{ name = "L2 Current (A)", byte_order = "ABCD", data_type = "UINT32", scale=0.0001, address = [9,10]}, 
{ name = "L3 Current (A)", byte_order = "ABCD", data_type = "UINT32", scale=0.0001, address = [11,12]}, 
{ name = "L1-L2 Voltage (V)", byte_order = "ABCD", data_type = "UINT32", scale=0.01, address = [13,14]}, 
{ name = "L2-L3 Voltage (V)", byte_order = "ABCD", data_type = "UINT32", scale=0.01, address = [15,16]}, 
{ name = "L3-L1 Voltage (V)", byte_order = "ABCD", data_type = "UINT32", scale=0.01, address = [17,18]}, 
{ name = "L1 Active Power (W)", byte_order = "ABCD", data_type = "INT32", scale=0.01, address = [19,20]}, 
{ name = "L2 Active Power (W)", byte_order = "ABCD", data_type = "INT32", scale=0.01, address = [21,22]}, 
{ name = "L3 Active Power (W)", byte_order = "ABCD", data_type = "INT32", scale=0.01, address = [23,24]}, 
{ name = "L1 Reactive Power (Var)", byte_order = "ABCD", data_type = "INT32", scale=0.01, address = [25,26]}, 
{ name = "L2 Reactive Power (Var)", byte_order = "ABCD", data_type = "INT32", scale=0.01, address = [27,28]}, 
{ name = "L3 Reactive Power (Var)", byte_order = "ABCD", data_type = "INT32", scale=0.01, address = [29,30]}, 
{ name = "L1 Apparent Power (VA)", byte_order = "ABCD", data_type = "UINT32", scale=0.01, address = [31,32]}, 
{ name = "L2 Apparent Power (VA)", byte_order = "ABCD", data_type = "UINT32", scale=0.01, address = [33,34]}, 
{ name = "L3 Apparent Power (VA)", byte_order = "ABCD", data_type = "UINT32", scale=0.01, address = [35,36]}, 
{ name = "L1 Power Factor", byte_order = "ABCD", data_type = "INT32", scale=0.0001, address = [37,38]}, 
{ name = "L2 Power Factor", byte_order = "ABCD", data_type = "INT32", scale=0.0001, address = [39,40]}, 
{ name = "L3 Power Factor", byte_order = "ABCD", data_type = "INT32", scale=0.0001, address = [41,42]}, 
{ name = "Frequency (Hz)", byte_order = "ABCD", data_type = "UINT32", scale=0.001, address = [49,50]}, 
{ name = "Eqv. Phase Voltage (V)", byte_order = "ABCD", data_type = "UINT32", scale=0.01, address = [51,52]}, 
{ name = "Eqv. Phase-To-Phase Voltage (V)", byte_order = "ABCD", data_type = "UINT32", scale=0.01, address = [53,54]}, 
{ name = "Eqv. Current (A)", byte_order = "ABCD", data_type = "UINT32", scale=0.0001, address = [55,56]}, 
{ name = "Eqv. Active Power (W)", byte_order = "ABCD", data_type = "INT32", scale=0.01, address = [57,58]}, 
{ name = "Eqv. Reactive Power (Var)", byte_order = "ABCD", data_type = "INT32", scale=0.01, address = [59,60]}, 
{ name = "Eqv. Apparent Power (VA)", byte_order = "ABCD", data_type = "UINT32", scale=0.01, address = [61,62]}, 
{ name = "Eqv Power Factor", byte_order = "ABCD", data_type = "INT32", scale=0.0001, address = [63,64]}, 
{ name = "Phase-Phase Voltage Asymmetriy (%)", byte_order = "ABCD", data_type = "UINT32", scale=0.01, address = [65,66]}, 
{ name = "Phase-Neural Voltage Asymmetriy (%)", byte_order = "ABCD", data_type = "UINT32", scale=0.01, address = [67,68]}, 
{ name = "Current Asymmetry (%)", byte_order = "ABCD", data_type = "UINT32", scale=0.01, address = [69,70]}, 
{ name = "Neutral Current (A)", byte_order = "ABCD", data_type = "UINT32", scale=0.0001, address = [71,72]}, 
{ name = "L1 Voltage Thd (%)", byte_order = "ABCD", data_type = "UINT32", scale=0.01, address = [83,84]}, 
{ name = "L2 Voltage Thd (%)", byte_order = "ABCD", data_type = "UINT32", scale=0.01, address = [85,86]}, 
{ name = "L3 Voltage Thd (%)", byte_order = "ABCD", data_type = "UINT32", scale=0.01, address = [87,88]}, 
{ name = "L1 Current Thd (%)", byte_order = "ABCD", data_type = "UINT32", scale=0.01, address = [89,90]}, 
{ name = "L2 Current Thd (%)", byte_order = "ABCD", data_type = "UINT32", scale=0.01, address = [91,92]}, 
{ name = "L3 Current Thd (%)", byte_order = "ABCD", data_type = "UINT32", scale=0.01, address = [93,94]}, 
{ name = "L1-2 Voltage Thd (%)", byte_order = "ABCD", data_type = "UINT32", scale=0.01, address = [95,96]}, 
{ name = "L2-3 Voltage Thd (%)", byte_order = "ABCD", data_type = "UINT32", scale=0.01, address = [97,98]}, 
{ name = "L3-1 Voltage Thd (%)", byte_order = "ABCD", data_type = "UINT32", scale=0.01, address = [99,100]}, 
{ name = "KW L1-2 (W)", byte_order = "ABCD", data_type = "UINT32", scale=0.01, address = [111,112]}, 
{ name = "KW L2-3 (W)", byte_order = "ABCD", data_type = "UINT32", scale=0.01, address = [113,114]}, 
{ name = "KW L3-1 (W)", byte_order = "ABCD", data_type = "UINT32", scale=0.01, address = [115,116]},
srebhan commented 1 year ago

@lfarkas I don't see your point. Let's take a look at the resulting metrics instead of the modbus configuration. There are multiple variations of what you want, I'll show two which I used myself. The first one is to have what I call a sensor-centric view, where you put all fields in one measurement (basically your last example) and get a measurement like

device_A,some_tags=with_values V_L1=...,V_L2=...,V_L3=...,...,Q_L1=...,Q_L2=,Q_L3=...,Q_total=...,etc 1669113259000000

This has the advantage that you get all information of that device at once for a timestamp. The other view I used is what I call physical quantity centric, i.e. you get multiple measurements one for each quantity

voltage,device=A,some_tags=with_values L1=..., L2=...,L3=... 1669113259000000
reactive_power,device=A,some_tags=with_values L1=..., L2=...,L3=...,total=... 1669113259000000
...

This has the advantage that each measurement corresponds to a physical quantity with a fixed format (i.e. the fields etc) across devices. There are of cause also mixed formed and there is also the possibility to use a phase centric view in the form

L1,device=A,some_tags=with_values voltage=...,reactive_power=...,etc 1669113259000000
L2,device=A,some_tags=with_values voltage=...,reactive_power=...,etc 1669113259000000
L3,device=A,some_tags=with_values voltage=...,reactive_power=...,etc 1669113259000000
total,device=A,some_tags=with_values current=...,reactive_power=...,etc 1669113259000000

Why do I bring this up? Because by adding tags to fields you effectively create new series, which corresponds effectively to the last example. So if you write down how your final metric set should look like, you can definitively create it my grouping all elements sharing the same tag into a measurement and the use processors to add the tags later.

lfarkas commented 1 year ago

@srebhan my only problem with all of the above solutions is that ALL of them can be solved with the current modbus plugin, BUT none of them can be solved with one modbus read sequence. modbus is a VERY slow protocol (eg in the above case 7200 baud) so even in a gigabit network age it's very important how fast we can read data. if we can read data in only one pass then we can achieve even 3 5 or 10 fps. and here comes the problem with your examples which currently slow it down to less then 1 fps . at least if i understand well the modbus plugin configuration possibility that's the case. but please correct me if I'm wrong!

SokoFromNZ commented 1 year ago

Which brings me back to my original question: Would be multiple [[inputs.modbus.request]] to the same slave+register bundled to one request to the modbus device?

srebhan commented 1 year ago

@SokoFromNZ no each request is exactly this "a request".

@lfarkas, all of the above can be done with one request! The first one is what you get if you you everything into one measurement and use the name properties to set the corresponding field name

 [[inputs.modbus.request]]
   slave_id = 9
   byte_order = "ABCD"
   register = "holding"
   measurement="your_device_name"
   fields = [
     { address=32016, name="ActivePower_L1", type="INT16", scale=0.1, output="FLOAT64" },
     { address=32018, name="ActivePower_L2", type="INT16", scale=0.1, output="FLOAT64" },
     { address=32020, name="ActivePower_L3", type="INT16", scale=0.1, output="FLOAT64" },
     { address=32022, name="ReactivePower_L1", type="INT16", scale=0.1, output="FLOAT64" },
     { address=32024, name="ReactivePower_L2", type="INT16", scale=0.1, output="FLOAT64" },
     { address=32026, name="ReactivePower_L3", type="INT16", scale=0.1, output="FLOAT64" },
     ...
   ]
   [inputs.modbus.request.tags]
     some_tags = "with value"

the second example is similar

 [[inputs.modbus.request]]
   slave_id = 9
   byte_order = "ABCD"
   register = "holding"
   fields = [
     { address=32016, measurement="ActivePower", name="L1", type="INT16", scale=0.1, output="FLOAT64" },
     { address=32018, measurement="ActivePower", name="L2", type="INT16", scale=0.1, output="FLOAT64" },
     { address=32020, measurement="ActivePower", name="L3", type="INT16", scale=0.1, output="FLOAT64" },
     { address=32022, measurement="ReactivePower", name="L1", type="INT16", scale=0.1, output="FLOAT64" },
     { address=32024, measurement="ReactivePower", name="L2", type="INT16", scale=0.1, output="FLOAT64" },
     { address=32026, measurement="ReactivePower", name="L3", type="INT16", scale=0.1, output="FLOAT64" },
     ...
   ]
   [inputs.modbus.request.tags]
     some_tags = "with value"
     device = "your_device_name"

and similar for the last one (I think you see the pattern)...

If you do have combinatorics, you should use a combinatoric measurement name and then use a processor, e.g.

 [[inputs.modbus.request]]
   slave_id = 9
   byte_order = "ABCD"
   register = "holding"
   fields = [
     { address=12016, measurement="ActivePower_Device1", name="L1", type="INT16", scale=0.1, output="FLOAT64" },
     { address=12018, measurement="ActivePower_Device1", name="L2", type="INT16", scale=0.1, output="FLOAT64" },
     { address=12020, measurement="ActivePower_Device1", name="L3", type="INT16", scale=0.1, output="FLOAT64" },
     { address=12022, measurement="ReactivePower_Device1", name="L1", type="INT16", scale=0.1, output="FLOAT64" },
     { address=12024, measurement="ReactivePower_Device1", name="L2", type="INT16", scale=0.1, output="FLOAT64" },
     { address=12026, measurement="ReactivePower_Device1", name="L3", type="INT16", scale=0.1, output="FLOAT64" },
     ...
     { address=32016, measurement="ActivePower_DeviceN", name="L1", type="INT16", scale=0.1, output="FLOAT64" },
     { address=32018, measurement="ActivePower_DeviceN", name="L2", type="INT16", scale=0.1, output="FLOAT64" },
     { address=32020, measurement="ActivePower_DeviceN", name="L3", type="INT16", scale=0.1, output="FLOAT64" },
     { address=32022, measurement="ReactivePower_DeviceN", name="L1", type="INT16", scale=0.1, output="FLOAT64" },
     { address=32024, measurement="ReactivePower_DeviceN", name="L2", type="INT16", scale=0.1, output="FLOAT64" },
     { address=32026, measurement="ReactivePower_DeviceN", name="L3", type="INT16", scale=0.1, output="FLOAT64" },
   ]
   [inputs.modbus.request.tags]
     some_tags = "with value"

[[processors.override]]
  order = 1
  namepass = ["*_Device1"]
  [processors.override.tags]
    device = "first"

[[processors.override]]
  order = 1
  namepass = ["*_Device2"]
  [processors.override.tags]
    device = "second"

[[processors.override]]
  order = 1
  namepass = ["*_Device3"]
  [processors.override.tags]
    device = "third"

[[processors.regex]]
  order = 2
  [[processors.regex.metric_rename]]
    pattern = '^(\w+)_.*$'
    replacement = '${1}'

It might also be helpful to use the starlark processor if the combinations are too big or the mapping logic more complex...

lfarkas commented 1 year ago

@srebhan do you mean in the last example like this?:

[[inputs.modbus.request]]
   slave_id = 9
   byte_order = "ABCD"
   register = "holding"
   fields = [
     { address=12016, measurement="electricity_L1", name="ActivePower", type="INT16", scale=0.1, output="FLOAT64" },
     { address=12018, measurement="electricity_L2", name="ActivePower", type="INT16", scale=0.1, output="FLOAT64" },
     { address=12020, measurement="electricity_L3", name="ActivePower", type="INT16", scale=0.1, output="FLOAT64" },
     { address=12022, measurement="electricity_L1", name="ReactivePower", type="INT16", scale=0.1, output="FLOAT64" },
     { address=12024, measurement="electricity_L2", name="ReactivePower", type="INT16", scale=0.1, output="FLOAT64" },
     { address=12026, measurement="electricity_L3", name="ReactivePower", type="INT16", scale=0.1, output="FLOAT64" },
     ...
   ]
   [inputs.modbus.request.tags]
     some_tags = "with value"

[[processors.override]]
  order = 1
  namepass = ["*_L1"]
  [processors.override.tags]
    phase = "L1"

[[processors.override]]
  order = 1
  namepass = ["*_L2"]
  [processors.override.tags]
    phase = "L2"

[[processors.override]]
  order = 1
  namepass = ["*_L3"]
  [processors.override.tags]
    phase = "L3"

[[processors.regex]]
  order = 2
  [[processors.regex.metric_rename]]
    pattern = '^(\w+)_.*$'
    replacement = '${1}'
srebhan commented 1 year ago

@lfarkas exactly.

srebhan commented 1 year ago

@lfarkas can we close this issue?