Apply filters only for a specific measurement name

[Laszlo-JFLMTCO]

Hi Everyone,

Started using Telegraf recently and I am sorry if I missed, but I cannot seem to find a way to filter out metrics for specific network devices only.

Our inputs_sysstat.conf has a very simple structure:

# Sysstat metrics collector
[[inputs.sysstat]]
  sadc_path = "/usr/lib64/sa/sadc" # required
  ## The more activities that are added, the more data is collected.
  activities = ["DISK"]
  # Dropping cpu metrics as those are going to be collected through inputs_cpu.conf
  namedrop = ["*cpu*"]
  [inputs.sysstat.options]
    -C = "cpu"
    -B = "paging"
    -b = "io"
    -d = "disk"             # requires DISK activity
    "-n ALL" = "network"
    "-P ALL" = "per_cpu"
    -q = "queue"
    -R = "mem"
    -r = "mem_util"
    -S = "swap_util"
    -u = "cpu_util"
    -v = "inode"
    -W = "swap"
    -w = "task"

This generates output with all available network devices as well as other metrics, such as disk, mem,task, etc. So simply using the following filter will not work as it will drop all metrics, except network for device=bond0.

  [inputs.sysstat.tagpass]
    device = ["bond0"]

I would be looking for filtering of within network only, so something like this expecting that it would "leave alone" all other non-network metrics:

  [inputs.sysstat.tagpass]
    network.device = ["bond0"]

Can you please advice how to set up filtering properly? Than you for your help.

[danielnelson]

This should drop all metrics with a device not starting with a b and not ending with a 0. Does this work for you?:

  [inputs.sysstat.tagdrop]
    device = ["[!b]*", "*[!0]"]

[Laszlo-JFLMTCO]

@danielnelson Many thanks for the suggestion. Unfortunately it still drops other metrics that has the device value populated. Here is an example.

Before updating conf with your suggestion:

Before update:
> queue,host=test.hostname.com blocked=0,ldavg-1=0.01,ldavg-15=0,ldavg-5=0.02,plist-sz=688,runq-sz=0 1534546688000000000
> task,host=test.hostname.com cswch_per_s=1238,proc_per_s=0 1534546688000000000
> mem_util,host=test.hostname.com kbactive=22204380,kbbuffers=0,kbcached=50296200,kbcommit=6324644,kbdirty=64,kbinact=28561772,kbmemfree=45442572,kbmemused=53427692,pct_commit=6.14,pct_memused=54.04 1534546688000000000
> inode,host=test.hostname.com dentunusd=2343859,file-nr=1808,inode-nr=342367,pty-nr=1 1534546688000000000
> paging,host=test.hostname.com fault_per_s=24,majflt_per_s=0,pct_vmeff=0,pgfree_per_s=21,pgpgin_per_s=0,pgpgout_per_s=0,pgscand_per_s=0,pgscank_per_s=0,pgsteal_per_s=0 1534546688000000000
> cpu,device=all,host=test.hostname.com pct_idle=99.96,pct_iowait=0,pct_nice=0,pct_steal=0,pct_system=0.04,pct_user=0 1534546688000000000
> swap,host=test.hostname.com pswpin_per_s=0,pswpout_per_s=0 1534546688000000000
> io,host=test.hostname.com bread_per_s=0,bwrtn_per_s=0,rtps=0,tps=0,wtps=0 1534546688000000000
> swap_util,host=test.hostname.com kbswpcad=9812,kbswpfree=4134652,kbswpused=59648,pct_swpcad=16.45,pct_swpused=1.42 1534546688000000000
> cpu_util,device=all,host=test.hostname.com pct_idle=99.96,pct_iowait=0,pct_nice=0,pct_steal=0,pct_system=0.04,pct_user=0 1534546688000000000
> mem,host=test.hostname.com bufpg_per_s=0,campg_per_s=3,frmpg_per_s=46 1534546688000000000
> disk,device=sda,host=test.hostname.com avgqu-sz=0,avgrq-sz=0,await=0,pct_util=0,rd_sec_per_s=0,svctm=0,tps=0,wr_sec_per_s=0 1534546688000000000
> disk,device=centos-root,host=test.hostname.com avgqu-sz=0,avgrq-sz=0,await=0,pct_util=0,rd_sec_per_s=0,svctm=0,tps=0,wr_sec_per_s=0 1534546688000000000
> disk,device=centos-swap,host=test.hostname.com avgqu-sz=0,avgrq-sz=0,await=0,pct_util=0,rd_sec_per_s=0,svctm=0,tps=0,wr_sec_per_s=0 1534546688000000000
> disk,device=centos-home,host=test.hostname.com avgqu-sz=0,avgrq-sz=0,await=0,pct_util=0,rd_sec_per_s=0,svctm=0,tps=0,wr_sec_per_s=0 1534546688000000000
> network,device=bond0,host=test.hostname.com coll_per_s=0,rxcmp_per_s=0,rxdrop_per_s=0,rxerr_per_s=0,rxfifo_per_s=0,rxfram_per_s=0,rxkB_per_s=1.43,rxmcst_per_s=1,rxpck_per_s=13,txcarr_per_s=0,txcmp_per_s=0,txdrop_per_s=0,txerr_per_s=0,txfifo_per_s=0,txkB_per_s=0.64,txpck_per_s=6 1534546688000000000
> network,device=eno4,host=test.hostname.com coll_per_s=0,rxcmp_per_s=0,rxdrop_per_s=0,rxerr_per_s=0,rxfifo_per_s=0,rxfram_per_s=0,rxkB_per_s=0,rxmcst_per_s=0,rxpck_per_s=0,txcarr_per_s=0,txcmp_per_s=0,txdrop_per_s=0,txerr_per_s=0,txfifo_per_s=0,txkB_per_s=0,txpck_per_s=0 1534546688000000000
> network,device=ens2f0,host=test.hostname.com coll_per_s=0,rxcmp_per_s=0,rxdrop_per_s=0,rxerr_per_s=0,rxfifo_per_s=0,rxfram_per_s=0,rxkB_per_s=1.43,rxmcst_per_s=1,rxpck_per_s=13,txcarr_per_s=0,txcmp_per_s=0,txdrop_per_s=0,txerr_per_s=0,txfifo_per_s=0,txkB_per_s=0.64,txpck_per_s=6 1534546688000000000
> network,device=eno3,host=test.hostname.com coll_per_s=0,rxcmp_per_s=0,rxdrop_per_s=0,rxerr_per_s=0,rxfifo_per_s=0,rxfram_per_s=0,rxkB_per_s=0,rxmcst_per_s=0,rxpck_per_s=0,txcarr_per_s=0,txcmp_per_s=0,txdrop_per_s=0,txerr_per_s=0,txfifo_per_s=0,txkB_per_s=0,txpck_per_s=0 1534546688000000000
> network,device=ens1f1,host=test.hostname.com coll_per_s=0,rxcmp_per_s=0,rxdrop_per_s=0,rxerr_per_s=0,rxfifo_per_s=0,rxfram_per_s=0,rxkB_per_s=0,rxmcst_per_s=0,rxpck_per_s=0,txcarr_per_s=0,txcmp_per_s=0,txdrop_per_s=0,txerr_per_s=0,txfifo_per_s=0,txkB_per_s=0,txpck_per_s=0 1534546688000000000
> network,host=test.hostname.com access_per_s=0,badcall_per_s=0,call_per_s=0,getatt_per_s=0,hit_per_s=0,ip-frag=0,miss_per_s=0,packet_per_s=0,rawsck=0,read_per_s=0,retrans_per_s=0,saccess_per_s=0,scall_per_s=0,sgetatt_per_s=0,sread_per_s=0,swrite_per_s=0,tcp-tw=3,tcp_per_s=0,tcpsck=17,totsck=242,udp_per_s=0,udpsck=4,write_per_s=0 1534546688000000000
> network,device=eno2,host=test.hostname.com coll_per_s=0,rxcmp_per_s=0,rxdrop_per_s=0,rxerr_per_s=0,rxfifo_per_s=0,rxfram_per_s=0,rxkB_per_s=0,rxmcst_per_s=0,rxpck_per_s=0,txcarr_per_s=0,txcmp_per_s=0,txdrop_per_s=0,txerr_per_s=0,txfifo_per_s=0,txkB_per_s=0,txpck_per_s=0 1534546688000000000
> network,device=ens2f1,host=test.hostname.com coll_per_s=0,rxcmp_per_s=0,rxdrop_per_s=0,rxerr_per_s=0,rxfifo_per_s=0,rxfram_per_s=0,rxkB_per_s=0,rxmcst_per_s=0,rxpck_per_s=0,txcarr_per_s=0,txcmp_per_s=0,txdrop_per_s=0,txerr_per_s=0,txfifo_per_s=0,txkB_per_s=0,txpck_per_s=0 1534546688000000000
> network,device=vethab6cfbb,host=test.hostname.com coll_per_s=0,rxcmp_per_s=0,rxdrop_per_s=0,rxerr_per_s=0,rxfifo_per_s=0,rxfram_per_s=0,rxkB_per_s=0,rxmcst_per_s=0,rxpck_per_s=0,txcarr_per_s=0,txcmp_per_s=0,txdrop_per_s=0,txerr_per_s=0,txfifo_per_s=0,txkB_per_s=0,txpck_per_s=0 1534546688000000000
> network,device=docker0,host=test.hostname.com coll_per_s=0,rxcmp_per_s=0,rxdrop_per_s=0,rxerr_per_s=0,rxfifo_per_s=0,rxfram_per_s=0,rxkB_per_s=0,rxmcst_per_s=0,rxpck_per_s=0,txcarr_per_s=0,txcmp_per_s=0,txdrop_per_s=0,txerr_per_s=0,txfifo_per_s=0,txkB_per_s=0,txpck_per_s=0 1534546688000000000
> network,device=br-5cd31b474843,host=test.hostname.com coll_per_s=0,rxcmp_per_s=0,rxdrop_per_s=0,rxerr_per_s=0,rxfifo_per_s=0,rxfram_per_s=0,rxkB_per_s=0,rxmcst_per_s=0,rxpck_per_s=0,txcarr_per_s=0,txcmp_per_s=0,txdrop_per_s=0,txerr_per_s=0,txfifo_per_s=0,txkB_per_s=0,txpck_per_s=0 1534546688000000000
> network,device=ens1f0,host=test.hostname.com coll_per_s=0,rxcmp_per_s=0,rxdrop_per_s=0,rxerr_per_s=0,rxfifo_per_s=0,rxfram_per_s=0,rxkB_per_s=0,rxmcst_per_s=0,rxpck_per_s=0,txcarr_per_s=0,txcmp_per_s=0,txdrop_per_s=0,txerr_per_s=0,txfifo_per_s=0,txkB_per_s=0,txpck_per_s=0 1534546688000000000
> network,device=br-29cc25c9f21b,host=test.hostname.com coll_per_s=0,rxcmp_per_s=0,rxdrop_per_s=0,rxerr_per_s=0,rxfifo_per_s=0,rxfram_per_s=0,rxkB_per_s=0,rxmcst_per_s=0,rxpck_per_s=0,txcarr_per_s=0,txcmp_per_s=0,txdrop_per_s=0,txerr_per_s=0,txfifo_per_s=0,txkB_per_s=0,txpck_per_s=0 1534546688000000000
> network,device=lo,host=test.hostname.com coll_per_s=0,rxcmp_per_s=0,rxdrop_per_s=0,rxerr_per_s=0,rxfifo_per_s=0,rxfram_per_s=0,rxkB_per_s=0,rxmcst_per_s=0,rxpck_per_s=0,txcarr_per_s=0,txcmp_per_s=0,txdrop_per_s=0,txerr_per_s=0,txfifo_per_s=0,txkB_per_s=0,txpck_per_s=0 1534546688000000000
> network,device=eno1,host=test.hostname.com coll_per_s=0,rxcmp_per_s=0,rxdrop_per_s=0,rxerr_per_s=0,rxfifo_per_s=0,rxfram_per_s=0,rxkB_per_s=0,rxmcst_per_s=0,rxpck_per_s=0,txcarr_per_s=0,txcmp_per_s=0,txdrop_per_s=0,txerr_per_s=0,txfifo_per_s=0,txkB_per_s=0,txpck_per_s=0 1534546688000000000
> network,device=br-6b7646a16bb2,host=test.hostname.com coll_per_s=0,rxcmp_per_s=0,rxdrop_per_s=0,rxerr_per_s=0,rxfifo_per_s=0,rxfram_per_s=0,rxkB_per_s=0,rxmcst_per_s=0,rxpck_per_s=0,txcarr_per_s=0,txcmp_per_s=0,txdrop_per_s=0,txerr_per_s=0,txfifo_per_s=0,txkB_per_s=0,txpck_per_s=0 1534546688000000000

After updating conf with your suggestion:

> swap,host=test.hostname.com pswpin_per_s=0,pswpout_per_s=0 1534546749000000000
> mem_util,host=test.hostname.com kbactive=22183572,kbbuffers=0,kbcached=50296196,kbcommit=6045624,kbdirty=52,kbinact=28561316,kbmemfree=45474160,kbmemused=53396104,pct_commit=5.87,pct_memused=54.01 1534546749000000000
> inode,host=test.hostname.com dentunusd=2343915,file-nr=1856,inode-nr=342412,pty-nr=1 1534546749000000000
> swap_util,host=test.hostname.com kbswpcad=9812,kbswpfree=4134652,kbswpused=59648,pct_swpcad=16.45,pct_swpused=1.42 1534546749000000000
> task,host=test.hostname.com cswch_per_s=1179,proc_per_s=0 1534546749000000000
> io,host=test.hostname.com bread_per_s=0,bwrtn_per_s=0,rtps=0,tps=0,wtps=0 1534546749000000000
> queue,host=test.hostname.com blocked=0,ldavg-1=0.08,ldavg-15=0,ldavg-5=0.03,plist-sz=656,runq-sz=0 1534546749000000000
> paging,host=test.hostname.com fault_per_s=23,majflt_per_s=0,pct_vmeff=0,pgfree_per_s=21,pgpgin_per_s=0,pgpgout_per_s=0,pgscand_per_s=0,pgscank_per_s=0,pgsteal_per_s=0 1534546749000000000
> mem,host=test.hostname.com bufpg_per_s=0,campg_per_s=0,frmpg_per_s=-31 1534546749000000000
> network,device=bond0,host=test.hostname.com coll_per_s=0,rxcmp_per_s=0,rxdrop_per_s=0,rxerr_per_s=0,rxfifo_per_s=0,rxfram_per_s=0,rxkB_per_s=0.31,rxmcst_per_s=0,rxpck_per_s=2,txcarr_per_s=0,txcmp_per_s=0,txdrop_per_s=0,txerr_per_s=0,txfifo_per_s=0,txkB_per_s=0.12,txpck_per_s=1 1534546749000000000
> network,host=test.hostname.com access_per_s=0,badcall_per_s=0,call_per_s=0,getatt_per_s=0,hit_per_s=0,ip-frag=0,miss_per_s=0,packet_per_s=0,rawsck=0,read_per_s=0,retrans_per_s=0,saccess_per_s=0,scall_per_s=0,sgetatt_per_s=0,sread_per_s=0,swrite_per_s=0,tcp-tw=3,tcp_per_s=0,tcpsck=17,totsck=242,udp_per_s=0,udpsck=4,write_per_s=0 1534546749000000000

Please notice, how it drops the followings because they have device=all:

> cpu,device=all,host=test.hostname.com pct_idle=99.96,pct_iowait=0,pct_nice=0,pct_steal=0,pct_system=0.04,pct_user=0 1534546688000000000
> cpu_util,device=all,host=test.hostname.com pct_idle=99.96,pct_iowait=0,pct_nice=0,pct_steal=0,pct_system=0.04,pct_user=0 1534546688000000000
> disk,device=sda,host=test.hostname.com avgqu-sz=0,avgrq-sz=0,await=0,pct_util=0,rd_sec_per_s=0,svctm=0,tps=0,wr_sec_per_s=0 1534546688000000000
> disk,device=centos-root,host=test.hostname.com avgqu-sz=0,avgrq-sz=0,await=0,pct_util=0,rd_sec_per_s=0,svctm=0,tps=0,wr_sec_per_s=0 1534546688000000000
> disk,device=centos-swap,host=test.hostname.com avgqu-sz=0,avgrq-sz=0,await=0,pct_util=0,rd_sec_per_s=0,svctm=0,tps=0,wr_sec_per_s=0 1534546688000000000
> disk,device=centos-home,host=test.hostname.com avgqu-sz=0,avgrq-sz=0,await=0,pct_util=0,rd_sec_per_s=0,svctm=0,tps=0,wr_sec_per_s=0 1534546688000000000

I was trying to update the filter you provided, but I do not have much luck with it. Do you have any additional suggestions? Thank you for your help.

[Laszlo-JFLMTCO]

@danielnelson Actually playing a bit more with the filter, here is what I have now:

  [inputs.sysstat.tagdrop]
    device = ["[!b|a]*", "*[!0|l]"]

Resulting in the following output => disk metrics are still missing as they have device=centos-...:

> paging,host=test.hostname.com fault_per_s=67,majflt_per_s=0,pct_vmeff=0,pgfree_per_s=79,pgpgin_per_s=0,pgpgout_per_s=0,pgscand_per_s=0,pgscank_per_s=0,pgsteal_per_s=0 1534548019000000000
> io,host=test.hostname.com bread_per_s=0,bwrtn_per_s=0,rtps=0,tps=0,wtps=0 1534548019000000000
> swap,host=test.hostname.com pswpin_per_s=0,pswpout_per_s=0 1534548019000000000
> swap_util,host=test.hostname.com kbswpcad=9812,kbswpfree=4134652,kbswpused=59648,pct_swpcad=16.45,pct_swpused=1.42 1534548019000000000
> inode,host=test.hostname.com dentunusd=2344007,file-nr=1808,inode-nr=342279,pty-nr=1 1534548019000000000
> mem,host=test.hostname.com bufpg_per_s=0,campg_per_s=3,frmpg_per_s=91 1534548019000000000
> cpu,device=all,host=test.hostname.com pct_idle=99.96,pct_iowait=0,pct_nice=0,pct_steal=0,pct_system=0.04,pct_user=0 1534548019000000000
> queue,host=test.hostname.com blocked=0,ldavg-1=0,ldavg-15=0,ldavg-5=0,plist-sz=643,runq-sz=0 1534548019000000000
> task,host=test.hostname.com cswch_per_s=1297,proc_per_s=0 1534548019000000000
> mem_util,host=test.hostname.com kbactive=22180060,kbbuffers=0,kbcached=50296696,kbcommit=5940760,kbdirty=72,kbinact=28560048,kbmemfree=45481704,kbmemused=53388560,pct_commit=5.76,pct_memused=54 1534548019000000000
> cpu_util,device=all,host=test.hostname.com pct_idle=99.96,pct_iowait=0,pct_nice=0,pct_steal=0,pct_system=0.04,pct_user=0 1534548019000000000
> network,host=test.hostname.com access_per_s=0,badcall_per_s=0,call_per_s=0,getatt_per_s=0,hit_per_s=0,ip-frag=0,miss_per_s=0,packet_per_s=0,rawsck=0,read_per_s=0,retrans_per_s=0,saccess_per_s=0,scall_per_s=0,sgetatt_per_s=0,sread_per_s=0,swrite_per_s=0,tcp-tw=3,tcp_per_s=0,tcpsck=17,totsck=238,udp_per_s=0,udpsck=4,write_per_s=0 1534548019000000000
> network,device=bond0,host=test.hostname.com coll_per_s=0,rxcmp_per_s=0,rxdrop_per_s=0,rxerr_per_s=0,rxfifo_per_s=0,rxfram_per_s=0,rxkB_per_s=0.25,rxmcst_per_s=0,rxpck_per_s=3,txcarr_per_s=0,txcmp_per_s=0,txdrop_per_s=0,txerr_per_s=0,txfifo_per_s=0,txkB_per_s=0.59,txpck_per_s=2 1534548019000000000

[danielnelson]

Maybe it would be better to try to tagdrop all the unwanted interface name types:

  [inputs.sysstat.tagdrop]
    device = ["br*", "lo*", "en*", "docker*", "veth*"]

[Laszlo-JFLMTCO]

Thank you for your help @danielnelson I think for now I will go with the tagdrop you posted, but I am not sure if it will work long term as it might force us to have unique sysstat configs for different servers.

I would have really liked to be able to apply a tagpass filter for bond0 only within network metrics, as that would leave the other metrics alone and that is exactly what we need in terms of output.

Correct me if I am wrong, but it seems that it is not feasible currently.

If there would be any of the followings feasible, it would be awesome and would be a very straight forward way to set up filters:

  [inputs.sysstat.tagpass]
    network.device = ["bond0"]

OR

  [inputs.sysstat.network.tagpass]
    device = ["bond0"]

Anyways, many thanks again for your help and please let me know if there is anything similar already available that I might have missed.

[danielnelson]

Your are correct, it's not possible right now. I'm going to update the title of this issue and we will consider it a feature request.

[Laszlo-JFLMTCO]

Sounds good. Thank you for your help.

[srebhan]

I think this should be possible now using the metricpass option (see selection documentation). Closing as fixed now. If metricpass is not working for you, please let me know and reopen the issue!