Enhancement: Populate interface name from ifIndex

[DanSheps]

I think it would be beneficial to populate the interface name. There are a couple of ways to accomplish this, and not sure how robust the logstash ruby plugin is, but polling SNMP + Caching response for a time might be one way to accomplish this.

[robcowart]

I agree with you completely, the challenge is figuring out how to best achieve this. What is needed is an snmpget filter with caching, much like the DNS filter. So definitely worth doing, I just can't promise it quickly unless someone else tackles development of the Logstash filter.

[slinderud]

Cisco supports sending the port/snmp mappings along with the netflow records: (config)#flow exporter test (config-flow-exporter)#option interface-table timeout 30

[DanSheps]

I think the problem is some people may still be using older versions of IOS or using ASA's which don't support Flexible Netflow.

[sumo-89]

+1 :)

[skipper00]

+1 :)

[jlixfeld]

+1 :)

[coredidan]

+1

[dfrolov95]

+1

[jlixfeld]

I've been poking around trying to hack this into 3.1.0 for my XR boxes, which are exporting v9, and the options template and the record (verified by tcpdump (attached)), and I've made these changes:

root@ls1:/home/jlixfeld/elastiflow# git diff
diff --git a/kibana/elastiflow.index_pattern.json b/kibana/elastiflow.index_pattern.json
index 69e656b..f4f55f7 100644
--- a/kibana/elastiflow.index_pattern.json
+++ b/kibana/elastiflow.index_pattern.json
@@ -2,7 +2,7 @@
        "attributes": {
                "title": "elastiflow-*",
                "timeFieldName": "@timestamp",
-               "fields": "
<snip>
+               "fields": 
<snip>
{\"name\":\"flow.if_desc\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},
<snip>
diff --git a/logstash/elastiflow/conf.d/20_filter_20_netflow.logstash.conf b/logstash/elastiflow/conf.d/20_filter_20_netflow.logstash.conf
index 49bd7bc..755e1fb 100644
--- a/logstash/elastiflow/conf.d/20_filter_20_netflow.logstash.conf
+++ b/logstash/elastiflow/conf.d/20_filter_20_netflow.logstash.conf
@@ -87,6 +87,7 @@ filter {
           "[netflow][sampling_interval]" => "[flow][sampling_interval]"
           "[netflow][input_snmp]" => "[flow][input_snmp]"
           "[netflow][output_snmp]" => "[flow][output_snmp]"
+          "[netflow][if_desc]" => "[flow][if_desc]"
           "[netflow][protocol]" => "[flow][ip_protocol]"
           "[netflow][tcp_flags]" => "[flow][tcp_flags]"
         }
diff --git a/logstash/elastiflow/templates/elastiflow.template.json b/logstash/elastiflow/templates/elastiflow.template.json
index 975f8a1..9735623 100755
--- a/logstash/elastiflow/templates/elastiflow.template.json
+++ b/logstash/elastiflow/templates/elastiflow.template.json
@@ -12149,6 +12149,9 @@
             "dst_rep_tags": {
               "type": "keyword"
             },
+            "if_desc": {
+              "type": "keyword"
+            },
             "input_snmp": {
               "type": "keyword"
             },

I deleted the index pattern in Kibana and re-added it, and re-started logstash, but I'm not seeing anything when inspecting the flow records in Kibana.

I also checked for support in logstash-codec-netflow, and the field definitions seems to include code 83, which is if_desc. I'm not sure if what I've done to try and make this work is even remotely adequate tho :(

Help? :)

[robcowart]

I have added support for populating ingress and egress interface names from a yaml dictionary file. This will be added to the next release.

[MHammett]

I see https://github.com/robcowart/elastiflow/blob/master/logstash/elastiflow/dictionaries/ifName.yml

Except how do you actually use it?

Example

"192.0.2.11::ifName.2": "eth0" "10.0.254.21::ifName.1": "sfp-sfpplus1" "10.0.254.21::ifName.18": "VLAN811-NETFLIX" "208.72.111.161::ifName.18": "VLAN811-NETFLIX"

I have a box at 208.72.111.161 sending IPFIX to 10.0.254.21. There, nProbe Pro (eventually, not configured yet) adds BGP data to the flow and passes it on to elastiflow. I assumed it was 10.0.254.21 that I needed to enter, but when that didn't work, I tried the IP of the original source. That didn't work either.

How do I troubleshoot this? Also, to make sure the changes I made take effect, I reboot the server. What's the minimum I have to do to get Elastiflow to re-read the config files?

[astronode]

Can't seem to get this working. I'm getting interface names such as "index; 43962354" from Nexus 7K, I've defined that in the ifName file but nothing happens.

[MHammett]

@astronode I put in what I had above and got nothing. Rebooted the server and more nothing. Sometime later, the VLAN811 was identified in the GUI, but of course, now I don't know if it was flagging on 10.0.254.21 or 208.72.111.161 as the other interface on 10.0.254.21 wasn't being flagged. I put in all of the rest of the interface definitions, rebooted, and the rest are still not identified. I'm not sure what triggers the reading of this file.

[robcowart]

Consider this example:

"192.0.2.11::ifName.2": "eth0"

The convention is:

"<exporter_ip>::ifName.<ifIndex>": "<whatever_name_you_want>"

Where exporter_ip must be the value of node.ipaddr from the flow data, and ifIndex will come from either flow.input_snmp or flow.output_snmp.

The logic that implements this is...

if [flow][input_snmp] {
  mutate {
    id => "elastiflow_postproc_translate_in_if_key"
    add_field => { "[@metadata][in_if_key]" => "%{[node][ipaddr]}::ifName.%{[flow][input_snmp]}" }
  }
  translate {
    id => "elastiflow_postproc_translate_input_ifname"
    dictionary_path => "${ELASTIFLOW_DICT_PATH:/etc/logstash/elastiflow/dictionaries}/ifName.yml"
    field => "[@metadata][in_if_key]"
    destination => "[flow][input_ifname]"
    fallback => "index: %{[flow][input_snmp]}"
    refresh_behaviour => "replace"
  }
}
if [flow][output_snmp] {
  mutate {
    id => "elastiflow_postproc_translate_out_if_key"
    add_field => { "[@metadata][out_if_key]" => "%{[node][ipaddr]}::ifName.%{[flow][output_snmp]}" }
  }
  translate {
    id => "elastiflow_postproc_translate_output_ifname"
    dictionary_path => "${ELASTIFLOW_DICT_PATH:/etc/logstash/elastiflow/dictionaries}/ifName.yml"
    field => "[@metadata][out_if_key]"
    destination => "[flow][output_ifname]"
    fallback => "index: %{[flow][output_snmp]}"
    refresh_behaviour => "replace"
  }
}

Make sure that the fields that build the key are in your data, and match the values configured in ifName.yml.

[MHammett]

Okay, so I was right with the 10.0.254.21 IP as that's what shows up in the node.hostname filter dropdown.

So then what triggers the reading of this file? My ifName.yml should be correct, yet only part of it is being read.

[RohanAdhith]

I tried to modify the fields to match the values configured in ifName.yml, but no luck. Probably I am missing something somewhere.

Could you please share which field needs to be changed to Populate interface name from ifIndex?