Idea: Add various field translations

ppascher commented 7 years ago

Hello,

I just started using logstash-codec-sflow and added some dictionary files to add more information: In order to translate interface Ids from sflow messages to names I could more easily associate with my switches interfaces I first used this command to get a list of id => name: snmpwalk -v2c -c public <switch IP> 1.3.6.1.2.1.2.2.1.2

Afterwards I modified that output a little to get a valid yaml dictionary file. I used this file to translate the following fields: "source_id_index" "input_interface" "output_interface"

One example:

 translate {
      field => "input_interface"
      dictionary_path => "/etc/logstash/dictionaries/ex2200_interface_names.yaml"
      fallback => "UNKNOWN"
      destination => "input_interface_name"
    }

I got the idea from here: https://github.com/NETWAYS/sflow/blob/master/lib/sflow/snmp/iface_names.rb

I also looked at https://whiskeyalpharomeo.com/2015/06/13/logstash-and-sflow/ and included the iana_services and iana_protocols dictionary files:

translate {
      field => "ip_protocol"
      dictionary_path => "/etc/logstash/dictionaries/iana_protocols.yaml"
      fallback => "UNKNOWN"
      destination => "ip_protocol_name"
    }
translate {
      field => "src_port"
      dictionary_path => "/etc/logstash/dictionaries/iana_services.yaml"
      fallback => "UNKNOWN"
      destination => "src_port_name"
    }
translate {
      field => "dst_port"
      dictionary_path => "/etc/logstash/dictionaries/iana_services.yaml"
      fallback => "UNKNOWN"
      destination => "dst_port_name"
    }

Would it be possible or make sense to include those translations directly into the plugin (services and protocols static, interface names dynamic)? Thanks for your work.

ashangit commented 7 years ago

Hi,

There is already a way to do so with this plugin. You should set those parameters:

snmp_interface to true
snmp_community the snmp community with read rights (default public) This will create 4 new fields source_id_index_descr, input/output_interface_descr, interface_index_descr

In order to not do a snmp request for each sflow packet there is an lru cache. You can set is size and the ttl updating those parameters:

interface_cache_size: size of the cache (default 1000). An element is a - interface_cache_ttl: duration for each element in the cache (default 3600)

I do not had any chance to fully validate this part of the plugin so I would be happy to have your feedback.

ppascher commented 7 years ago

Awesome, thanks. I modified my input-sflow.conf as you suggested:

input {
  udp {
    port => 6543
    codec => sflow {
      snmp_interface => "true"
      snmp_community => "public"
      interface_cache_size => 1000
      interface_cache_ttl => 3600
    }
    type => "sflow"
  }
}

And the fields you mentioned get added correctly:

{
  "_index": "sflow-2016.09.24",
  "_type": "sflow",
  "_id": "AVdcNG4hxg12CVppWgSX",
  "_score": null,
  "_source": {
    "@version": "1",
    "@timestamp": "2016-09-24T12:38:41.807Z",
    "agent_ip": "xx.xx.xx.xx",
    "ip_version": "4",
    "sub_agent_id": "16",
    "uptime_in_ms": "737216214",
    "source_id_type": "0",
    "source_id_index": "525",
    "sampling_rate": "1000",
    "sample_pool": "53881000",
    "drops": "0",
    "input_interface": "537",
    "output_interface": "525",
    "protocol": "1",
    "frame_length": "198",
    "stripped": "4",
    "eth_dst": "AB:CD:EF:GH:IJ:KL",
    "eth_src": "LK:JI:HG:FE:DC:BA",
    "eth_type": "2048",
    "ip_protocol": "6",
    "src_ip": "xx.xx.xx.xx",
    "dst_ip": "xx.xx.xx.xx",
    "src_port": "2049",
    "dst_port": "976",
    "padded": "0",
    "src_vlan": "<vlan_id>",
    "src_priority": "0",
    "dst_vlan": "<vlan_id>",
    "dst_priority": "0",
    "frame_length_times_sampling_rate": 198000,
    "sflow_type": "flow_sample",
    "source_id_index_descr": "ge-0/0/2",
    "input_interface_descr": "ge-0/0/4",
    "output_interface_descr": "ge-0/0/2",
    "type": "sflow",
    "host": "xx.xx.xx.xx",
    "protocol_name": "ETHERNET",
    "eth_type_name": "IP",
    "ip_protocol_name": "TCP",
    "src_port_name": "nfs",
    "dst_port_name": "UNKNOWN",
    "src_vlan_name": "<vlan_name>",
    "dst_vlan_name": "<vlan_name>",
  },
  "fields": {
    "@timestamp": [
      1474720721807
    ]
  },
  "sort": [
    1474720721807
  ]
}

Works great. Any thougth on adding the dictionaries for IANA protocols and services (for ip_protocol, src_port, dst_port? Thanks for your work!

ashangit commented 7 years ago

I prefer to rely on the translate plugin for those kind of requirement But if you already have those dictionnaries we can embeded them in the plugin so any one could then use them in a translate plugin

path-network / logstash-codec-sflow

Idea: Add various field translations #3