Support for ESXi VDS - ESXi 5.1+

[benlavender]

Ticket request to support IPFIX for ESXi 5.1 and above. Any NetFlow exports sent from ESXi devices on ESXi 5.1+ now only support IPFIX.

I have this implemented myself using this plugin including the @bodgit IPFIX support and receive the below in the logstash.log file:

:message=>"Unsupported enterprise", :enterprise=>6876, :level=>:warn}

[bodgit]

So somewhere VMware should document any custom fields that they send which will be a tuple of id, name and size for each field. I had a quick look but couldn't see a list published anywhere.

[benlavender]

Does this help? http://sourceforge.net/p/nfdump/mailman/nfdump-discuss/thread/1492FDFF-6FAE-4FA4-9D7D-787064E5FCFB@onthenet.com.au/

I've logged a support request to VMware to see if the information can be provided.

[jorritfolmer]

Can you do some packet capturing on your logstash instance so we can look at it? tcpdump -i eth0 -s 0 -w /tmp/ipfix_esxi.pcap udp and port 9995 It seems you can attach files to posts like this, or you can use something like filebin.ca to upload it to.

[jorritfolmer]

I created this patch to the ipfix.yaml in the @bodgit ipfix branch to support the IPFIX fields exported by a vSphere 5.5 virtual distributed switch I had access to. Since these fields are undocumented, I've simply named them after their enterprise and element id's. Any word from VMware yet on this, @benlavender ?

diff --git a/lib/logstash/codecs/netflow/ipfix.yaml b/lib/logstash/codecs/netflow/ipfix.yaml
index 70f2829..51ab7c4 100644
--- a/lib/logstash/codecs/netflow/ipfix.yaml
+++ b/lib/logstash/codecs/netflow/ipfix.yaml
@@ -1202,6 +1202,37 @@
   433:
   - :uint64
   - :ignoredLayer2FrameTotalCount
+6876:
+  880:
+  - :uint8
+  - :6876_880
+  881:
+  - :uint32
+  - :6876_881
+  882:
+  - :uint32
+  - :6876_882
+  883:
+  - :string
+  - :6876_883
+  884:
+  - :string
+  - :6876_884
+  886:
+  - :uint16
+  - :6876_886
+  887:
+  - :uint16
+  - :6876_887
+  888:
+  - :uint16
+  - :6876_888
+  889:
+  - :uint8
+  - :6876_889
+  890:
+  - :uint16
+  - :6876_890
 29305:
   1:
   - :uint64

A sample rubydebug output from Logstash looks like this:

{
    "@timestamp" => "2015-11-07T21:47:04.000Z",
       "netflow" => {
                         "version" => 10,
               "sourceIPv4Address" => "172.30.11.202",
          "destinationIPv4Address" => "172.30.11.110",
                 "octetDeltaCount" => 41,
                "packetDeltaCount" => 1,
           "flowStartMilliseconds" => "2015-11-07T21:46:50.000Z",
             "flowEndMilliseconds" => "2015-11-07T21:46:50.000Z",
             "sourceTransportPort" => 1433,
        "destinationTransportPort" => 38691,
                "ingressInterface" => 202,
                 "egressInterface" => 151,
                 "layer2SegmentId" => 0,
              "protocolIdentifier" => 6,
                   "flowEndReason" => 1,
                  "tcpControlBits" => 16,
                "ipClassOfService" => 0,
                      "maximumTTL" => 128,
                   "flowDirection" => 1,
                        "6876_890" => 1,
                        "6876_888" => 2,
                        "6876_889" => 0
    },
      "@version" => "1",
          "host" => "172.16.32.201"
}

[benlavender]

Vendor hasn’t been much use as they’re wanting to know why we think their implementation deviates from the standard RFC5101, basically I’ve not managed to speak with anyone that is aware of their own modifications to the IPFIX exports records or understands the difference. I’ve noticed private entries in the records as below:

These correspond to your updates in the ipfix.yaml.

[hkshirish]

Hi JorritFolmerto

Upgraded the latest logstash codec from the https://github.com/logstash-plugins/logstash-codec-netflow/pull/10 and then applied the patch for vmware ipfix. https://github.com/logstash-plugins/logstash-codec-netflow/issues/28

Then I played the IPFix packets from VMware. Attached the same.(to your email id)

But I see following messages for the attached pcap. Let us know if there is any fix available for parsing the VMWare IPfix packets

---

No matching template for flow id 257 {:level=>:warn} No matching template for flow id 256 {:level=>:warn} Unsupported enterprise field {:type=>950, :enterprise=>6876, :length=>4, :level=>:warn} No matching template for flow id 256 {:level=>:warn} Unsupported enterprise field {:type=>950, :enterprise=>6876, :length=>4, :level=>:warn} No matching template for flow id 256 {:level=>:warn} No matching template for flow id 256 {:level=>:warn} No matching template for flow id 256 {:level=>:warn} No matching template for flow id 256 {:level=>:warn} No matching template for flow id 256 {:level=>:warn} No matching template for flow id 256 {:level=>:warn} No matching template for flow id 256 {:level=>:warn}

---

[jorritfolmer]

Thanks for sending the .pcap. However there are no template packets, only data packets. Without template packets the data packets cannot be decoded, so I can't replay them to my local Logstash to test with.

Could you try capturing them too? I assume than making a capture of several minutes will include both data and template packets.

[hkshirish]

Attached the pcap which has templates.

The following packets have template in it packet 1, 135,195

Regards, Shirish

On Fri, Jun 17, 2016 at 1:01 PM, Jorrit Folmer notifications@github.com wrote:

Thanks for sending the .pcap. However there are no template packets, only data packets. Without template packets the data packets cannot be decoded, so I can't replay them to my local Logstash to test with.

Could you try capturing them too? I assume than making a capture of several minutes will include both data and template packets.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/logstash-plugins/logstash-codec-netflow/issues/28#issuecomment-226700468, or mute the thread https://github.com/notifications/unsubscribe/AN3bzm7yPQd1zX_j7YMiUtKv2HgyFEcpks5qMk1KgaJpZM4GXal7 .

[jorritfolmer]

I haven't found any documentation about there fields, especially the 951 field, and don't have time to look into it. So the quickest way to get NSX data in ES is to skip these unknown fields. If you have already modified the ipfix.yaml with the vSphere additions earlier in this thread, then add:

  950:
  - skip
  951:
  - skip
  952:
  - skip

Otherwise add:

6876:
  950:
  - skip
  951:
  - skip
  952:
  - skip

[schif1]

Hello,

Some news about the support of vDS 6.0 netflow 10 ( IPFIX) ? Do you want some sample ? How can i help to improve the support of VMware ?

Thanks

[jorritfolmer]

If you can send me a .pcap of vDS netflow traffic I can take a look at it. Email in profile. It would be especially helpful if you have any information about the vendor specific fields.

[jorritfolmer]

Thanks for sending me a .pcap. It seems there is no difference between a vDS 5.5 and vDS 6.0, as far as Netflow/IPFIX is concerned.

This means you can apply the patch (see thread above) to ipfix.yaml in your Logstash install.

If you have a VMware support contract, you might want to inquire about the meaning of the 883,...,889,890 fields. Much appreciated :)

Save the patch in something like /tmp/vDS_ipfix.patch and then:

cd /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-codec-netflow-3.1.2/lib/logstash/codecs/netflow
patch < /tmp/vDS_ipfix.patch

[hail100]

Hi jorritfolmer

I cannot get any data use vDS_ipfix.patch. VDS6.0, Logstash5.0.1, Logstash-codec-netflow2.1.3. VDS config select Process Internal flow only.

input {
  udp {
     port => 4739
     codec => netflow {
        versions => [10]
        target => ipfix
    }
    type => ipfix
  }
}

some debug logs

[2016-11-24T11:23:44,484][WARN ][logstash.codecs.netflow  ] No matching template for flow id 256

2016-11-24T03:13:46.000Z 10.10.7.5 %{message}

Thanks

[jorritfolmer]

@hail100 That is very little information to work with. What is your Logstash version, Logstash-codec-netflow version, logstash config? What other possible causes have you already eliminated?

[sliddjur]

@jorritfolmer Here are captured flows with template from vmware vsphere.

vmware.zip

I'm also seeing [WARN ][logstash.codecs.netflow ] Unsupported enterprise {:enterprise=>6876} in my log

I also made a netflow_definitions file for the flows. But I don't know if they're correct? And how do I handle the Type 889 [pen: VMware Inc.] (6876) fields? I never see my definitions file being used in the log though.

---
1:
- 8
- :in_bytes
2:
- 8
- :in_pkts
4:
- :uint8
- :protocol
5:
- :uint8
- :src_tos
6:
- :uint8
- :tcp_flags
7:
- :uint16
- :l4_src_port
8:
- :ip4_addr
- :ipv4_src_addr
10:
- 4
- :input_snmp
11:
- :uint16
- :l4_dst_port
12:
- :ip4_addr
- :ipv4_dst_addr
14:
- 4
- :output_snmp
27:
- :ip6_addr
- :ipv6_src_addr
28:
- :ip6_addr
- :ipv6_dst_addr
53:
- :uint8
- :max_ttl
61:
- :uint8
- :direction
136:
- :uint8
- :flow_end_reason
152:
- 8
- :flow_start_msec
153:
- 8
- :flow_start_msec
210:
- :uint8
- :padding_octets
351:
- 8
- :layer2_segment_id

888:
- skip
889:
- skip
890:
- skip
6876:
  888:
  - skip
  889:
  - skip
  890:
  - skip

[jorritfolmer]

Closing this issue. With the most recent commits support for these fields has been added, although they are marked as vmwareUnknown880 etc.

If someone obtains the proper field definitions from VMware, feel free to create a new issue or PR.

[jimmypw]

@jorritfolmer I have done the legwork and figured out what these id's are. Here is my logstash configuration.

filter {
  mutate {
    # remame 6876_890 to ingress interface
    id => "vmware_890"
    rename => { "[ipfix][6876_890]" => "[ipfix][vmwareIngressInterfaceTypeID]" }
  }
  translate {
    # Give it a friendly name.
    # Note: Even if two hosts communicate on the same dv port group the packet
    # will appear to come from the Uplink Port group if they are on different
    # hosts. Rendering this data pretty much useless.
    id => "vmware_890_friendly"
    field => "[ipfix][vmwareIngressInterfaceTypeID]"
    destination => "[ipfix][ingressInterfaceTypeFriendly]"
    dictionary => [
                    "1", "UplinkPG",
                    "2", "DVPG"
                  ]
    fallback => "Unknown"
  }
}

filter {
  mutate {
    # remame 6876_888 to egress interface
    id => "vmware_888"
    rename => { "[ipfix][6876_888]" => "[ipfix][vmwareEgressInterfaceTypeID]" }
  }
  translate {
    # Give it a friendly name.
    # Note: Even if two hosts communicate on the same dv port group the packet
    # will appear to come from the Uplink Port group if they are on different
    # hosts. Rendering this data pretty much useless.
    id => "vmware_888_friendly"
    field => "[ipfix][vmwareEgressInterfaceTypeID]"
    destination => "[ipfix][egressInterfaceTypeFriendly]"
    dictionary => [
                    "1", "UplinkPG",
                    "2", "DVPG"
                  ]
    fallback => "Unknown"
  }
}

filter {
  mutate {
    # remame 6876_889 to observation domain id
    id => "vmware_889"
    rename => { "[ipfix][6876_889]" => "[ipfix][vmwareObservationDomainID]" }
  }
}

[jorritfolmer]

@jimmypw Thanks!