search

logstash-plugins / logstash-codec-netflow

Apache License 2.0
78 stars 88 forks source link

Problem with zero-filled export packets #146

Closed maxcaines closed 6 years ago

maxcaines commented 6 years ago
jorritfolmer commented 6 years ago

Thanks for the pcap, I'll look into it later.

maxcaines commented 6 years ago

If it's any help, I've verified that Logstash 2.3.2 with version 2.1.0 of the codec works fine on records from our Palo Alto firewall. However, I can't use that version with Logstash 5. I'll see if I can work out what the relevant difference is in the code

maxcaines commented 6 years ago

Hi Jorrit

I may have a solution to this. I enclose updated copies of netflow.rb and util.rb. They pass all the "rspec" tests, and I've been running this version of the plugin for several days receiving Netflow data from our Palo Alto without issues.

The approach I've taken is:

  1. The only way to avoiding reading the zero padding as an extra flowset would be to count the individual flow records and stop when the total reached the value in the flow_records field. But for data flowsets that requires knowledge of the lengths of the flow records, and that information is in the templates. I can't come up with any way to get at that information in declarative code, so I have to create an extra flowset with a zero flowset_length. This is checked for and ignored in netflow.rb.
  2. The zero padding looks like an extra flowset with a zero flowset_length field, I've changed the "assert" in util.rb that checks its validity to allow a zero value.
  3. The padding looks like a template flowset, because the flowset_id is zero. I have changed the template record definition in util.rb to stop reading flow records if flowset_length is zero. I've also added an onlyif field which reads the rest of the packet in that case.

The overall effect of these changes should accommodate the padding that Palo Alto devices add to the Netflow packets, but still leave most of the checks on invalid packets that were added on 27/4/16. The code base I started from is 4.0.1

I hope this is of use

Max

netflow.zip

jorritfolmer commented 6 years ago

Thanks for investigating!

So the root cause appears to be L418 of util.rb This causes the codec to keep reading because the PA packets are all 1442 bytes even though some only contain 1 record with 160 bytes.

Changing :records, :read_until => :eof to :initial_length => :flow_records works to address the root cause, but breaks other stuff. Not sure why yet. It may be this exposed other hacky workarounds.

maxcaines commented 6 years ago

Hi Jorrit

Well, what gave me the original problem was line 420, because the zero bytes in the padding look like a zero length flowset, which it errors on, but yes, if one knew when to stop reading from the packet, the problem wouldn’t happen. The trouble is that the flow_records count is a count of flow records, not flow sets. What the array contains is flow sets, which can contain multiple flow records. So setting initial_length to flow_records can easily break. The only way I can find to fix this is the one I suggested, which means creating a dummy flow set with a zero length field that uses up all the padding, then ignoring it when the flow sets are processed

Regards

Max

Sent from my iPhone

On 23 Jun 2018, at 11:46, Jorrit Folmer notifications@github.com<mailto:notifications@github.com> wrote:

Thanks for investigating!

So the root cause appears to be L418 of util.rbhttps://url6.mailanyone.net/v1/?m=1fWg3K-000Z25-3W&i=57e1b682&c=fB6VlGXesxfnFSMkccEv9ZF5qz6aifpmG8wD2GA88mW-lwWUxoF50y89hdeNWOjS5NCtB42kvtWUD7TZgFIdWADtuQbMIOSdF5GPiJrHsJGJOQ8sMOtnS49qcv9sXQB3eFeAbhhCQseTU4RN13EPjMm1nMEhV9IF1W3bOhvbNNJkw1Cq00wlRiPFeyonk0tdj_R5eXsVijjRJx4hpgVOLKHwpNjmZSUJug_nQDe37lgfkf2wBn6_AtMyB58Tln2nfXzWnWzcTSFoiCl2Rp_wjo4DbzYpB4tyXNcyqlIfhWYhXPhNZ2BAzsgo_FgBky0nO_ij_WerDa2imVrKsRNK2A This causes the codec to keep reading because the PA packets are all 1442 bytes even though some only contain 1 record with 160 bytes.

Changing :records, :read_until => :eof to :initial_length => :flow_records works to address the root cause, but breaks other stuff. Not sure why yet. It may be this exposed other hacky workarounds.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://url6.mailanyone.net/v1/?m=1fWg3K-000Z25-3W&i=57e1b682&c=mWD-ltE3qd9Bxd0OkI-BRerYIqUv586jifNhibYoOKNWv0cb5pq7DuxpA4yJCZUHhF5HIz9HKGNqsGWuK5MWMEKotbguYxhEqsRWTc4jrUNhlPwZ-bJTx60NHvCXhwqlTAbfdptnRvZ_V9zxqTSXAPHur_y_Z1HQqM9ddu2YQdVw-tWa3yOC5E5FYZx2plUxTlZX6qFLeHcIMKR0FxpO1dw2nW_V3uU6XvU44N-NBat-KKwI-trCO2ia2hewvTI2CC3i0frtnlvBzA_knvSIf2M5oJP34SaWH4vXdq86Rd0kmirzcx36vNua6grQhI-f, or mute the threadhttps://url6.mailanyone.net/v1/?m=1fWg3K-000Z25-3W&i=57e1b682&c=MYFNyjxCjVqM0eb4gRlb4uy7K-DiLCTSrtYCMk11esee8LnaEk46qrEMvoBxSP_Y5m0145zHINR8OfbuJc1jaMgL-VyoSOyLlMbkJaBabdgT4XbRYTWsTjZI7i7CKPOwH2XQKE3vuWAoTDU5bvD9CiXHUgzbtRBFQux_W6-vBlXkM7OutomJRHJISysCnyZAouhnCZKS45IN5_3I_IPL7gc3Lg2wRwzOE1WKo4A33IaQtkwHNS-slsRUI3hTC7oTTv6dCxg0mpYpFMv3pBvwjbxfpDik7w1z_0Hr637sDsCeWWWIBCVeZBkk6rFxx_faH4XUlWAAwoENdyUXe6lbqA.

This email has been scanned for spam & viruses. If you believe this email should have been stopped by our filters, click herehttps://portal.mailanyone.net/index.html#/outer/reportspam?token=dXNlcj1tYXguY2FpbmVzQHdsdi5hYy51azt0cz0xNTI5NzUwNzU4O3V1aWQ9NUIyRTI0RTZBNzI3N0Q0RDEyREUxQ0RBNEI3MzZGMTQ7dG9rZW49YTQyNWU4NDNjMGJjNmYxODZkMDVlM2U0YjBkM2NiYmQ3ZjdlODk0ZTs%3D to report it.

jorritfolmer commented 6 years ago

Thanks, good point!

jorritfolmer commented 6 years ago

v4.0.2 includes your fix.

maxcaines commented 6 years ago

Thanks!

Sent from my iPhone

On 24 Jun 2018, at 12:43, Jorrit Folmer notifications@github.com<mailto:notifications@github.com> wrote:

Closed #146https://url6.mailanyone.net/v1/?m=1fX3Qj-00054X-5V&i=57e1b682&c=dcuZz72V1mSQO7qwc_alMJ1oLt4TpR3ItnHgEhdI2RvCtMB_BeV1L6rps2GTxnui4Oe2KgfJwbgZoOe3rYluC1_fcvoQd_lun1r7qjnE03HO2phulzJBqR7EkHln_1C066PSb7i00_A-Zon0Km2igrIIeMhHDfuIkxUm46e92wP3An98slCw3Wia9bHfW_bYTxNgpARnlI7IpMOXBlGwgdI0doVLhhGY_opFgIkcoNuZEESR13J8HC40d4vBZo6y93nJpB-R60uD-j62Yp-FDg.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://url6.mailanyone.net/v1/?m=1fX3Qj-00054X-5V&i=57e1b682&c=M3zk26gzmh6afPWARMfj_P2eqviLAF-sGJnDJ1im4FqSg5Cz_iuqTLJThr2rUq_PuBhPKKrCbXm6oloqEf0bCctqOgoSKJRVcEqNJeSa_KhR17i_mFDJt9CQUIJry5T-oFafi2IpniqI6doXTQIWbrQYMrfvUGdexdCiXCsuuzRer_3rRBiHWMqeRyIAGEyJd-pSkq1fuRJfMOOnlsHXhGpebgxl1JXvkaWx-wlpHIzWO1YqcB_iH2Jun1DimuAVDw4N25-V9yZnN06OX0F1862oRIZotTswbaYbinmFggFa4qGfnOMh0x6L4dGB7Sxn, or mute the threadhttps://url6.mailanyone.net/v1/?m=1fX3Qj-00054X-5V&i=57e1b682&c=njiRgyGdOLGHz5CMPL85q3MKgbCBD4UCRL5mhC2-JywXWX_C_vc3NebT-bRGdHEfqBgcHLQNht_m7fTZJrtDZ4THKjTz1pp7QT0TuEhK3jc_eONLj6hChlms9Ci-mhIxQbdDrHMO5Af3eQ-TrqCoUHSCWVsaH14TMk6TXIO3wOltYfZGcj-ITYUF5Qo_4BG9DiOE3N97u4HsdbSJVltYKiRQQQ8aSBrUQ6T6aW2i-PkfCPdUy0Ef6iMCMx4TmwtEOu74dIAvsrAQCp1g49rhhxm3RqzOAaocPunkNSVENkMK4h-QQHOd-2PGOKlzpiNPr_CeheVkv2a9D6sU1ODTUg.

This email has been scanned for spam & viruses. If you believe this email should have been stopped by our filters, click herehttps://portal.mailanyone.net/index.html#/outer/reportspam?token=dXNlcj1tYXguY2FpbmVzQHdsdi5hYy51azt0cz0xNTI5ODQwNjIyO3V1aWQ9NUIyRjgzRURENTUwRjc2MTkxMjNDRjlDMUNFRTQ3RDc7dG9rZW49NDRmN2ZmNzkxOGNmNDFmMjlkZTk5YWM3Y2ZmYTMzMDE3ZjE0NDNlNTs%3D to report it.