search

ntop / nProbe

Open source components and extensions for nProbe
http://ntop.org
GNU General Public License v2.0
1.66k stars 44 forks source link

nProbe -> Logstash (ElastiFlow) -> Invalid netflow packet received #446

Closed tbard1 closed 4 years ago

tbard1 commented 4 years ago

Hello there,

Here is my setup :

Pcap (port mirroring) -> nProbe -> logstash (ElastiFlow).

No records are made in Elasticsearch I have some logs: 
[2020-10-12T09:58:38,508][WARN ][logstash.codecs.netflow  ][elastiflow][e05f3254de6bd67a513a705bf17a338b50ed37230f6a5299a9c256d6402a4371] Invalid netflow packet received (value '0' not as expected for obj.records[1].flowset_data.templates[0].option_length)
[2020-10-12T09:58:44,736][WARN ][logstash.codecs.netflow  ][elastiflow][e05f3254de6bd67a513a705bf17a338b50ed37230f6a5299a9c256d6402a4371] Can't (yet) decode flowset id 888 from source id 168, because no template to decode it with has been received. This message will usually go away after 1 minute.
[2020-10-12T09:58:55,118][WARN ][logstash.codecs.netflow  ][elastiflow][e05f3254de6bd67a513a705bf17a338b50ed37230f6a5299a9c256d6402a4371] Can't (yet) decode flowset id 888 from source id 168, because no template to decode it with has been
 received. This message will usually go away after 1 minute.
[2020-10-12T09:58:56,163][WARN ][logstash.codecs.netflow  ][elastiflow][e05f3254de6bd67a513a705bf17a338b50ed37230f6a5299a9c256d6402a4371] Can't (yet) decode flowset id 888 from source id 168, because no template to decode it with has been received. This message will usually go away after 1 minute.
[2020-10-12T09:58:56,166][WARN ][logstash.codecs.netflow  ][elastiflow][e05f3254de6bd67a513a705bf17a338b50ed37230f6a5299a9c256d6402a4371] Can't (yet) decode flowset id 889 from source id 168, because no template to decode it with has been received. This message will usually go away after 1 minute.
[2020-10-12T09:59:03,202][WARN ][logstash.codecs.netflow  ][elastiflow][e05f3254de6bd67a513a705bf17a338b50ed37230f6a5299a9c256d6402a4371] Can't (yet) decode flowset id 888 from source id 168, because no template to decode it with has been received. This message will usually go away after 1 minute.
[2020-10-12T09:59:03,205][WARN ][logstash.codecs.netflow  ][elastiflow][e05f3254de6bd67a513a705bf17a338b50ed37230f6a5299a9c256d6402a4371] Can't (yet) decode flowset id 889 from source id 168, because no template to decode it with has been received. This message will usually go away after 1 minute.
[2020-10-12T09:59:07,362][WARN ][logstash.codecs.netflow  ][elastiflow][e05f3254de6bd67a513a705bf17a338b50ed37230f6a5299a9c256d6402a4371] Can't (yet) decode flowset id 888 from source id 168, because no template to decode it with has been received. This message will usually go away after 1 minute.
[2020-10-12T09:59:10,481][WARN ][logstash.codecs.netflow  ][elastiflow][e05f3254de6bd67a513a705bf17a338b50ed37230f6a5299a9c256d6402a4371] Can't (yet) decode flowset id 888 from source id 168, because no template to decode it with has been received. This message will usually go away after 1 minute.
[2020-10-12T09:59:10,483][WARN ][logstash.codecs.netflow  ][elastiflow][e05f3254de6bd67a513a705bf17a338b50ed37230f6a5299a9c256d6402a4371] Can't (yet) decode flowset id 888 from source id 168, because no template to decode it with has been received. This message will usually go away after 1 minute.
[2020-10-12T09:59:10,488][WARN ][logstash.codecs.netflow  ][elastiflow][e05f3254de6bd67a513a705bf17a338b50ed37230f6a5299a9c256d6402a4371] Can't (yet) decode flowset id 889 from source id 168, because no template to decode it with has been received. This message will usually go away after 1 minute.
[2020-10-12T09:59:11,521][WARN ][logstash.codecs.netflow  ][elastiflow][e05f3254de6bd67a513a705bf17a338b50ed37230f6a5299a9c256d6402a4371] Can't (yet) decode flowset id 888 from source id 168, because no template to decode it with has been received. This message will usually go away after 1 minute.

versions :

Any idea ?

simonemainardi commented 4 years ago

nprobe doesn't talk directly to log stash. It talks to ElasticSearch.

tbard1 commented 4 years ago

hi Simone,

here is my nprobe config : 

-V=9
-T="%SAMPLING_INTERVAL %IN_BYTES %IN_PKTS %IPV4_SRC_ADDR %IPV4_DST_ADDR %IPV4_NEXT_HOP %IPV6_SRC_ADDR %IPV6_DST_ADDR %IPV6_NEXT_HOP %L4_SRC_PORT %L4_DST_PORT %SRC_VLAN %DOT1Q_SRC_VLAN %SRC_TOS %TCP_FLAGS %PROTOCOL %IP_PROTOCOL_VERSION %DIRECTION %FLOW_START_MILLISECONDS %FLOW_END_MILLISECONDS %INPUT_SNMP %OUTPUT_SNMP %IN_SRC_MAC %OUT_DST_MAC %ICMP_TYPE %BIFLOW_DIRECTION %L7_PROTO_NAME"
-U=888
-n=192.168.0.1:2055
-i=eth1
-t=60
-d=15
-l=60
--discard-unknown-flows=mode:0
--biflows-export-policy=0

192.168.0.1:2055 is the IP:port of my Logstash instance

Actually, nProbe exports flows to Logstash : https://www.elastic.co/guide/en/logstash/current/netflow-module.html

simonemainardi commented 4 years ago

oh, I see. So you are using nProbe to export to LogStash via NetFlow.

I think the issue is just that no template has reached LogStash so there's no way for it to decode packets with records.

By the way, it says This message will usually go away after 1 minute. and Indeed, nProbe exports templates periodically. Did you wait for a few minutes of execution?

tbard1 commented 4 years ago

I waited for several days !

simonemainardi commented 4 years ago

by default nProbe exports the template every 10 packets:

[--flows-intra-templ|-o] <num>      | Specify how many flow pkts are exported
                                    | between template exports [default: 10]

So I am not sure what it could be the LogStash error. Did you try and follow up with the LogStash folks?

tbard1 commented 4 years ago

no not for the moment. What I don't understand is this log (first line in the first post) : 

[2020-10-12T09:58:38,508][WARN ][logstash.codecs.netflow  ][elastiflow][e05f3254de6bd67a513a705bf17a338b50ed37230f6a5299a9c256d6402a4371] Invalid netflow packet received (value '0' not as expected for obj.records[1].flowset_data.templates[0].option_length)
novaksam commented 4 years ago

I'm also getting this, in addition to the following when I point it at filebeat instead: Error parsing NetFlow packet of length 224 from 127.0.0.1:2810: error parsing set: invalid template id

My config:

-g=/etc/nprobe/nprobe-rss0.pid
-n=udp://127.0.0.1:2055
--all-collectors
# Incoming - -u
# IF-MIB::ifDescr.16 = STRING: enp1s0f1
# Outgoing - -Q
# IF-MIB::ifDescr.15 = STRING: enp1s0f0
-u=16
-Q=15
--as-list /usr/share/GeoIP/GeoLite2-ASN.mmdb
--city-list /usr/share/GeoIP/GeoLite2-City.mmdb
--local-traffic-direction
-V 9
-o 100
--max-num-flows 2097152
--cpu-affinity 0
--export-thread-affinity 4
--lru-cache-size 524288
--capture-direction 0
-T "%IPV4_SRC_ADDR %IPV4_DST_ADDR %INPUT_SNMP %OUTPUT_SNMP %IN_PKTS %IN_BYTES %OUT_PKTS %OUT_BYTES %FIRST_SWITCHED %LAST_SWITCHED %L4_SRC_PORT %L4_DST_PORT %TCP_FLAGS %IP_PROTOCOL_VERSION %PROTOCOL %SRC_TOS %L7_PROTO_NAME %BITTORRENT_HASH %DIRECTION"
--smart-udp-frag
novaksam commented 4 years ago

Looking at some packet captures, is there a way to turn off the 'options-template' part of the template export?

novaksam commented 4 years ago

My logstash logs lead me to believe they can't handle some or all of the options-template, or perhaps the ones with a 0 length.

value '0' not as expected for obj.records[1].flowset_data.templates[0].option_length

Screen Shot 2020-11-02 at 4 55 57 PM

novaksam commented 4 years ago

and I'm getting this in nprobe-9.2.201102-6992.x86_64 and nprobe-9.3.201102-6991.x86_64. nprobe-9.1.200730-6920.x86_64 worked fine.

novaksam commented 4 years ago

@tbard1 if you go into logstash's install folder \vendor\bundle\jruby\2.5.0\gems\logstash-codec-netflow-4.2.1\lib\logstash\codecs\netflow\util.rb line 398 and change it from uint16 :option_length, :assert => lambda { option_length > 0 } to uint16 :option_length, :assert => lambda { option_length >= 0 }

and restart logstash it should work for now.

tbard1 commented 4 years ago

Confirmed !

Thanks a lot for your support.

novaksam commented 4 years ago

Well that fixes it for now but I wouldn't say it's closed... I think there's an issue in nProbe sending a bad options template.

@simonemainardi what do you think?

tbard1 commented 4 years ago

OK.

FYI, I had the same issue with pmacct as a nProbe replacement. I'll try to reproduce with the Logstash fix.

robcowart commented 3 years ago

I would agree with @novaksam. An option template, and presumably later option data, which only has a scope but no data is unnecessary overhead and wasted collector resources.