Denial of service by sending invalid netflow packets to logstash-codec-netflow

[jorritfolmer]

Based on issue #17, I've been fuzzing the netflow codec with ProxyFuzz.py and a Netflow stream to see if crashes on other invalid input as well. ProxyFuzz.py does random bitflipping, adding data etc. So far I've found it crashes like this:

Exception in inputworker 
{"exception"=>#<IOError: data truncated>, 
 "backtrace"=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/bindata-2.1.0/lib/bindata/io.rb:83:in `readbytes'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/bindata-2.1.0/lib/bindata/string.rb:110:in `read_and_return_value'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/bindata-2.1.0/lib/bindata/base_primitive.rb:124:in `do_read'", "(eval):2:in `do_read'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/bindata-2.1.0/lib/bindata/struct.rb:131:in `do_read'", "org/jruby/RubyArray.java:1613:in `each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/bindata-2.1.0/lib/bindata/struct.rb:131:in `do_read'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/bindata-2.1.0/lib/bindata/array.rb:301:in `do_read'", "org/jruby/RubyKernel.java:1511:in `loop'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/bindata-2.1.0/lib/bindata/array.rb:299:in `do_read'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/bindata-2.1.0/lib/bindata/struct.rb:131:in `do_read'", "org/jruby/RubyArray.java:1613:in `each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/bindata-2.1.0/lib/bindata/struct.rb:131:in `do_read'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/bindata-2.1.0/lib/bindata/base.rb:146:in `read'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/bindata-2.1.0/lib/bindata/base.rb:23:in `read'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-codec-netflow-1.0.0/lib/logstash/codecs/netflow.rb:136:in `decode'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-1.0.0/lib/logstash/inputs/udp.rb:95:in `inputworker'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-1.0.0/lib/logstash/inputs/udp.rb:74:in `udp_listener'"],
 :level=>:error
}

on this input:

Internet Protocol Version 4, Src: 172.16.32.201 (172.16.32.201), Dst: 172.16.32.202 (172.16.32.202)
    Version: 4
    Header Length: 20 bytes
    Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00: Not-ECT (Not ECN-Capable Transport))
    Total Length: 1424
    Identification: 0xbe66 (48742)
    Flags: 0x02 (Don't Fragment)
    Fragment offset: 0
    Time to live: 64
    Protocol: UDP (17)
    Header checksum: 0xdd42 [correct]
    Source: 172.16.32.201 (172.16.32.201)
    Destination: 172.16.32.202 (172.16.32.202)
    [Source GeoIP: Unknown]
    [Destination GeoIP: Unknown]
User Datagram Protocol, Src Port: 53906 (53906), Dst Port: iop (2055)
    Source Port: 53906 (53906)
    Destination Port: iop (2055)
    Length: 1404
    Checksum: 0x17cc [correct]
    [Stream index: 0]
Cisco NetFlow/IPFIX
    Version: 10
    Length: 1396
    Timestamp: Aug 27, 2015 16:04:56.000000000 CEST
    FlowSequence: 4997093
    Observation Domain Id: 0
    Set 1
        FlowSet Id: (Data) (269)
        FlowSet Length: 100
        Data (96 bytes), no template found
    Set 2
        FlowSet Id: (Data) (268)
        FlowSet Length: 504
        Data (500 bytes), no template found

[jorritfolmer]

To reproduce this issue, it is proably easiest to use this .pcap file. Use this to create the packet_IOError_1_1pkt.pcap so you can replay it with tcprelay or bittwist

cat << EOF |base64 -d |gunzip -dc > packet_IOError_1_1pkt.pcap
H4sICL6d4VUAA2Z1enpfSU9FcnJvcl9kYXRhX3RydWNhdGVkXzFwYWNrZXQucGNhcAC7cnjTQiYG
FgYYuMPKwMAIpMNmPAwN9WRguAXkvwJiBh7NnTH9QLKgjZODwZWB9U5tiwKDg6DtzTUCCieB+NTh
dxzsoo1pcxm4WCtC70uyMvjYO46CUTAKRsEoGLwAANS0vAcCBgAA
EOF

[jorritfolmer]

Adding IOError to the rescue, like this:

-  rescue BinData::ValidityError => e
+  rescue BinData::ValidityError, IOError => e

seems to help. Instead of crashing it prints a warning and continues. On another stream of invalid netflow v5 packets this looks like:

Ignoring Netflow version v45573 {:level=>:warn}
Ignoring Netflow version v69 {:level=>:warn}
Invalid Netflow packet received (End of file reached) {:level=>:warn}
Invalid Netflow packet received (End of file reached) {:level=>:warn}
Ignoring Netflow version v22 {:level=>:warn}
Ignoring Netflow version v58885 {:level=>:warn}
Ignoring Netflow version v131 {:level=>:warn}
Invalid Netflow packet received (End of file reached) {:level=>:warn}
Ignoring Netflow version v137 {:level=>:warn}
Ignoring Netflow version v47 {:level=>:warn}
Invalid Netflow packet received (End of file reached) {:level=>:warn}
Ignoring Netflow version v140 {:level=>:warn}
Ignoring Netflow version v68 {:level=>:warn}
Invalid Netflow packet received (End of file reached) {:level=>:warn}
Invalid Netflow packet received (data truncated) {:level=>:warn}
Invalid Netflow packet received (End of file reached) {:level=>:warn}
Invalid Netflow packet received (data truncated) {:level=>:warn}
Invalid Netflow packet received (End of file reached) {:level=>:warn}
Invalid Netflow packet received (End of file reached) {:level=>:warn}

However, then this happens:

Exception in thread "<udp.0" java.lang.UnsupportedOperationException
    at java.lang.Thread.stop(Thread.java:869)
    at org.jruby.RubyThread.exceptionRaised(RubyThread.java:1221)
    at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:112)
    at java.lang.Thread.run(Thread.java:745)
Ignoring Netflow version v115 {:level=>:warn}
Invalid Netflow packet received (End of file reached) {:level=>:warn}
Invalid Netflow packet received (End of file reached) {:level=>:warn}
Ignoring Netflow version v58 {:level=>:warn}
Exception in thread "<udp.1" java.lang.UnsupportedOperationException
    at java.lang.Thread.stop(Thread.java:869)
    at org.jruby.RubyThread.exceptionRaised(RubyThread.java:1221)
    at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:112)
    at java.lang.Thread.run(Thread.java:745)

So perhaps rescue is not enough or we've hit something else? Any ideas?

[jorritfolmer]

The stream of netflow v5 packets that reliably leads to the java.lang.UnsupportedOperationException is available here: http://filebin.ca/2DuuXqxcXNMW/fuzz_UnsupportedOperationException.pcap

[bodgit]

If you add something like this to the rescue block:

yield LogStash::Event.new("message" => payload, "tags" => ["_netflowparsefailure"])

Does it make the exceptions go away? I wonder if codecs currently require you to always yield an event from some input, the few codecs I've checked do this

[jorritfolmer]

No, unfortunately still exceptions:

Exception in thread "<udp.0" java.lang.UnsupportedOperationException
    at java.lang.Thread.stop(Thread.java:869)
    at org.jruby.RubyThread.exceptionRaised(RubyThread.java:1221)
    at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:112)
    at java.lang.Thread.run(Thread.java:745)
Invalid Netflow packet received (End of file reached) {:level=>:warn}
Ignoring Netflow version v150 {:level=>:warn}
Invalid Netflow packet received (End of file reached) {:level=>:warn}
Exception in thread "<udp.1" java.lang.UnsupportedOperationException
    at java.lang.Thread.stop(Thread.java:869)
    at org.jruby.RubyThread.exceptionRaised(RubyThread.java:1221)
    at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:112)
    at java.lang.Thread.run(Thread.java:745)

[jorritfolmer]

This appears to be the offending packet from the provided pcap earlier:

Internet Protocol Version 4, Src: 172.16.32.201 (172.16.32.201), Dst: 172.16.32.202 (172.16.32.202)
User Datagram Protocol, Src Port: 53589 (53589), Dst Port: iop (2055)
    Source Port: 53589 (53589)
    Destination Port: iop (2055)
    Length: 1472
    Checksum: 0x6c19 [correct]
    [Stream index: 0]
Cisco NetFlow/IPFIX
    Version: 5
    Count: 55582
    SysUptime: 213.213263000 seconds
    Timestamp: Apr 17, 2015 08:32:57.000000576 CEST
    FlowSequence: 2389
    EngineType: RP (0)
    EngineId: 67
    00.. .... .... .... = SamplingMode: No sampling mode configured (0)
    ..00 0000 0000 0000 = SampleRate: 0
    pdu 1/55582
    pdu 2/55582
    pdu 3/55582
    pdu 4/55582
    pdu 5/55582
    pdu 6/55582
    pdu 7/55582
    pdu 8/55582
    pdu 9/55582
    pdu 10/55582
    pdu 11/55582
    pdu 12/55582
    pdu 13/55582
    pdu 14/55582
    pdu 15/55582
    pdu 16/55582
    pdu 17/55582
    pdu 18/55582
    pdu 19/55582
    pdu 20/55582
    pdu 21/55582
    pdu 22/55582
    pdu 23/55582
    pdu 24/55582
    pdu 25/55582
    pdu 26/55582
    pdu 27/55582
    pdu 28/55582
    pdu 29/55582
    pdu 30/55582

[jorritfolmer]

Closed with merge of #22