search

Graylog2 / graylog-plugin-integrations

A collection of open source Graylog integrations that will be released together.
Other
14 stars 14 forks source link

Errors while ingesting IPFIX #395

Open maniel opened 4 years ago

maniel commented 4 years ago

Research, to see, if this is actually a bug and can be marked as triaged.

Description

Following error is appearing in graylog logs when using IPFIX input with our Stormshield UTM:

2020-02-06 14:03:21,878 ERROR: org.graylog2.shared.buffers.processors.DecodingProcessor - Error processing message RawMessage{id=6f35ad37-48e9-11ea-93fc-0242ac120007, journalOffset=3671397671, codec=ipfix, payloadSize=518, timestamp=2020-02-06T14:03:21.859Z, remoteAddress=/192.168.0.254:7804}
java.lang.IndexOutOfBoundsException: readerIndex(150) + length(8) exceeds writerIndex(152): UnpooledHeapByteBuf(ridx: 150, widx: 152, cap: 152/152)
        at io.netty.buffer.AbstractByteBuf.checkReadableBytes0(AbstractByteBuf.java:1477) ~[graylog.jar:?]
        at io.netty.buffer.AbstractByteBuf.checkReadableBytes(AbstractByteBuf.java:1463) ~[graylog.jar:?]
        at io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:896) ~[graylog.jar:?]
        at org.graylog.integrations.ipfix.IpfixParser.parseDataSet(IpfixParser.java:364) ~[?:?]
        at org.graylog.integrations.ipfix.codecs.IpfixCodec.lambda$decodeMessages$3(IpfixCodec.java:206) ~[?:?]
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[?:1.8.0_242]
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382) ~[?:1.8.0_242]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) ~[?:1.8.0_242]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) ~[?:1.8.0_242]
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_242]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_242]
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566) ~[?:1.8.0_242]
        at org.graylog.integrations.ipfix.codecs.IpfixCodec.decodeMessages(IpfixCodec.java:212) ~[?:?]
        at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:148) ~[graylog.jar:?]
        at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:91) [graylog.jar:?]
        at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:90) [graylog.jar:?]
        at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:47) [graylog.jar:?]
        at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]

Steps To Reproduce

  1. Set up an IPFIX input
  2. Set up appliance to send IPFIX to the input
  3. Look in to graylog logs

Environment

cobaltclaudia commented 4 years ago

Originated from community post https://community.graylog.org/t/ipfix-from-sophos-utm/13815

jalogisch commented 4 years ago

fileshare graylog-oss/ipfix contains a sample pcap

cobaltclaudia commented 4 years ago

@maniel a response has been provided. https://community.graylog.org/t/ipfix-from-sophos-utm/13815/12?u=claudia

Steve4524 commented 4 years ago

I'm seeing a similar error now after adding IPFIX field definition json files in https://community.graylog.org/t/ipfix-from-sophos-utm/13815/14?u=steveu

2020-02-07T18:58:38.657-05:00 ERROR [DecodingProcessor] Unable to decode raw message RawMessage{id=c27dfcf0-4a05-11ea-808a-000c29404c16, journalOffset=394120189, codec=ipfix, payloadSize=1420, timestamp=2020-02-07T23:58:38.655Z, remoteAddress=/192.168.0.1:37828} on input <5e38c5e829ccde06888b7552>.
2020-02-07T18:58:38.657-05:00 ERROR [DecodingProcessor] Error processing message RawMessage{id=c27dfcf0-4a05-11ea-808a-000c29404c16, journalOffset=394120189, codec=ipfix, payloadSize=1420, timestamp=2020-02-07T23:58:38.655Z, remoteAddress=/192.168.0.1:37828}
java.lang.IndexOutOfBoundsException: readerIndex(126) + length(4) exceeds writerIndex(128): UnpooledHeapByteBuf(ridx: 126, widx: 128, cap: 128/128)
        at io.netty.buffer.AbstractByteBuf.checkReadableBytes0(AbstractByteBuf.java:1477) ~[graylog.jar:?]
        at io.netty.buffer.AbstractByteBuf.checkReadableBytes(AbstractByteBuf.java:1463) ~[graylog.jar:?]
        at io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:896) ~[graylog.jar:?]
        at io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:904) ~[graylog.jar:?]
        at org.graylog.integrations.ipfix.IpfixParser.parseDataSet(IpfixParser.java:430) ~[?:?]
        at org.graylog.integrations.ipfix.codecs.IpfixCodec.lambda$decodeMessages$3(IpfixCodec.java:206) ~[?:?]
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[?:1.8.0_242]
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382) ~[?:1.8.0_242]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) ~[?:1.8.0_242]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) ~[?:1.8.0_242]
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_242]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_242]
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566) ~[?:1.8.0_242]
        at org.graylog.integrations.ipfix.codecs.IpfixCodec.decodeMessages(IpfixCodec.java:212) ~[?:?]
        at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:148) ~[graylog.jar:?]
        at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:91) [graylog.jar:?]
        at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:90) [graylog.jar:?]
        at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:47) [graylog.jar:?]
        at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]
2020-02-07T18:58:39.668-05:00 ERROR [DecodingProcessor] Unable to decode raw message RawMessage{id=c3184120-4a05-11ea-808a-000c29404c16, journalOffset=394120190, codec=ipfix, payloadSize=3429, timestamp=2020-02-07T23:58:39.666Z, remoteAddress=/192.168.0.1:37828} on input <5e38c5e829ccde06888b7552>.
2020-02-07T18:58:39.668-05:00 ERROR [DecodingProcessor] Error processing message RawMessage{id=c3184120-4a05-11ea-808a-000c29404c16, journalOffset=394120190, codec=ipfix, payloadSize=3429, timestamp=2020-02-07T23:58:39.666Z, remoteAddress=/192.168.0.1:37828}
java.lang.IndexOutOfBoundsException: readerIndex(630) + length(4) exceeds writerIndex(632): UnpooledHeapByteBuf(ridx: 630, widx: 632, cap: 632/632)
        at io.netty.buffer.AbstractByteBuf.checkReadableBytes0(AbstractByteBuf.java:1477) ~[graylog.jar:?]
        at io.netty.buffer.AbstractByteBuf.checkReadableBytes(AbstractByteBuf.java:1463) ~[graylog.jar:?]
        at io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:896) ~[graylog.jar:?]
        at io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:904) ~[graylog.jar:?]
        at org.graylog.integrations.ipfix.IpfixParser.parseDataSet(IpfixParser.java:430) ~[?:?]
        at org.graylog.integrations.ipfix.codecs.IpfixCodec.lambda$decodeMessages$3(IpfixCodec.java:206) ~[?:?]
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[?:1.8.0_242]
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382) ~[?:1.8.0_242]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) ~[?:1.8.0_242]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) ~[?:1.8.0_242]
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_242]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_242]
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566) ~[?:1.8.0_242]
        at org.graylog.integrations.ipfix.codecs.IpfixCodec.decodeMessages(IpfixCodec.java:212) ~[?:?]
        at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:148) ~[graylog.jar:?]
        at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:91) [graylog.jar:?]
        at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:90) [graylog.jar:?]
        at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:47) [graylog.jar:?]
        at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
        at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]