Streaming parser example: `error while reading: NomError(Eof)`

[kokostek]

I was trying to use streaming parser example to count number of blocks in a very large .pcapng file and got this error:

error while reading: NomError(Eof)

I suppose this error should be treated like PcapError::Incomplete case. Not sure though.

Suggesting this change to the example code:

use pcap_parser::*;
use pcap_parser::traits::PcapReaderIterator;
use std::fs::File;
use nom::error::ErrorKind;

let file = File::open(path).unwrap();
let mut num_blocks = 0;
let mut reader = PcapNGReader::new(65536, file).expect("PcapNGReader");
loop {
    match reader.next() {
        Ok((offset, _block)) => {
            println!("got new block");
            num_blocks += 1;
            reader.consume(offset);
        },
        Err(PcapError::Eof) => break,
        Err(PcapError::Incomplete) | Err(PcapError::NomError(ErrorKind::Eof)) => {
            reader.refill().unwrap();
        },
        Err(e) => panic!("error while reading: {:?}", e),
    }
}
println!("num_blocks: {}", num_blocks);

[chifflier]

Hi, Sorry for the delay in the response. Does this mean that Eof was returned but the file was not entirely parsed, and you had to ignore the error?

[kokostek]

Hi. Yes, Eof was returned when parser reached the end of circular buffer. But rather than ignore this, I had to refill(). Similar to the Incomplete case.

[chifflier]

That would be a bug, but it will be hard to investigate without more information. Do you have a way to instrument pcap-parser (maybe print the buffer state / length if the specific condition is reached), or (privately) share some pcap?

[kokostek]

Yes, I can send you the pcap. But I think you can reproduce this on any pcap larger than 65536 bytes. Or maybe reduce size of buffer and try even smaller pcap. Either way, post your contact info and I will share mine some time later.

[chifflier]

This is not related only the file size (I used the crate on files > 150GB). I suspect a specific condition related to the buffer size, and structs alignment (for ex, a header ends exactly at the end of the buffer and there is no content, or something like that)

[inferiorhumanorgans]

BTW I'm seeing the same thing with a 650MB compressed pcapng file. In this case I'm creating a bufreader with the flate2 crate over the file and passing that to the pcap-parser crate. I can duplicate this with smaller files as well. For instance:

https://wiki.wireshark.org/SampleCaptures#Apache_Cassandra

[chifflier]

Thank you for the link, I was able to reproduce the problem. I'm investigating.

[chifflier]

Found the issue, this was caused by using nom complete parsers instead of streaming. I'm releasing a new patch release (0.11.1)