Streaming parser does return refill the buffer and returns Incomplete

[lbckmnn]

Hey, thanks for the library. I tried using the sample code from the documentation on a larger pcapng file but the refilling of the buffer doesn't seem to work properly.

See the following example: pcapng file: https://wiki.wireshark.org/uploads/__moin_import__/attachments/SampleCaptures/iperf3-udp.pcapng.gz

minimal example

``` rust use pcap_parser::data::{get_packetdata, PacketData}; use pcap_parser::traits::PcapReaderIterator; use pcap_parser::*; use std::env; use std::fs::File; fn main() { let args: Vec<_> = env::args().collect(); let file = File::open(args[1].clone()).unwrap(); let mut reader = PcapNGReader::new(65536, file).expect("PcapNGReader"); let mut num_packets = 0; loop { match reader.next() { Ok((offset, block)) => { match block { PcapBlockOwned::NG(Block::EnhancedPacket(ref epb)) => { num_packets += 1; } PcapBlockOwned::NG(Block::SimplePacket(ref spb)) => { num_packets += 1; } PcapBlockOwned::NG(_) => { eprintln!("unsupported block"); } PcapBlockOwned::Legacy(_) | PcapBlockOwned::LegacyHeader(_) => unreachable!(), } reader.consume(offset); println!("{}", reader.position()); } Err(PcapError::Eof) => break, Err(PcapError::Incomplete) => { eprintln!("Could not read complete data block."); eprintln!("Hint: the reader buffer size may be too small, or the input file nay be truncated."); break; } Err(e) => panic!("error while reading: {:?}", e), } } println!("num packets: {}", num_packets); } ```

output

``` ./minimal-example ./Downloads/iperf3-udp.pcapng unsupported block 228 unsupported block 300 408 516 640 804 912 1020 1120 1256 1356 1456 1556 1660 1760 1992 2092 2192 2292 2400 2508 2632 2796 2876 2976 3056 3156 4680 4780 4880 6404 7928 9452 10976 12500 14024 15548 17072 18596 20120 21644 23168 24692 26216 27740 29264 30788 32312 0 1524 3048 4572 6096 7620 9144 10668 12192 13716 15240 16764 18288 19812 21336 22860 24384 25908 27432 28956 30480 Could not read complete data block. Hint: the reader buffer size may be too small, or the input file nay be truncated. num packets: 67 ```

The reader position indicates that the buffer shifts properly when half the capacity is reached but when about the buffersize is parsed, an PcapError::Incomplete is returned. The documentation says the consume method will refill but that does not seem to be the case. If I double the buffer size I can parse about double the amount of the packets before I get an PcapError::Incomplete.

[lbckmnn]

Hey,

I just realized I need to call reader.refill() myself and found another example which actually calls the refill method when next() returns an Incomplete so this is just a documentation issue.

I found at least two places: 1.) The example here contains no refill. This is the example which is used: https://docs.rs/pcap-parser/latest/pcap_parser/struct.PcapNGReader.html#example

2.) One sentence which is probably outdated?: https://docs.rs/pcap-parser/latest/pcap_parser/traits/trait.PcapReaderIterator.html "consume takes care of refilling the buffer if required."

[evanrichter]

I just got bit by this also

[chifflier]

Hi, You're right on both points, I will fix this as soon as possible.

Better code/example is available in the pcap-analyzer project: https://github.com/rusticata/pcap-analyzer/blob/f91f83ab03ea27a512873131abbc26130b8d0e85/libpcap-tools/src/block_engine.rs#L48-L101

Note on the referenced code: when an incomplete block is encountered, you need to call reader.refill(), but you have to be careful not to create an infinite loop: if the buffer is too small, the next block may be incomplete even after refill, or if the incomplete block is at end of file. The current design is like that because it allows adjusting the buffer size dynamically

[evanrichter]

thanks for the updated docs!!