Writing PCAP files

[nh2]

dump is designed so it can be easily used as a default callback function by dispatch or loop

Could you make an example? Callback is

PktHdr -> Ptr Word8 -> IO ()

but dump takes a Ptr PktHdr - is this intentional?

... -> Ptr PktHdr -> Ptr Word8 -> IO ()

How am I supposed to write/dump my own PCAP files (e.g. for generating them with QuickCheck)?

(See also http://www.haskell.org/pipermail/haskell-cafe/2007-May/025265.html)

[nh2]

What about a Storable instance? Proposal:

-- From: http://www.haskell.org/haskellwiki/FFI_cook_book
#let alignment t = "%lu", (unsigned long) offsetof(struct { char x__; t (y__); }, y__)

instance Storable PktHdr where
    alignment _ = (#alignment struct pcap_pkthdr)
    sizeOf _ = (#size struct pcap_pkthdr)
    peek = toPktHdr
    poke ptr PktHdr { hdrSeconds = s
                    , hdrUseconds = u
                    , hdrCaptureLength = cl
                    , hdrWireLength = wl } = do
        let ts = (#ptr struct pcap_pkthdr, ts) ptr

        (#poke struct timeval, tv_sec) ts s
        (#poke struct timeval, tv_usec) ts u
        (#poke struct pcap_pkthdr, caplen) ptr cl
        (#poke struct pcap_pkthdr, len) ptr wl

(See also https://gist.github.com/4145774)

The only problem: We might need two storable instances, depending on the endianness (defined in the PCAP header). If so, we could do that with newtypes, or at least provide this storable instance.

What do you think?

[ntc2]

I just implemented your suggestion in #10. I was not aware of the endianness concern and did not consider it.

Could you say more about the endianness concern? Is it related to dumping packets on one machine and then reading them back in on a different machine?

[nh2]

@ntc2

Could you say more about the endianness concern?

I last commented in 2012, so my memory might not be super fresh here, but it's about the byte ordering as defined by the header:

Check https://wiki.wireshark.org/Development/LibpcapFileFormat section Global Header, the bit about

magic_number: used to detect the file format itself and the byte ordering

Most likely I meant that we'd need two Storable instances to be able to write both byte orders.

Also useful (and with more useful links): https://stackoverflow.com/questions/14989608/pcap-files-and-endianness

[ntc2]

@nh2: thanks for the explanation and links. Won't the Storable instance just write whatever byte order the machine it's on supports? Or does "poking" and "peeking" Haskell's C types somehow not respect the underlying byte order? I would expect the machine byte order to be used, but this is just intuition and I don't have any docs supporting this. I'm an FFI newb, so I could be way off here.

Here is the new "poke": https://github.com/GaloisInc/pcap/blob/f115d376a481578e9ef13365db2b43e5de8e9c7b/Network/Pcap/Base.hsc#L555

And here is the existing "peek": https://github.com/GaloisInc/pcap/blob/f115d376a481578e9ef13365db2b43e5de8e9c7b/Network/Pcap/Base.hsc#L539

I only have access to Intel machines, so I don't know how to test this, but my expectation is that we'll get the same behavior as the underlying libpcap: either that supports creating dump files on one machine and then reading them on another machine with different endianness, or it doesn't, but in either case it's not really our problem as wrappers of libpcap. If the dump files aren't portable, it would make sense to document this of course.

[nh2]

@ntc2:

Won't the Storable instance just write whatever byte order the machine it's on supports?

I also don't havea big endian machine (well probably I have given that newer ARMs are bi-endian, but I don't know how to mess with that from Haskell), but I think you're right here.

I only have access to Intel machines, so I don't know how to test this

A VM running in QEMU should be able to virtualise OSs running under different endianness.

my expectation is that we'll get the same behavior as the underlying libpcap

Generally yes, but I think what I had in mind was that while all pcap_* functions handle endianness (reading) for you, your peek doesn't.

I suspect that in order to read a pcap file that was created on a machine with different endianness, your toPktHdr :: Ptr PktHdr -> IO PktHdr needs to inspect isSwapped (pcap_is_swapped) when reading CLongs.

I did a quick search, seems like the storable-endian package does that / shows how to do it (but only for Int64, not for CLong, which I guess should be adde).

For fromPktHdr :: Ptr PktHdr -> PktHdr -> IO () it's not clear to me if this is needed -- typically it's OK for machines to be able to write only their own endianness (but read other machines' endiannes). So I think you'd have to do that only if you can open an existing pcap file that's in different endianness than your machine, and mutably change some of the headers. Then it would make sense to maintain the endianness of the file, instead of having some flipped endiannness somewhere in the middle (since as far as I can tell, pcap_is_swapped applies to the entire file).

[ntc2]

@nh2: I looked into this a little more and I think everything is fine.

• I can read both big-endian and little-endian .pcap capture files with the current code: I downloaded a big-endian capture file mentioned in the SO answer you referenced above. When I read it I get sensible PktHdrs, with dates in 2012 and capture lengths of 98 bytes. I can also read little-endian capture files I create on my machine. I can verify the the endianness of the capture files using file <capture file>. I expect that the reason we can already read big-endian capture files on a little-endian machine is that libpcap already handles the conversion for us behind the scenes.

• The libpcap docs say that the headers are always written in host byte order, so we don't need to worry about endianness when creating pcap_pkthdr structs in fromPktHdr:

 * We use the "receiver-makes-right" approach to byte order,
 * because time is at a premium when we are writing the file.
 * In other words, the pcap_file_header and pcap_pkthdr,
 * records are written in host byte order.

[nh2]

OK, sounds good then! Thanks for doing this!