Mixed Data Link Types in Unified2 File results in partly unreadable Pcap output

[maxtors]

When a unified2 file that barnyard2 reads, contains Unified2PacketRecords with varying datalink types, the resulting pcap output (via tcpdump log ouput plugin) is partly unreadable.

Example: First packet that is registered is with RawIPv4 datalink type. The total Pcap file before rotating is 50% (just an example) of LINKTYPE_IPV4[1] datalink headers and 50% LINKTYPE_ETHERNET[1]. This will result in tcpdump (and similar tools) not being able to read a half of the pcap, since the Pcap header format has a datalink field[2] being set to only one of the contained datalink types.

My suggestion is to add an option to barnyard2 that gives the user the oppertunity to "rotate to a new pcap file if datalink type changes from the current pcap file i am writing to".

Yes, i know that it is possible to output one packet per alert, but in some environments, with regards to the amount of alerts and the network speeds that are being handled, this is not a viable option.

Yes, the product / service that is ouputting / creating the unified2 files could conform to a single datalink type in the unified2 records. But as an example, one could be listening on a SPAN where the input will have flows / sessions that are tunneled (wich means the resulting decoded readable data cannot be re-combined with the etherframe without "not altering the packet", aka creating a fake packet just to make by2 work is not a valid option), or from different datalinks.

[1] http://www.tcpdump.org/linktypes.html [2] https://wiki.wireshark.org/Development/LibpcapFileFormat

[pauloangelo]

Maxtors,

There is some commented code (lines 373 to 385) in spo_log_tcpdump.c (URL below). Try to uncomment, compile barnyard2 and see if it helps. Don't forget to give us a feedback.

https://github.com/firnsy/barnyard2/blob/master/src/output-plugins/spo_log_tcpdump.c#L373

regards,

PA

[maxtors]

Thanks for the reply. I took alook at the code and it seems to be just what im after. Ive uncommented the code, and added a logmessage to visualize when this switch occures. The issue im having now is that the pcap is not rotating. Im still stuck with one pcap file.

[maxtors]

In the function TcpdumpRollLogFile. It checks if the current epoch second is <= the data->lastTime. So if a rotate is called for within the same second, then it will not happen. When running in batch mode on a fast system / small log, this will probably occur, and when running in continual mode with changing data link type within the same second this will probably also happen.

Maybe there should be added another suffix, f.eks .1, .2, etc (tcpdump.log.1442401544.1, tcpdump.log.1442401544.2)?

[pauloangelo]

Maxtors,

The problem is theoretically complicated. Suppose that you have 1k packets per second alternating the link type. Maybe you should have a tcpdump.log to each link type, for example tcpdump.log.eth.1442401544 and tcpdump.log.ipv4.1442401562 ?!

[maxtors]

Yes, there could be that amount many packets with alternating linktypes. This would stress the system, but probably less (cant realy see a real life scenario with 50% split of alternating linktypes) than outputting one pcap per alert that is found (which is allready an option).

Even though this might stress the system, it is better to give people the option, rather than loosing insight into valuable packetdata.

One per linktype could work, i see no issues with that. Would possibly be helpfull for some people by having the opertunity to quickly(er) find the PCAP if they know, and then can filter on, the linktype.

[maxtors]

I've taken a closer look at this scenario that i am experiencing. Around 4% (+- 2%) of the packets in my tcpdump.log files are of a different link type than what the header of the pcap file indicates. This is output from Suricata (single sensor, one link, same datalink on input, so its a suricata based issue and not barnyard2).

This is very specific to my scenario. It seems wrong to do a "swtich to new pcap at every new observed link type", hence it would be better to do "per-link-type-pcap" type of output as you mentioned.