802.11 protocol analysis

[lieranderl]

Before opening a new issue, please make sure you've reviewed the troubleshooting guide: https://github.com/brimdata/brimcap/wiki/Troubleshooting

Describe the bug Failed to upload pcap with wireless data 802.11n https://wiki.wireshark.org/SampleCaptures?action=AttachFile&do=get&target=Http.cap

To Reproduce Try to upload https://wiki.wireshark.org/SampleCaptures?action=AttachFile&do=get&target=Http.cap file to Brim.

Expected behavior The summary data generated should be produced

Screenshots

Desktop Info

• OS: macOS 11.4
• Brim Version: 0.24.0

[philrz]

@jfedotov: Thanks for reporting. As you may be aware, Brim typically relies on some other tools (specifically Zeek and Suricata) to generate summary logs from pcaps. This particular error is coming from Zeek. To study a bit more closely, I reproduced the problem outside of Brim.

The Zeek version bundled with Brim happens to be several revs back (based on v3.2.1), so I tried with a couple other versions. v3.2.4 exhibited the same symptom. The latest GA rev v4.0.2 did not choke on the pcap, but also didn't produce any useful logs, just a weird event saying "unknown_ip_version".

As it turns out, Suricata would choke on this as well, but you didn't see that error because Zeek failed first. I tried with the latest stable release v6.0.2 (once again, even newer than the one bundled with Brim) and it also choked on it:

$ suricata -r Http.pcap 
24/6/2021 -- 08:08:10 - <Notice> - This is Suricata version 6.0.2 RELEASE running in USER mode
24/6/2021 -- 08:08:18 - <Error> - [ERRCODE: SC_ERR_UNIMPLEMENTED(88)] - datalink type 192 not (yet) supported in module PcapFile.
24/6/2021 -- 08:08:18 - <Warning> - [ERRCODE: SC_ERR_PCAP_DISPATCH(20)] - Failed to init pcap file Http.pcap, skipping
24/6/2021 -- 08:08:19 - <Notice> - all 13 packet processing threads, 4 management threads initialized, engine started.
24/6/2021 -- 08:08:19 - <Error> - [ERRCODE: SC_ERR_INVALID_ARGUMENT(13)] - pcap file reader thread failed to initialize
24/6/2021 -- 08:08:19 - <Notice> - Signal Received.  Stopping engine.

I looked on the Wireshark web site where the capture was from and I see it's labeled as an "802.11n capture with PPI encapsulation containing HTTP data." Out of curiosity, is this PPI encapsulation something you're close studying, or were you just seeking out sample captures to try in Brim and this happened to be a random one you tried? I ask because I've seen this class of failures before, as these tools need to have special code to remove encapsulation layers before reconstructing flows for analysis. Since these projects all have their own feature backlogs, some of the more "niche" encapsulation protocols might not get as much attention as the best-known ones like GRE. I looked for open issues in each project regarding PPI encapsulation and I don't see any, so I'd be ready to open them. But I'm guessing the first question they might ask help to establish priority is what the use case is, so I figured I'd establish that with you first.

Even if these tools are not quick to support the format, this seems like it'll be an interesting issue to keep mulling on in the Brimcap context. With its approach, one of the things that Brimcap could make easier is preprocessing captures. So for instance if there were some separate tool that existed (or could easily be written) that Brimcap could invoke to strip off the encapsulation layer before passing along the packet data to packet analyzers (Zeek/Suricata), that might make for an interesting cookbook. Right now Brimcap is still in its infancy so we're not quite yet ready for that kind of experimentation, but I'll plan to give that more thought as time allows.

[lieranderl]

Hi @philrz! Thank you very much for your answer and the tests you made already! I'm interested to use brim with pcaps that our wireless team captures.

Here is the real pcap we want to parse: https://drive.google.com/file/d/1zPh-eQlLpIU-JIq_m9nu3qJRradyu9p7/view?usp=sharing

I tried to open such a pcap with Brim and filtered for anything 802.11 but it seems it doesn’t see the frames. it only looks above layer 3. The radiotap header is a header added by the device sniffing and that does not exist in the actual frame. It’s the sniffing device reporting at what signal strength and how it received that frame from a layer 1 perspective

Then the 802.11 header replaces the ethernet header when the frame is wireless, then “data” is the IP or whatever upper layer but often not decoded as such when the traffic is WPA-encrypted We typically need to create signature and detection scripts based on the 802.11 frame header (it contains info if the frame is a retry for example, it’s not only data, you have management frames when devices try to connect to an AP and so on). It will be great if we can convert such wireless data from pcap to log format using Brim (brimcap).

[philrz]

@jfedotov: Thanks for the additional detail. If your goal is to actually have the logs reflect meaningful analysis of the 802.11 layer, it does seem that Zeek (and therefore, by association, Brimcap/Brim) doesn't have an off-the-shelf solution for this analysis. I did some poking around and ultimately opened up a new Zeek issue zeek/zeek#1641 to ask their team if they have any pointers as to where one might get started with doing this analysis. As I get into in the comments there (and you can read more detail following the links I put in there, then on to the linked PRs), they do seem to have added some kind of "low-level analyzer framework" that's capable of doing the parsing, but I'm not sure how well-documented it is, if there's reference implementations that use it which you could follow, etc. Since you seem to have a solid working knowledge of the protocols involved, it does seem like you'd be in a decent position to put together the appropriate parsing. Hopefully they can provide some tips. Feel free to keep an eye on that issue, and I'll do the same and update here if I see anything.

If you were so inclined as to come up with the parsing, the easier follow-on work would then be to create a custom Brimcap config that points at a Zeek install that can do the parsing. Per that article, Brim could be pointed at that Brimcap config and allow pcaps to be dragged into the app and logs generated from it. We can certainly assist with that if you hit any troubles there.

Even if you're not game to take on the protocol analysis, I'm happy to hold this issue open indefinitely in the event other community users have the same inquiry and maybe one of them would be game to take up the analysis work.

[philrz]

@jfedotov: The Zeek Dev folks responded quickly, and you can see their response here. I'll paste it here for ease of reference and search-ability within the Brimcap repo:

The packet analysis framework was added as part of Zeek 4.0, which allows us to build analyzers for things lower than the IP layer now. It should be fairly simple to implement an 802.11 that either chains into the existing analyzer tree or does its own logging of some sort.

The documentation for packet analysis can be found at https://docs.zeek.org/en/current/frameworks/packet-analysis.html. There's a sample in the documentation, but we have already implemented a number of packet analyzers under the src/packet_analysis directory in the Zeek repo.

If you do decide to dig into this, the Zeek team does have their own public Slack (described here) and are quite responsive if people come with good questions, so that's another resource if you get stuck.

(I don't know if this user is still active, but I'm cc'ing @lealog as well, as I actually discovered an older 802.11-related issue https://github.com/brimdata/brim/issues/570 that had been opened one time and it was closed back then, but I'm happy to keep this one open and see who all is available to pitch in.)

[lealog]

Hi. Thanks for the CC :) I'm also interested to have 802.11 supported on Brim. I'm also following the zeek "low level analyzer" https://github.com/zeek/zeek/issues/248#issuecomment-794149214. Unfortunately, I don't have the skills to implement the plugin for zeek. Hope someone can implement in the future.