cisco / mercury

Mercury: network metadata capture and analysis
Other
437 stars 75 forks source link

"Null/Loopback" protocol captures not supported in "pmercury" or "mercury" #14

Open IrPgFKS0 opened 3 years ago

IrPgFKS0 commented 3 years ago

When capturing on a tunnel interface (at least on a MAC), the L2 header information is set to Null (more details below)...

https://wiki.wireshark.org/NullLoopback

See example below... "null": { "null.family": "2" }

% tshark -T json -i utun0
Capturing on 'USB 10/100/1000 LAN: en7'
[
  {
    "_index": "packets-2021-05-12",
    "_type": "doc",
    "_score": null,
    "_source": {
      "layers": {
        "frame": {
          "frame.interface_id": "0",
          "frame.interface_id_tree": {
            "frame.interface_name": "en7",
            "frame.interface_description": "USB 10/100/1000 LAN"
          },
          "frame.encap_type": "1",
          "frame.time": "May 12, 2021 11:54:07.695295000 MST",
          "frame.offset_shift": "0.000000000",
          "frame.time_epoch": "1620845647.695295000",
          "frame.time_delta": "0.000000000",
          "frame.time_delta_displayed": "0.000000000",
          "frame.time_relative": "0.000000000",
          "frame.number": "1",
          "frame.len": "1514",
          "frame.cap_len": "1514",
          "frame.marked": "0",
          "frame.ignored": "0",
          "frame.protocols": "null:ip:tcp:data"
        },
        "null": {
          "null.family": "2"
        },
        "ip": {
          "ip.version": "4",
          "ip.hdr_len": "20",
          "ip.dsfield": "0x00000002",
          "ip.dsfield_tree": {
            "ip.dsfield.dscp": "0",
            "ip.dsfield.ecn": "2"
          },
          "ip.len": "1500",
          "ip.id": "0x00006391",
          "ip.flags": "0x00000040",
          "ip.flags_tree": {
            "ip.flags.rb": "0",
            "ip.flags.df": "1",
            "ip.flags.mf": "0"
          },
          ...
davidmcgrew commented 3 years ago

Can you please add the mercury invocation that you used here? thx!

IrPgFKS0 commented 3 years ago

I should clarify the above output is from tshark only (see command above for reference). Once I captured those packets with this cmd tshark -i utun0 -F pcap -w test_null.pcap, I then attempted to read them with the following commands using "pmercury" and "mercury" respectively with no output (attempting both w/ and w/o piping to jq).

python3.8 pmercury -r test_null.pcap -e -w -a -x | jq --tab

*Running in zsh shell hence redirect "> >(jq --tab .)"

./mercury -r test_null.pcap --dns-json --certs-json --metadata > >(jq --tab .)

Note: I also made another capture with this command tshark -i en10 -F pcap -w test_no_null.pcap and both "pmercury" and "mercury" output to jq as expected; "pmercury" output all the TLS packets and "mercury" output all TLS/HTTP and DNS packets (my test mix for both test_null.pcap and test_no_null.pcap).