Pcap timestamp nanoseconds not equal to Wireshark

m2martin commented 4 years ago

Brief description

The timestamp of Pcap or PcapNG files read using rdpcap() are interpreted differently to Wireshark.

Environment

Scapy Version: 2.4.3.dev170 Python: 3.7.5 OS: Kali 2019.4 (Linux kali 5.3.0-kali2-amd64 #1 SMP Debian 5.3.9-3kali1 (2019-11-20) x86_64 GNU/Linux) Wireshark: 3.0.5-1on Linux Cross-checked timestamp on Windows with Wireshark 3.0.6-0 ProfiShark Manager: 2.5.54

How to reproduce

Open a Pcap or PcapNG file of your choice with data in nanosecond resolution. To reproduce the 10³ factor error you need a capture from ProfiTAP's ProfiShark.

Example view in Wireshark:

Plain text timestamp: 2019-11-15 15:00:16.444223380

Code used in Scapy:

pcap = rdpcap('c1.pcapng', 1)
for p in pcap:
    print(p.time)

Actual result

Output displayed:

1573830016444.2234

Adding some formatter:

pcap = rdpcap('c1.pcapng', 1)
for p in pcap:
    print(p.time)
    print(p.time/1000)
    print('%.9f' % (p.time/1000))

Results:

1573830016444.2234
1573830016.4442234
1573830016.444223404

Actual problems:

Timestamp is wrong by factor 10³ (only from ProfiShark files)
Timestamp is wrong by 24 ns

Expected result

I expected to get exactly the same timestamp as seen in Wireshark.

Related information

Changing read_packet in utils.py

p.time = float((tshigh << 32) + tslow) / tsresol

to

p.time = Decimal((tshigh << 32) + tslow) / tsresol

seem to resolve the conversion problem.

Printing the options = block[16:] variable in read_block_idb looks like the following.

Wireshark saved capture:

b'\t\x00\x01\x00\t\x00\x00\x00\x0c\x00\x17\x00Linux 5.3.0-kali2-amd64\x00\x00\x00\x00\x00'

ProfiShark saved capture:

b'\x00\x00\x00\x00'
b'\x00\x00\x00\x00'
b'\x00\x00\x00\x00'
b'\x00\x00\x00\x00'

I think this leads to the default value of tsresol = 1000000.

I don't know if ProfiShark breaks the PcapNG format here or if it's a Scapy issue. Nevertheless, Wireshark does not have any problem to open the ProfiShark files.

gpotter2 commented 4 years ago

About the precision issue: nice catch, see https://github.com/secdev/scapy/pull/2352
about the Profishark issue: it could either means that Scapy has a bug, but more likely that Profishark format is wrong. We follow the pcapng doc: if option 9 (ts_resol) exists, use it. Otherwise,

If this option is not present, a resolution of 10^-6 is assumed (i.e. timestamps have the same resolution of the standard 'libpcap' timestamps).

In the "Profishark saved capture" you posted above, you can clearly see option 9 isn't specified.

Maybe we have a parsing issue before that point. Most likely, wireshark is so smart it understands the packet was wrongly built and fixed it itself.

Could you share a Profishark pcap so that we check which option it is? Thanks

m2martin commented 4 years ago

I digged into the PcapNG definitions and compared the files generated by Wireshark and ProfiShark.

Wireshark creates the following IDB with version 3.0.6 (Win):

01 00 00 00 (Type = IDB)
8C 00 00 00 (Length = 140 octets)
01 00 00 00 (Link Type = Ethernet)
00 00 04 00 (Snap Length = 262144)
02 00 32 00 (Option Code = if_name / Option Length = 50)
(   ...   ) (value if_name)
.. .. 00 00 (end if_name / padded to 32 bits)
03 00 04 00 (Option Code = if_description / Option Length = 4)
.. .. .. .. (value if_description)
09 00 01 00 (Option Code = if_tsresol / Option Length = 1)
06 00 00 00 (if_tsresol = 6, padded to 32 bits)
0C 00 25 00 (Option Code = if_os / Option Length = 37)
(   ...   ) (value if_os)
.. 00 00 00 (end if_os, padded to 32 bits) 
00 00 00 00 (end of options)
8C 00 00 00 (Length = 140 octets)

In comparison, ProfiShark generates the following IDB:

01 00 00 00 (Type = IDB)
20 00 00 00 (Length = 32 octets)
01 00 00 00 (Link Type = Ethernet)
FF FF 00 00 (Snap Length = 65535)
09 00 01 00 (Option Code = if_tsresol / Option Length = 1)
09 00 00 00 (if_tsresol = 9, padded to 32 bits)
00 00 00 00 (end of options)
20 00 00 00 (Length = 32 octets)

As we can see, option if_tsresol is set by ProfiShark.

Let's look into utils.py. The data passed as parameter block to function read_block_idb does not contain the complete IDB as shown above. block does not contain the first 8 octets (block type and block total length) and the last 4 octets (block total length)

Afterwards, options get set as options = block[16:]. This leads to the following cases: Wireshark: if_name option is corrupted - not catched, if_tsresol option follows later - interpreted Profishark: end of options (0x00 0x00 0x00 0x00) is left - if_tsresol is stripped

The right way should be: block[:8] -> Link type / snap length (don't know if you'd like to interpret link type) options = block[8:] -> The option blocks

Additionally, I'd like to mention that multiple interfaces with different if_tsresol can exist in a single PcapNG file. I didn't try to catch how this is currently handled by Scapy.

Hope this explanation saves you some time.

gpotter2 commented 4 years ago

That's great digging, thanks. Could you share a pcap file that triggers this issue?

m2martin commented 4 years ago

Example Capture ProfiShark capture_00035_20191130121306.zip

gpotter2 commented 4 years ago

I think I got it. Can you check https://github.com/secdev/scapy/pull/2352

m2martin commented 4 years ago

Chapeau! Absolutely great work! You also included the 802.3br L2 type fro my test pcap. I commented fix #2352 with just one small change request regarding the preamble representation.

secdev / scapy