Npcap timestamp discrepancy

matman13 commented 4 days ago

We have a piece of software that uses the Npcap OEM version under the hood to collect packets so that our application can dissect the payloads. We use Wireshark as a kind of benchmark to make sure that we aren't dropping packets that Wireshark sees, that timestamps are equivalent for the same packets, etc.

We have a 20-minute data run where we replay our log files to a system that is running our software and Wireshark v4.2.8 x64 at the same time. If we use Npcap 1.60, we see no timestamp discrepancy between our timestamps and Wireshark's timestamps. Ever since 1.60, however, the timestamps are slightly off. With Npcap 1.80, we are seeing up a 2-millisecond discrepancy between our software and Wireshark. The only difference in the test harness is the Npcap version.

Is there something that changed in Npcap 1.60 that might explain this discrepancy? Because timestamps are sensitive in our domain, we are continuing to deploy our software with Npcap 1.60, but we'd like to stay current. This time discrepancy is preventing that.

We are testing on Windows 10/11.

Any hints explaining why we might be seeing this discrepancy are greatly appreciated.

guyharris commented 4 days ago

How is your software getting timestamps? GetSystemTimeAsFileTime() or GetSystemTimePreciseAsFileTime() if it's in user mode, KeQuerySystemTime() or KeQuerySystemTimePrecise() if it's in kernel mode, or something else?

matman13 commented 4 days ago

We're converting the timeval struct in the pcap_pkthdr struct to the timestamp we use:

var tv_sec = pkthdr.tv.tv_sec * 10_000_000; // ticks per second
var tv_usec = pkthdr.tv.tv_used * 10; // ticks per microsecond
var timestamp = NodaTime.Instant.FromUnixTimeTicks(tv_sec + tv_usec);

I see your answer here. We aren't setting the TimestampMode registry key for Npcap like we once did for WinPcap. Should we be doing that so that Wireshark and our software use the same time under the hood? Or should we using the Pcap API to set the timestamp mode when we create the Npcap device? If so, what is the incantation that will give us the most accurate timestamps on Windows that stay in sync with real wall-clock time?

One more question. If our software and Wireshark are run concurrently and both create Npcap devices, are these two distinct Npcap instances under the hood, or are they shared? I would have thought they'd be distinct, so I'm surprised that our test group is claiming that timestamps for the same packets are identical between our software and Wireshark using Npcap v1.60. That suggests a shared device to me. If the devices are distinct, i.e., a device for our software and a separate device for Wireshark would have timestamps that are very close, but I wouldn't expect them to line up every time. Any thoughts on this?

@guyharris I really appreciate your help on this. You have saved me countless hours over the years with your expertise in this arena. If I don't talk to you again before tomorrow, have a Happy Thanksgiving!

guyharris commented 3 days ago

If so, what is the incantation that will give us the most accurate timestamps on Windows that stay in sync with real wall-clock time?

If by "real wall-clock time" you mean "time as provided by the operating system", then the time stamp types that are specified as being in sync with the OS clock are PCAP_TSTAMP_HOST_LOWPREC/"host_lowprec" and PCAP_TSTAMP_HOST_HIPREC/"host_hiprec"; the latter is the one with the highest precision.

The default time stamp for libpcap is PCAP_TSTAMP_HOST/"host", which is not guaranteed either to be high-precision or synchronized with the OS clock (but also isn't guaranteed to be low-precision or unsynchronized with the OS clock).

In WinPcap and Npcap, the TimestampMode Registry key specifies what PCAP_TSTAMP_HOST is. As per my answer, 4 gives you PCAP_TSTAMP_HOST_HIPREC by default.

But Wireshark, by default, doesn't set the timestamp type; it only does so if you specify a --time-stamp-type option on the command line - there's no way to do it through the GUI. So, if your software that's receiving the packets also isn't calling pcap_set_tstamp_type(), I'm not sure why the time stamps would differ between Wireshark and your app running on the same machine.

If our software and Wireshark are run concurrently and both create Npcap devices, are these two distinct Npcap instances under the hood, or are they shared?

One driver, separate devices, each of which can have a different time stamp type, although they all default to the same type.

matman13 commented 3 days ago

Thank you.

So the TimestampMode registry key sets the default for all Npcap devices, while calling pcap_set_tstamp_type() allows the timestamp mechanism to vary with each Npcap device. In other words, the registry key sets the master default value, which may or may not match libpcap's PCAP_TSTAMP_HOST default value, and the API call allows the default to be changed per Npcap device. Am I interpreting that correctly?

Last question. Do you have any idea of how much more expensive PCAP_TSTAMP_HOST_HIPREC is over PCAP_TSTAMP_HOST_LOWPREC? I'll test it, but was curious if you have any ballpark notion.

Thanks again. I really appreciate it.

guyharris commented 3 days ago

So the TimestampMode registry key sets the default for all Npcap devices, while calling pcap_set_tstamp_type() allows the timestamp mechanism to vary with each Npcap device. In other words, the registry key sets the master default value, which may or may not match libpcap's PCAP_TSTAMP_HOST default value, and the API call allows the default to be changed per Npcap device. Am I interpreting that correctly?

Mostly.

The behavior is:

the registry key sets the master default value;
a newly-opened device gets the default value - the open happens at pcap_activate() time;
the pcap_set_tstamp_type() API call, made before pcap_activate(), allows the device's value to be changed per Npcap device - but if its set to PCAP_TSTAMP_HOST, it's left as the default value.

Last question. Do you have any idea of how much more expensive PCAP_TSTAMP_HOST_HIPREC is over PCAP_TSTAMP_HOST_LOWPREC? I'll test it, but was curious if you have any ballpark notion.

From looking at the Npcap driver, if any Npcap device is using PCAP_TSTAMP_HOST_HIPREC, any device with either PCAP_TSTAMP_HOST_HIPREC or PCAP_TSTAMP_HOST_LOWPREC uses a time stamp from KeQuerySystemTimePrecise(), otherwise any device with PCAP_TSTAMP_HOST_LOWPREC uses a time stamp from KeQuerySystemTime().

According to Microsoft's documentation for KeQuerySystemTimePrecise():

On some hardware platforms, a KeQuerySystemTimePrecise call might be slower than a KeQuerySystemTime call. The reason is that KeQuerySystemTimePrecise reads the performance counter, which can introduce an additional delay. For more information, see KeQueryPerformanceCounter.

The KeQueryPerformanceCounter page doesn't provide much more information; it links to their Acquiring high-resolution time stamps page, which may provide more information than you want. :-)

Thanks again. I really appreciate it.

You're welcome - and I apologize for not thanking you for your Thanksgiving wishes, and wishing you a happy THanksgiving in turn, earlier.

nmap / npcap

Npcap timestamp discrepancy #767