CPU usage

[Niek]

I know Python is not the fastest language, but is this expected to use 50%+ CPU on a fast machine?

root@niek:~# ps auwx | grep tcp_fingerprint | head -1
root     28930 53.9  0.6 147268 53904 pts/7    Rl+  09:39   1:46 /root/.local/share/virtualenvs/zardaxt-zSy1Lubt/bin/python tcp_fingerprint.py -i ens3 --classify

Is there any other way to optimize this? I tried PyPy3 but it fails on the pcapy dependency. Any other ideas?

[NikolaiT]

Sorry for the late reply, but if you did already

I tried PyPy3 but it fails on the pcapy dependency

then you know more than I do.

Best idea I currently have: Rewrite the whole thing and use C.

Also: Maybe the scoring algorithm might be a bit slow, since the database lookups occur very often...

[Niek]

Probably the best would be to rewrite in golang using https://pkg.go.dev/github.com/google/gopacket/pcap - that makes is easier to run on Windows too. Are you interested in collaborating on this? The high-level stuff (OS guesser etc) is quite trivial, but decoding the TCP info is more of a challenge.

[Xxx-Bin]

Have you ever tried to set the filter conditions more precisely? According to the code, all data passing through port 443 or port 80 will be parsed. In fact, only TCP SYN is required. TCP ack is not required. It is used for logging.

[Xxx-Bin]

By the way, if your server is actively requested, you'd better add the destination port limit, such as tcp dst port 443 and (tcp[tcpflags] & (tcp-syn) != 0)

[Niek]

Setting a stricter pcap filter definitely helps. I also rewrote zardaxt in Golang, which helps quite a bit as well. But ultimately the best option would be to run a HTTP server, access the local socket directly and get rid of the pcap dependency.

[Xxx-Bin]

I don't quite understand. You mean Golang acts as a web server, accepts all traffic, and only parses the SYN in it?

[Niek]

Yes - spawn a HTTP server in Golang, access the raw TCP socket and parse the incoming traffic through gopacket. That should be much faster than intercepting through pcap.

[Xxx-Bin]

I have a similar idea for this project. Although workman can be used as a web server directly, I didn't get the TLS in it. So I put an nginx directly in front of it. The workload of parsing data is too difficult, come on.

https://github.com/Xxx-Bin/php-ja3#wkm_ja3php

[h4ckitt]

Probably the best would be to rewrite in golang using https://pkg.go.dev/github.com/google/gopacket/pcap - that makes is easier to run on Windows too. Are you interested in collaborating on this? The high-level stuff (OS guesser etc) is quite trivial, but decoding the TCP info is more of a challenge.

Hi, Is This Still On The Table?

I'd Love To Contribute To A Golang Version Of Zardaxt.

[NikolaiT]

Seems to be rewritten in Go already: https://headers.cf/tcp

They could have open sourced it ;)

[Niek]

Seems to be rewritten in Go already: https://headers.cf/tcp

They could have open sourced it ;)

That's me :) I need to improve the code (not using pcap but using the raw TCP socket) and then I can look into releasing it.

[NikolaiT]

I am correctly collecting new classification data, this time the number of samples will be a bit larger.

I am also including the TCP timestamp clock frequency in the next classification data release. It is quite interesting that there seem to be three different TCP Timestamp frequencies out in the wild: 100hz, 250hz and 1000hz (by far the most common). I think the clock frequency should also correlate with the OS type.

What I also observed is that there seems to be sometimes a sudden change in TCP timestamps base value. Any idea?

[valsha]

Seems to be rewritten in Go already: https://headers.cf/tcp They could have open sourced it ;)

That's me :) I need to improve the code (not using pcap but using the raw TCP socket) and then I can look into releasing it.

@Niek is not ready yet? Thanks.

[milksense]

@Niek

Seems to be rewritten in Go already: https://headers.cf/tcp They could have open sourced it ;)

That's me :) I need to improve the code (not using pcap but using the raw TCP socket) and then I can look into releasing it.

@Niek is not ready yet? Thanks.