search

FoxIO-LLC / ja4

JA4+ is a suite of network fingerprinting standards
https://foxio.io
Other
903 stars 78 forks source link

rust crashes against Arkime pcaps #51

Closed awick closed 8 months ago

awick commented 8 months ago

Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it. Run with RUST_BACKTRACE=full to include source snippets.

Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it. Run with RUST_BACKTRACE=full to include source snippets.

Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it. Run with RUST_BACKTRACE=full to include source snippets.

Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it. Run with RUST_BACKTRACE=full to include source snippets.

Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it. Run with RUST_BACKTRACE=full to include source snippets.

vvv commented 8 months ago

@awick Thanks for the bug reports!

Could you please show the output of tshark --version and ja4 --version?

The Rust app (ja4) in this repo parses the output of tshark -Tpdml. The schema of this output may change between major tshark versions. If your version of tshark is older than 4.0.6, ja4 should have refused to start.

I ran the latest release of the Rust app (ja4 0.16.2) against the mentioned pcap files (taking them from arkime/arkime), but could not reproduce the problem:

❯ tshark --version | head -1
TShark (Wireshark) 4.2.1 (v4.2.1-0-gcfe37f471da9).

❯ ja4 --version
ja4 0.16.2

❯ ja4 CVE-2018-6794.pcap
- stream: 0
  transport: tcp
  src: 192.168.235.1
  dst: 192.168.235.136
  src_port: 53649
  dst_port: 8089
  ja4l_c: 0_128
  ja4l_s: 2219_255
  http:
  - ja4h: ge11nn07ruru_6cd0fb54989b_000000000000_000000000000
- stream: 1
  transport: tcp
  src: 192.168.235.1
  dst: 192.168.235.136
  src_port: 53656
  dst_port: 8089
  ja4l_c: 0_128
  ja4l_s: 1513_255
  http:
  - ja4h: ge11nr06ruru_cc6ec9a91856_000000000000_000000000000
- stream: 2
  transport: tcp
  src: 192.168.235.1
  dst: 192.168.235.136
  src_port: 53648
  dst_port: 8089
  ja4l_c: 0_128
  ja4l_s: 1948_255

❯ ja4 gre-erspan-vxlan.pcap
- stream: 0
  transport: tcp
  src: 100.20.9.2
  dst: 100.20.9.1
  src_port: 65174
  dst_port: 80
  ja4l_c: 953_64
  ja4l_s: 997_64

❯ ja4 nflog.pcap
- stream: 0
  transport: tcp
  src: 10.10.10.10
  dst: 52.43.228.156
  src_port: 54064
  dst_port: 80
  ja4l_c: 0_64
  ja4l_s: 5_226
  http:
  - ja4h: ge11nn030000_32287c24fa5b_000000000000_000000000000

❯ ja4 ospf.pcap
[]

❯ ja4 v6.pcap
- stream: 0
  transport: tcp
  src: 3ffe:507:0:1:200:86ff:fe05:80da
  dst: 3ffe:501:410:0:2c0:dfff:fe47:33e
  src_port: 1022
  dst_port: 22
  ja4l_c: 271_64
  ja4l_s: 28494_61
  ja4ssh:
  - c20s20_c18s23_c11s2
  ssh_extras:
    hassh: null
    hassh_server: null
    ssh_protocol_client: SSH-1.5-1.2.26
    ssh_protocol_server: SSH-1.5-1.2.26
    encryption_algorithm: null
awick commented 8 months ago

I should have mentioned I'm compiling the rust code myself. Looks like maybe I need to upgrade tshark.

TShark (Wireshark) 4.2.0 (v4.2.0-0-g54eedfc63953).

./target/debug/ja4 --version ja4 0.16.2 (would be cool if this also printed out tshark version)

vvv commented 8 months ago

(would be cool if this also printed out tshark version)

Yes, good idea! And shortened git SHA1 as well.

vvv commented 8 months ago

I should have mentioned I'm compiling the rust code myself. Looks like maybe I need to upgrade tshark.

TShark (Wireshark) 4.2.0 (v4.2.0-0-g54eedfc63953).

./target/debug/ja4 --version ja4 0.16.2 (would be cool if this also printed out tshark version)

@awick So were you able to generate JA4+ fingerprints with this combination of ja4 and tshark? If yes, please consider closing this issue.

P.S.: I'd recommend building ja4 with --release:

cargo build --release -p ja4

Debug version is noticeably slower on some capture files:

❯ time target/release/ja4 ../pcap/ssh2.pcapng >/dev/null
target/release/ja4 ../pcap/ssh2.pcapng > /dev/null  0.58s user 0.05s system 138% cpu 0.454 total

❯ time target/debug/ja4 ../pcap/ssh2.pcapng >/dev/null
target/debug/ja4 ../pcap/ssh2.pcapng > /dev/null  4.77s user 0.08s system 104% cpu 4.657 total
awick commented 8 months ago

I was just sharing which tshark I'm using, 4.2.0. It has an issue.

awick commented 8 months ago

Ok, so if I run the release version of ja4 it works, when i run the debug version I get these errors. I guess that makes sense because its the asserts that are failing!!! So seems like either the asserts should be removed or code fixed?

vvv commented 8 months ago

But of course!

Now I can see it:

❯ ./target/debug/ja4 /tmp/arkime/tests/pcap/CVE-2018-6794.pcap
The application panicked (crashed).
Message:  assertion failed: ja4l_c > 0
Location: ja4/src/time/tcp.rs:181

Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.

@awick Apologies that it took me so long to face the right direction. And thanks for your persistence.

I'll write a fix tomorrow (it's past 2 AM for me).

The simplest remedy, as you say, is to get rid of those debug_assert!s. But first I'd like to understand why they get triggered.

https://github.com/FoxIO-LLC/ja4/blob/fcfce8bed98a392b2d77225f2e4c14e5c8ab734c/rust/ja4/src/time/tcp.rs#L181-L182

vvv commented 8 months ago

I'll submit a PR with both fixes as soon as I can.