search

FoxIO-LLC / ja4

JA4+ is a suite of network fingerprinting standards
https://foxio.io
Other
774 stars 65 forks source link

JA4+ is not compatible with Zeek 5 (Corelight) #86

Closed donutpuncher closed 2 months ago

donutpuncher commented 3 months ago

testing package using corelight-client prior to deployment and produces following:-

Validating Bro packages Error in ja4/./helper.zeek, line 72: syntax error, at or near "]"

Looks like additional '}' for if function on lines 67-72 causing error when trying to test package for deployment in corelight

if (!c$fp$client_hello?$version) { c$fp$client_hello$version = version; } c$fp$client_hello$cipher_suites = no_grease_ciphers; c$fp$client_hello$compression_methods = no_grease_comp_methods; }

john-althouse commented 3 months ago

@donutpuncher I'm not sure what could be causing it to say "syntax error, at or near "]"" As line 72 does not have a "]" in it.

The curly bracket on line 72 closes out the event that starts on line 51. There's no extra bracket. This issue appears to be with Corelight as the scripts run without issue on vanilla Zeek.

J0eJ0h commented 3 months ago

@donutpuncher Did you follow up with Corelight on the issue?

donutpuncher commented 3 months ago

I’ve forwarded to tech lead on my Corelight account but have bit more info to add.

Corelight-client is the app used to validate then convert a zkg package into a bundle for deployment on corelight sensors.

Whilst validating with this tool, the following output is displayed:-

corelight-client bro bundle upload --bundle ja4.bundle

The Bro package bundle has been uploaded and is now being validated and installed. ............. Error: Bro package operation failed. Bro packages failed validation.

Diagnostics: Unbundling Bro packages Extracted "https://github.com/FoxIO-LLC/ja4" Found 1 Bro package Validating Bro packages Error in ja4/./helper.zeek, line 72: syntax error, at or near "]"

Looking at the correct file (my bad) in .zkg/clones/packages/ja4/zeek/helper.zeek, it looks like it’s referring to the following function

Format the signature and hashing algorithm codes into a single value

function make_quadword(byte1: count, byte2: count): count { local t: table[string] of count = [ ["0"] = 0, ["1"] = 1, ["2"] = 2, ["3"] = 3, ["4"] = 4, ["5"] = 5, ["6"] = 6, ["7"] = 7, ["8"] = 8, ["9"] = 9, ["a"] = 10, ["b"] = 11, ["c"] = 12, ["d"] = 13, ["e"] = 14, ["f"] = 15, ]; <-- line 72 local b1 = to_lower(fmt("%02x", byte1)); local b2 = to_lower(fmt("%02x", byte2)); local byte1_total: count = (t[b1[0]] 161616) + (t[b1[1]] 1616); local byte2_total: count = (t[b2[0]] 16) + (t[b2[1]] * 1);

return byte1_total + byte2_total; }

Definitely not my skill space but I can’t see anything obviously wrong with this.

From: Jo @.> Sent: Monday, March 18, 2024 4:16 PM To: FoxIO-LLC/ja4 @.> Cc: donutpuncher @.>; Mention @.> Subject: Re: [FoxIO-LLC/ja4] ja4/zeek/ja4/helpers.zeek (Issue #86)

@donutpuncherhttps://github.com/donutpuncher Did you follow up with Corelight on the issue?

— Reply to this email directly, view it on GitHubhttps://github.com/FoxIO-LLC/ja4/issues/86#issuecomment-2004354916, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AECOLCABHAUR7BV4PL3AUO3YY4HMPAVCNFSM6AAAAABEQNDJL6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMBUGM2TIOJRGY. You are receiving this because you were mentioned.Message ID: @.**@.>>

john-althouse commented 3 months ago

There is no ja4/zeek/helper.zeek file. There's a ja4/zeek/ja4/helpers.zeek file. And that's on line 41, not 72. This makes me think maybe you're working with an old version or maybe this is a copy-pasta mistake. In either case, can you try to clone this repo again and try again from scratch?

donutpuncher commented 3 months ago

I’m using Zeek Package Manager (zkg-3.0.1)

zkg install zeek/foxio/ja4

This downloads/creates package in

            ~/.zkg/clones/package/ja4

In ~/.zkg/clones/package/ja4/zeek subdirectory, there is a helper.zeek file and it is this that the “Error in ja4/./helper.zeek, line 72: syntax error, at or near "]"” is referring to - (make_quadword function).

I’ve purged all other packages from zkg and only using zeek/foxio/ja4 (v0.18.1)

From: John Althouse @.> Sent: Tuesday, March 19, 2024 3:57 PM To: FoxIO-LLC/ja4 @.> Cc: donutpuncher @.>; Mention @.> Subject: Re: [FoxIO-LLC/ja4] ja4/zeek/ja4/helpers.zeek (Issue #86)

There is no ja4/zeek/helper.zeek file. There's a ja4/zeek/ja4/helpers.zeek file. And that's on line 41, not 72. This makes me think maybe you're working with an old version or maybe this is a copy-pasta mistake. In either case, can you try to clone this repo again and try again from scratch?

— Reply to this email directly, view it on GitHubhttps://github.com/FoxIO-LLC/ja4/issues/86#issuecomment-2007560466, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AECOLCBXXUW556JQOUY3CM3YZBN3HAVCNFSM6AAAAABEQNDJL6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMBXGU3DANBWGY. You are receiving this because you were mentioned.Message ID: @.**@.>>

donutpuncher commented 3 months ago

I’m using Zeek Package Manager (zkg-3.0.1)

zkg install zeek/foxio/ja4

This downloads/creates package in

            ~/.zkg/clones/package/ja4

In ~/.zkg/clones/package/ja4/zeek subdirectory, there is a helper.zeek file and it is this that the “Error in ja4/./helper.zeek, line 72: syntax error, at or near "]"” is referring to (make_quadword function).

I’ve purged all other packages from zkg and only using zeek/foxio/ja4 (v0.18.1)

From: John Althouse @.> Sent: Tuesday, March 19, 2024 3:57 PM To: FoxIO-LLC/ja4 @.> Cc: donutpuncher @.>; Mention @.> Subject: Re: [FoxIO-LLC/ja4] ja4/zeek/ja4/helpers.zeek (Issue #86)

There is no ja4/zeek/helper.zeek file. There's a ja4/zeek/ja4/helpers.zeek file. And that's on line 41, not 72. This makes me think maybe you're working with an old version or maybe this is a copy-pasta mistake. In either case, can you try to clone this repo again and try again from scratch?

— Reply to this email directly, view it on GitHubhttps://github.com/FoxIO-LLC/ja4/issues/86#issuecomment-2007560466, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AECOLCBXXUW556JQOUY3CM3YZBN3HAVCNFSM6AAAAABEQNDJL6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMBXGU3DANBWGY. You are receiving this because you were mentioned.Message ID: @.**@.>>

john-althouse commented 3 months ago

@donutpuncher We see what the issue is. Corelight is still running on Zeek 5, the current LTS version of Zeek is 6.0.2 which is what JA4+ was written and tested on. We're working to see if we can make it backward compatible.

donutpuncher commented 3 months ago

That makes sense. Corelight support say they're going to Zeek 6 in April update

From: John Althouse @.> Sent: Wednesday, March 20, 2024 4:19:04 PM To: FoxIO-LLC/ja4 @.> Cc: donutpuncher @.>; Mention @.> Subject: Re: [FoxIO-LLC/ja4] ja4/zeek/ja4/helpers.zeek (Issue #86)

@donutpuncherhttps://github.com/donutpuncher We see what the issue is. Corelight is still running on Zeek 5, the current LTS version of Zeek is 6.0.2 which is what JA4+ was written and tested on. We're working to see if we can make it backward compatible.

— Reply to this email directly, view it on GitHubhttps://github.com/FoxIO-LLC/ja4/issues/86#issuecomment-2009972174, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AECOLCHYEPVF3DNTGVQZZF3YZGZHRAVCNFSM6AAAAABEQNDJL6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMBZHE3TEMJXGQ. You are receiving this because you were mentioned.Message ID: @.***>

ckreibich commented 3 months ago

The reason you guys are seeing confusion around helpers.zeek/helper.zeek or no "]" being on line 72 is that when you ask zkg to install the package, it will pick the latest tagged version, which is currently v0.18.1. So when working in git, look at that version, not latest main. You'll find helper.zeek, and the offending "]" etc.

ckreibich commented 3 months ago

Btw — the Zeek project did a community survey in April 2023 and at the time found that Zeek 5.0.x was the single most popular version in use (39%), 24% reported using a 3.x version and 51% reported using a 4.x version. This will surely have shifted to 6.0.x+ at this point, but if you want to broadly support existing installations you need to anticipate versions older than the currently supported ones. If you don't want to do that, you can make your package depend on 6.0+, so zkg won't install it in older Zeeks.

john-althouse commented 3 months ago

Thanks @ckreibich! That is great to know. It looks like the only way to force zkg to grab main is to delete all tags in a repo, which is unfortunate. It would be great if I could specify branch = main in zkg.meta.

I've updated the latest tag to v0.18.2 so zkg will grab the latest version. I've also updated zkg.meta to depend on >= 6.0.0 for now.

ckreibich commented 3 months ago

No problem — you can tell zkg to install main by saying zkg install --version main <package>. Same for any other version, branch, or commit. When zkg finds version tags, it defaults to picking the latest one in semantic version order.

J0eJ0h commented 2 months ago

This was fixed for everything but QUIC in https://github.com/FoxIO-LLC/ja4/commit/76cf529edc0ce964258a23ad73d5a7c756231003

QUIC support down level would be significant work as it is not supported by base Zeek 5 and is not currently in scope.