search

FoxIO-LLC / ja4

JA4+ is a suite of network fingerprinting standards
https://foxio.io
Other
775 stars 65 forks source link

rust ja4x issues with Arkime pcaps #52

Closed awick closed 5 months ago

awick commented 5 months ago

These are all missing ja4x output from rust ja4 against files in https://github.com/arkime/arkime/tree/main/tests/pcap

vvv commented 5 months ago

Indeed. The app returns 0: Parsing Error: Der(InvalidLength) for each of those files.

❯ ja4x socks4-https.pcap
Error:
   0: Parsing Error: Der(InvalidLength)

Location:
   ja4x/src/main.rs:53

Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.
vvv commented 5 months ago

@awick I'm looking at socks4-https.pcap as an example.

image

I'll update ja4x to handle the input like this gracefully and not fail, but there is no TLS frames to obtain certificates from, hence no JA4X.

vvv commented 5 months ago

I'll update ja4x to handle the input like this gracefully and not fail

Ah, I forgot that ja4x expects certificates in PEM or DER format as its input. It is not supposed to work with the capture files, that's what ja4 is for.

Still, an error message like this

Error:
   0: Parsing Error: Der(InvalidLength)

Location:
   ja4x/src/main.rs:53

is quite unfriendly to the user.

awick commented 5 months ago

So i'm just using ja4, not ja4x binary, if that matters.

It is https over socks, I guess wireshark doesn't decode it by default

vvv commented 5 months ago

So i'm just using ja4, not ja4x binary, if that matters.

Makes sense. Thanks for clarifying.

It is https over socks, I guess wireshark doesn't decode it by default

Wireshark needs secrets to decrypt TLS traffic. Secrets can either be embedded into a capture file or stored separately.

awick commented 5 months ago

There is no decryption here. This is https over socks

awick commented 5 months ago

Let's ignore the socks ones. I am using release build now. and i can see the certs in wireshark

./target/release/ja4 ~/arkime/tests/pcap/ssl-selfsign.pcap

./target/release/ja4 ~/arkime/tests/pcap/https3-301-get.pcap

vvv commented 5 months ago

@awick That's weird.

Works on my machine.™ ``` ❯ for f in https3-301-get.pcap socks-https-example.pcap socks4-https.pcap ssl-selfsign.pcap; do echo "--- $f ---"; ja4 $f; done; unset f --- https3-301-get.pcap --- - stream: 0 transport: tcp src: 10.180.156.141 dst: 192.30.252.130 src_port: 62599 dst_port: 443 tls_server_name: www.github.com ja4: t10d230100_6a57a6f57151_000000000000 ja4s: t100200_0005_696072bc484d tls_certs: - x509: - ja4x: 7d5dbb3783b4_59b7510e6266_5e17a2514980 issuerCountryName: US issuerOrganizationName: DigiCert Inc issuerOrganizationalUnit: www.digicert.com issuerCommonName: DigiCert High Assurance EV CA-1 subjectBusinessCategory: Private Organization subjectMsJurisdictionCountry: US subjectMsJurisdictionStateOrProvince: Delaware subjectSerialNumber: '5157550' subjectStreetAddress: 548 4th Street subjectPostalCode: '94107' subjectCountryName: US subjectStateOrProvinceName: California subjectLocalityName: San Francisco subjectOrganizationName: GitHub, Inc. subjectCommonName: github.com - ja4x: 7d5dbb3783b4_7d5dbb3783b4_897f3043ab93 issuerCountryName: US issuerOrganizationName: DigiCert Inc issuerOrganizationalUnit: www.digicert.com issuerCommonName: DigiCert High Assurance EV Root CA subjectCountryName: US subjectOrganizationName: DigiCert Inc subjectOrganizationalUnit: www.digicert.com subjectCommonName: DigiCert High Assurance EV CA-1 ja4l_c: 33_64 ja4l_s: 17805_50 --- socks-https-example.pcap --- - stream: 0 transport: tcp src: 10.180.156.185 dst: 10.180.156.249 src_port: 53554 dst_port: 1080 tls_server_name: www.example.com ja4: t10d230100_6a57a6f57151_000000000000 ja4s: t100100_0005_bc98f8e001b5 tls_certs: - x509: - ja4x: 7d5dbb3783b4_2bab15409345_5e17a2514980 issuerCountryName: US issuerOrganizationName: DigiCert Inc issuerOrganizationalUnit: www.digicert.com issuerCommonName: DigiCert High Assurance CA-3 subjectCountryName: US subjectStateOrProvinceName: California subjectLocalityName: Santa Monica subjectOrganizationName: EdgeCast Networks, Inc. subjectCommonName: gp1.wac.edgecastcdn.net - ja4x: 7d5dbb3783b4_7d5dbb3783b4_c519788dcb01 issuerCountryName: US issuerOrganizationName: DigiCert Inc issuerOrganizationalUnit: www.digicert.com issuerCommonName: DigiCert High Assurance EV Root CA subjectCountryName: US subjectOrganizationName: DigiCert Inc subjectOrganizationalUnit: www.digicert.com subjectCommonName: DigiCert High Assurance CA-3 ja4l_c: 23_64 ja4l_s: 210_64 - stream: 2 transport: tcp src: 10.180.156.185 dst: 10.180.156.249 src_port: 53555 dst_port: 1080 tls_server_name: www.example.com ja4: t10d230100_6a57a6f57151_000000000000 ja4s: t100100_0005_bc98f8e001b5 tls_certs: - x509: - ja4x: 7d5dbb3783b4_2bab15409345_5e17a2514980 issuerCountryName: US issuerOrganizationName: DigiCert Inc issuerOrganizationalUnit: www.digicert.com issuerCommonName: DigiCert High Assurance CA-3 subjectCountryName: US subjectStateOrProvinceName: California subjectLocalityName: Santa Monica subjectOrganizationName: EdgeCast Networks, Inc. subjectCommonName: gp1.wac.edgecastcdn.net - ja4x: 7d5dbb3783b4_7d5dbb3783b4_c519788dcb01 issuerCountryName: US issuerOrganizationName: DigiCert Inc issuerOrganizationalUnit: www.digicert.com issuerCommonName: DigiCert High Assurance EV Root CA subjectCountryName: US subjectOrganizationName: DigiCert Inc subjectOrganizationalUnit: www.digicert.com subjectCommonName: DigiCert High Assurance CA-3 ja4l_c: 22_64 ja4l_s: 280_64 - stream: 4 transport: tcp src: 10.180.156.185 dst: 10.180.156.249 src_port: 53556 dst_port: 1080 tls_server_name: www.example.com ja4: t10d230100_6a57a6f57151_000000000000 ja4s: t100100_0005_bc98f8e001b5 tls_certs: - x509: - ja4x: 7d5dbb3783b4_2bab15409345_5e17a2514980 issuerCountryName: US issuerOrganizationName: DigiCert Inc issuerOrganizationalUnit: www.digicert.com issuerCommonName: DigiCert High Assurance CA-3 subjectCountryName: US subjectStateOrProvinceName: California subjectLocalityName: Santa Monica subjectOrganizationName: EdgeCast Networks, Inc. subjectCommonName: gp1.wac.edgecastcdn.net - ja4x: 7d5dbb3783b4_7d5dbb3783b4_c519788dcb01 issuerCountryName: US issuerOrganizationName: DigiCert Inc issuerOrganizationalUnit: www.digicert.com issuerCommonName: DigiCert High Assurance EV Root CA subjectCountryName: US subjectOrganizationName: DigiCert Inc subjectOrganizationalUnit: www.digicert.com subjectCommonName: DigiCert High Assurance CA-3 ja4l_c: 15_64 ja4l_s: 233_64 --- socks4-https.pcap --- - stream: 0 transport: tcp src: 10.0.0.1 dst: 10.0.0.2 src_port: 50606 dst_port: 9901 ja4l_c: 119349_126 ja4l_s: 40155_52 --- ssl-selfsign.pcap --- - stream: 0 transport: tcp src: 10.180.152.137 dst: 10.180.152.137 src_port: 60101 dst_port: 4443 ja4: ts3i230000_6a57a6f57151_000000000000 ja4s: ts30100_0035_bc98f8e001b5 tls_certs: - x509: - ja4x: 7022c563de38_7022c563de38_795797892f9c issuerCommonName: molochftw subjectCommonName: molochftw - x509: - ja4x: bbd6cc0fca29_bbd6cc0fca29_795797892f9c issuerCommonName: MO issuerOrganizationName: LO issuerCountryName: CH subjectCommonName: MO subjectOrganizationName: LO subjectCountryName: CH ja4l_c: 5_64 ja4l_s: 30_64 ```
vvv commented 5 months ago

@awick Would you try to build from my branch and run

RUST_LOG=debug target/release/ja4 ~/arkime/tests/pcap/ssl-selfsign.pcap

?

awick commented 5 months ago

yep that works! I give up.

awick commented 5 months ago

I figured it out!!! The wireshark ja4.so was installed when I was running before and was crashing I guess. I uninstalled it earlier so I could check these files in wireshark. I just reinstalled it, and even your branch isn't working.

For rust ja4 tshark runs is there a way to disable the ja4.so wireshark plugin?

vvv commented 5 months ago

For rust ja4 tshark runs is there a way to disable the ja4.so wireshark plugin?

You can remove ja4.so from the Wireshark's epan/ directory. Or just rename the file. For macOS the path to the epan/ dir is /Applications/Wireshark.app/Contents/PlugIns/wireshark/4-2/epan/.

I'm not entirely sure, but I would expect wireshark and tshark to use the same plugins. So I don't think a plugin (e.g. epan/ja4.so) can be disabled for tshark only and remain activated for wireshark.

awick commented 5 months ago

Yes I know how to do it, I was asking if there is a way inside of the ja4 programs when running tshark to disable it so there is never an accidental interaction. Anyway, closing this issue since it looks like tshark doesn't decode socks and the other ones were caused by ja4.so crashing tshark.