Feature Request: Include Meta Data for Non-Decrypted Connections in HAR File

[techware01]

First of all, thank you for this fantastic library! It has saved me a significant amount of work.

Currently, it seems that entries are only added to the HAR file if they can be decrypted, meaning there is a recorded session key for that transmission. However, there are cases where these keys are missing, such as when an unknown encryption library is used. In these situations, I would like to see the connection details, including the destination IP address and domain, included in the HAR file. This would provide a better picture of which connections have been made, even if decryption has failed. Would this be possible to implement or rather what is needed to implement it?

In my setup, I am using PCAPdroid to record the traffic of an app on Android via the VPN interface to a PCAP file, and I retrieve the TLS keys through hooking with FriTap. When analyzing the app com.nbb.app, I notice that while some traffic is present in the resulting HAR file, other traffic is clearly missing. I suspect this is related to the Flutter framework, which may not be functioning as expected.

When I open the generated PCAPNG file in Wireshark, I can see the missing connections that could not be decrypted.

[U039b]

Hi! Thank you very much for your words.

By definition/by design, HAR (HTTP ARchive) files only contain HTTP traffic. This is why you don't see any TLSv1.x traffic in the HAR. If you need the list of the network flows (not individual packets), you can use NFStream. However, it will not extract payloads for you.

We will not add non-HTTP traffic to the HAR this tool generates, since it will produce an HAR that's not compliant with the specification.

[techware01]

Thanks for your quick response and the explanation. I think I understand the situation better now.

To clarify my understanding: TLSv1.x is typically used on top of HTTP, so if I had the correct decryption keys, I could decrypt these transmissions and potentially add them to a HAR file. However, the challenge arises because TLS can also encrypt other application protocols, such as SMTP. If we were to add new entries with just metadata for the TLS streams, it could lead to misrepresentations of the underlying protocols.

Additionally, is there a way to infer the protocol from the encrypted traffic?

I appreciate your insights on this issue!

[U039b]

Yes, exactly. Adding non-HTTP traffic to the HAR could lead to misrepresentation. To infer the protocol, you need to use Deep Packet Inspection, which is what NFStream, mentioned previously, does.

[techware01]

Alright thank you!