No HTTPS decryption on the web interface

[LetMeR00t]

Hello, I'm stuck since few days concerning the decryption of the HTTPS traffic in the web interface. mitmproxy is setup and is working (in transparent mode). What I understand is that the "DumpTLSMasterSecrets" auxiliary is used to get the server_random and master_secret information. According to my analysis.log, it seems to be well started : analysis.log

According to mitmproxy, I'm able to decrypt the traffic on the PCAP using Wireshark and the SSLKEYLOGFILE environment variable :

However, nothing in my web interface :

I'm not understand what is going wrong ... Can someone help me on this issue ? Thank you !

[jbremer]

Where is the SSLKEYLOGFILE stored? What if you copy it to ~/.cuckoo/storage/analyses/task_id/tlsmaster.txt and reprocess the analysis (cuckoo -d process -r task_id)?

[LetMeR00t]

Thanks for the reply,

As detailed in the PR https://github.com/cuckoosandbox/cuckoo/pull/2102, I created a file named "mitm.sslkeylogfile" for each analysis. It contains all TLS secrets as you can see here :

Now I copied the file mitm.sslkeylogfile in tlsmaster.txt (I would like to add that the previous tlsmaster.txt file was empty) And I rerun the process using your command "cuckoo -d process -r 1":

Nothing change

But I would like to notice that tlsmaster.txt is now empty ! Probably because of this line : https://github.com/cuckoosandbox/cuckoo/blob/59d32361c1636b2b3802a1746f480a7768f6384f/cuckoo/processing/dumptls.py#L41-L45 where the file is opened using the "w" mode. But well, I suppose that is the network processing that decrypt the traffic.

Moreover, the format of the file is not the same as the SSLKEYLOGFILE file ...

Another proof that the SSLKEYLOGFILE is working :

Tell me if you need more information on my analysis. PS : I'm using a Windows 7 VM with the last release of mitmproxy (v3-RC2)

[jbremer]

I'm confident you can make it work if you update the regex used to parse the tlsmaster.txt file. Feel free to make the file opening "a+b" instead, although in that case should make sure that TLS Master Secret entries aren't inserted twice (e.g., when reprocessing an analysis).

[LetMeR00t]

Which regexp are you talking about ?.. I don't find where the tlsmaster.txt is parsed using a regexp sorry

[jbremer]

Oh, sorry, it's elsewhere: https://github.com/jbremer/httpreplay/blob/master/httpreplay/misc.py

[LetMeR00t]

Thanks,

I tried to modify your code :

and compile with "python setup.py sdist develop" (only the httpreplay)

But same result, no decryption and I check the tlsmaster contains the expected format with no duplicate.

Is what you were thinking ?

[jbremer]

Just to be sure, can you check that the ret variable is filled with values?

[LetMeR00t]

Right, it seems that the code is never used ... I say that because none of the both print in the function appears...

[jbremer]

Well, I can assure you that code is used. Perhaps pip uninstall httpreplay & pip install -e . once more.

[LetMeR00t]

Even with these commands, the both print aren't used ... I assumed that we should at list see the "print filepath" no ? (even both ...)

[LetMeR00t]

Even if I put a log.debug and a print on these portions of code : Nothing appears in the "cuckoo -d process -r 1" results

[LetMeR00t]

Wait, I find the code where httpreplay is set to log everything only critical. If I set it to DEBUG, I have some errors : But nothing appears for the "read_tlsmaster" function anyway

[LetMeR00t]

If I understand, the code is using the "https_handler" function that already have the tlsmaster information from cuckoo.

And it has it from this line I think : https://github.com/cuckoosandbox/cuckoo/blob/59d32361c1636b2b3802a1746f480a7768f6384f/cuckoo/processing/network.py#L897-L908 And well, it doesn't care about the tlsmaster.txt file :)

What do you think ?

If you agree with me, I can propose a patch to use the mitmproxy SSLKEYLOGFILE with the new regexp (and keep the first in case when it's concerning Windows because I'm assuming that this format is coming from the lsass.exe process right ?)

[jbremer]

Oh, right, sorry for that - can't always be correct, hehe. How about just parsing the SSLKEYLOGFILE somewhere in network.py and making sure the get_tlsmaster() method returns it?

[LetMeR00t]

Don't worry :) I tried to understand all the code hehe Well, I would like to say that there are two things here :

• the TLS Master key from the lsass.exe program (which use a precise format)
• the TLS Master key from the mitmproxy process (which use another precise format)

There is a dumptls processing that happened after the network processing. The dumptls processing is just recovering all TLS master from lsass.exe program and store them in a tlsmaster.txt file (If I'm understanding what is doing). https://github.com/cuckoosandbox/cuckoo/blob/59d32361c1636b2b3802a1746f480a7768f6384f/cuckoo/processing/dumptls.py#L11-L45

I think we should use the dumptls processing BEFORE the network one and read the result stored in the tlsmaster.txt on the get_tlsmaster function based in network.py (and not recover it from results). Moreover, during the dumptls process, we could recover the TLS Master key from the mitm.sslkeylogfile generated thanks to one of my PR if the file exists. https://github.com/cuckoosandbox/cuckoo/blob/59d32361c1636b2b3802a1746f480a7768f6384f/cuckoo/processing/network.py#L897-L908

So, the solution would be :

• Change the priority of the dumptls processing
• Modify the get_tlsmaster function for recovering data from tlsmaster.txt and mitm.sslkeylogfile files using the same function read_tlsmaster from your httpreplay repository.
• Use the MITMPROXY_SSLKEYLOGFILE for storing TLS Master keys for the current analysis in a specific file named for example mitm.sslkeylogfile : https://github.com/cuckoosandbox/cuckoo/pull/2102/commits/3c39049127c48c2d0a98e7c6840a98b3445d5169 (Line 71 to 74)

What do you think about this ?

[jbremer]

All sounds good to me! Just one thing; rename the file to something like tlsmaster.mitm, would make more sense I think :-)

[saxenakanishk]

@LetMeR00t Hi, sorry for commenting randomly, but I wanted to ask you whether mitmproxy is same as wireshark? I am having trouble in dealing with mitm root certificate, so I was curious to know whether I can somehow get my network analysis results (pcap file) with Wireshark only.

[LetMeR00t]

Hi, No, mitmproxy is a proxy that can intercept and modify traffic between a client and a server. Wireshark is a tool for analyzing the traffic on the network, it can't modify anything. But of course, if you configure well the interaction between the host machine and the VM (on vboxnet0 for example), you can easily listen the traffic on this interface and see what is going on :)