MITM With SSL?

[TechnikEmpire]

I'm not sure if this is a bug or I'm just missing something, I'm assuming the latter. Trying to run the Mitm test, nothing happens on SSL connections. The browser just spins forever waiting for a response and I see no error or other activity coming from the proxy. I'm coming to LittleProxy from my own transparent proxy, and first thing I see missing is the creation, installation or any use of a root CA by which the proxy can spoof certs for the sake of intercepting SSL. Am I just missing some code or documentation? Thanks in advance.

[jekh]

LittleProxy currently doesn't provide this configuration out-of-the-box. It does provide the MitmManager interface, which you can implement to customize your certificate generation as needed. It would be wonderful to get this functionality into LittleProxy, it just hasn't been done yet. There is talk of an extras package (see issue #173) to support this, and @ganskef is currently working on some SSL functionality, but nothing has been released yet. If you're willing to put something together, I'm sure everybody would love to see it!

[TechnikEmpire]

@jekh I did this in C# with my own proxy, working on doing it in C++ as we speak. It's a real pain in the ass to be honest, but basically you've gotta just use bouncy castle to generate a root CA and then you can peek SSL sockets to read the TLS hello/sni/etc and generate an in-memory cert store on the fly. The only problem is, I've been out of Java for a long time (since android 2 :| ) so I'm fumbling a bit trying to dust off the Java section in my memory. lol I'l see if I can put something together though.

[jekh]

I definitely understand that it is a huge pain in the ass... I am completely out of my element when it comes to SSL certificate spoofing. If it helps, the latest commit from @ganskef on his implementation (using bouncy castle) is at b2bd8fbb9f180dcf66700318799ad8631a7d51e4. That PR has since been repurposed, but I believe he has moved the MITM code to https://github.com/ganskef/LittleProxy-mitm.

[TechnikEmpire]

Thanks for the links, I'll have a look tonight and see whats what. You're gonna wanna use ECDH for key exchange to balance security and performance. Anything else becomes so computationally expensive that user experience suffers to the point of it being unusable in any production scenario. TBH I'm out of my depth with this stuff too, I've just googled enough to make it work. :) lol Again I'll take a peek but for anyone coming across this thread:

https://github.com/dlundquist/sniproxy/blob/master/src/tls.c <- Code for parsing TLS hello for the sake of host discovery and hence upstream certificate spoofing.

Blog + Code for being a spoofing CA. Yes it's in C# but it's using bouncy castle and the difference between this and what it would look like in Java is just a matter of syntax. :) http://blog.differentpla.net/blog/2013/03/24/bouncy-castle-being-a-certificate-authority/ https://github.com/rlipscombe/bouncy-castle-csharp

[jekh]

Thanks for the links, @TechnikEmpire. I'll take a look. In terms of performance, assuming it doesn't take "too" long to generate certs (i.e. a few seconds or less), we should be able to leverage caching to make sure that certs are only generated once per host, rather than generating a new cert with every request. I can help with that though, once we have the foundations laid.

[BeamLiu]

It is happy that I found similar discussion here, I did a fork from main stream, and implemented the CA signed feature, it is really nice feature, user only need install the LittleProxy generated CA to his system, then the boring security warning dialog won't come up again.

I am also using bouncy castle to generate the different website certificates in the fly, then sign the certificates using CA, at last, populate the private key and corresponding signed certificates into a keystore in the fly. I did a testing, there isn't much performance issue, my local few seconds could finish all the work of publishing a new certificate for a website, and there is a 200 capicity of LRU map to had certificates cache.

This change is very isolated with other codes, it just had a new implementation of MitmManager.java @jekh , could u kindly take a look? Is there any possiblity for me to create a pull request to merge this feature? My code changes are here: https://github.com/adamfisk/LittleProxy/compare/master...BeamLiu:master

[jekh]

@BeamLiu Thank you! I've only been able to do a cursory glance so far, but I'm very excited about this. I'll be busy for the next few days but will have a deeper look as soon as I can. Hopefully in the meantime others like @ganskef will be able to review as well (he has done some work on this already and is much better-versed in SSL than I am).

[ganskef]

@BeamLiu Thank you! I would love it, to have a CA implementation in a LittleProxy release. Please have a look at CR #174 and ganskef/LittleProxy-mitm. I will have a closer look at your code - first thougts:

I use Guava instead of Apache LRU cache. But, I think caching is not really neccessary, since it benefits from LittleProxys connection cache. It's optionally in my implementation. I use a PKCS12 key store, since Android is not implemeting Oracles jks format. But, I've no success within Android and Netty SSL - unclear OOM errors - WIP. I consider to use Spongy Castle instead of Bouncy Castle. I had noticed truncated SAN lists at developer.chrome.com for example. Therefore I create the dynamic cert for the host name instead of the CN/SAN from the upstream cert. This simplifies my response while offline, too. I've build a caching proxy for offline use for http and https. It's in my daily use with Mac OS X and Linux (and Android http only).

[BeamLiu]

Thanks @jekh , @ganskef , interesting, it seems we are doing similar, sorry for that I didn't had a search before my starting. Anyway, both of u could have a round of reviewing first if u have time, my use case is to intercept browser traffic in desktop, I was trying to modify less code as possible, so I just forked from master, and had code changes directly. I will take a look at @ganskef 's code too.

[ganskef]

Hello @beamliu, please have a look at CR #173. That is what I'm waiting for. MITM seems not to be invited to be a part of LittleProxy core. I think, it have to be core, since https is growing in the web (ArchLinux, Debian, github, mdn, google, http2, spdy...).

Am 09.05.2015 um 12:09 schrieb BeamLiu:

Thanks @jekh https://github.com/jekh , @ganskef https://github.com/ganskef , interesting, it seems we are doing similar, sorry for that I didn't had a search before my starting. Anyway, both of u could have a round of reviewing first if u have time, my use case is to intercept browser traffic in desktop, I was trying to modify less code as possible, so I just forked from master, and had code changes directly. I will take a look at @ganskef https://github.com/ganskef 's code too.

— Reply to this email directly or view it on GitHub https://github.com/adamfisk/LittleProxy/issues/184#issuecomment-100458705.

[ganskef]

Hello @TechnikEmpire, in my experience it was not really neccessary to cache the SSLContext for the dynamic server certificate. I generate an 1024 bit RSA key pair using SHA1PRNG. Thoughts: 2048 takes much longer time on older CPUs. And for almost every client (every browser I've tried), 1024 is sufficient. With my 6 jear old notebook key generation is done in 50-500ms, once per host, up to 8 connections, handled by LittleProxy. If I use SSL caching, I can't notice it in my daily use.

[ganskef]

Hello @beamliu, I've installed your fork in Debian 8. All tests passes. I've run the Launcher with -port 9090 -ca_mitm. I've found a certificate and a key store in my home dir. Good.

Now I've tried to import littleproxy_root.cer as a Certificate Authority in both Chromium and Iceweasel (Mozilla Firefox) browsers, but it fails with this message: The file contained one certificate, which was not imported: LittleProxyRoot: Not a Certification Authority.

Whats wrong? My implementation exports a certificate which can be imported as Certificate Authority into the browser. I expect to use it to suppress the certificate warnings/failures in the browser.

[BeamLiu]

Hi @ganskef , according to https://wiki.mozilla.org/CA:Recommendations_for_Roots , I added the additional CA=true extension into the root certificate, please update the latest code and remove the old keystore and certificate in user home directory, it should work now. Thanks a lot. Later, I will also have a try in Linux, and the intecerption between android and server over https.

[ganskef]

Hello @BeamLiu, yepp it's working now like expected. But, this means it fails with https://developer.chrome.com/ :-( .

The reason is, the Subject Alternative Name field in the upstream certificate is truncated. I could not locate the reason.

My workaround is to create the fake certificate with the requested host name CN=developer.chrome.com instead of CN=misc.google.com and SAN= DNS-Name: misc.google.com DNS-Name: .888googler.com DNS-Name: .adgoogle.net DNS-Name: .baselinestudy.com DNS-Name: .chrome.com DNS-Name: .chromecast.com DNS-Name: .chromium.org DNS-Name: .developer.google.com DNS-Name: .developers.google.com DNS-Name: .feedburner.com DNS-Name: .flutterapp.com DNS-Name: .gbc.beatthatquote.com DNS-Name: .gerritcodereview.com DNS-Name: .glass-community.com DNS-Name: .google.org DNS-Name: .googleapps.com DNS-Name: .googlecompare.co.uk DNS-Name: .googleforveterans.com DNS-Name: .googletraveladservices.com DNS-Name: .hindiweb.com DNS-Name: .hwgo.com DNS-Name: .madewithcode.com DNS-Name: .mfg-inspector.com DNS-Name: .pagespeedmobilizer.com DNS-Name: .personfinder.google.org DNS-Name: .quickoffice.com DNS-Name: .quoteproxy.beatthatquote.com DNS-Name: .schemer.com DNS-Name: .screenwisetrendspanel.com DNS-Name: .shibboleth.tv DNS-Name: .staging.widevine.com DNS-Name: .support.google.com DNS-Name: .thinkwithgoogle.at DNS-Name: .thinkwithgoogle.ch DNS-Name: .thinkwithgoogle.co.il DNS-Name: .thinkwithgoogle.co.uk DNS-Name: .thinkwithgoogle.co.za DNS-Name: .thinkwithgoogle.com.au DNS-Name: .thinkwithgoogle.com.br DNS-Name: .thinkwithgoogle.com.hk DNS-Name: .thinkwithgoogle.de DNS-Name: .thinkwithgoogle.dk DNS-Name: .thinkwithgoogle.es DNS-Name: .thinkwithgoogle.fi DNS-Name: .thinkwithgoogle.fr DNS-Name: .thinkwithgoogle.it DNS-Name: .thinkwithgoogle.jp DNS-Name: .uat.widevine.com DNS-Name: .whatbrowser.org DNS-Name: .widevine.com DNS-Name: .womenwill.com DNS-Name: .xn--9trs65b.com DNS-Name: .youradmob.com DNS-Name: .youtubemobilesupport.com DNS-Name: .zagat.com DNS-Name: 888googler.com DNS-Name: adgoogle.net DNS-Name: baselinestudy.com DNS-Name: chrome.com DNS-Name: chromecast.com DNS-Name: chromium.org DNS-Name: clickserve.dartsearch.net DNS-Name: clickserve.eu.dartsearch.net DNS-Name: clickserve.uk.dartsearch.net DNS-Name: clickserve.us2.dartsearch.net DNS-Name: clickserver.googleads.com DNS-Name: cookiechoices.org DNS-Name: feedburner.com DNS-Name: flutterapp.com DNS-Name: gbc.beatthatquote.com DNS-Name: gerritcodereview.com DNS-Name: glass-community.com DNS-Name: google.domains DNS-Name: google.org DNS-Name: googleapps.com DNS-Name: googlecompare.co.uk DNS-Name: googletraveladservices.com DNS-Name: hindiweb.com DNS-Name: hwgo.com DNS-Name: images.zagat.com DNS-Name: madewithcode.com DNS-Name: mfg-inspector.com DNS-Name: n339.asp-cc.com DNS-Name: pagespeedmobilizer.com DNS-Name: quoteproxy.beatthatquote.com DNS-Name: schemer.com DNS-Name: screenwisetrendspanel.com DNS-Name: thinkwithgoogle.at DNS-Name: thinkwithgoogle.ch DNS-Name: thinkwithgoogle.co.il DNS-Name: thinkwithgoogle.co.uk DNS-Name: thinkwithgoogle.co.za DNS-Name: thinkwithgoogle.com.au DNS-Name: thinkwithgoogle.com.br DNS-Name: thinkwithgoogle.com.hk DNS-Name: thinkwithgoogle.de DNS-Name: thinkwithgoogle.dk DNS-Name: thinkwithgoogle.es DNS-Name: thinkwithgoogle.fi DNS-Name: thinkwithgoogle.fr DNS-Name: thinkwithgoogle.it DNS-Name: thinkwithgoogle.jp DNS-Name: whatbrowser.org DNS-Name: womenwill.com DNS-Name: www.cookiechoices.org DNS-Name: xn--9trs65b.com DNS-Name: youradmob.com DNS-Name: youtubemobilesupport.com DNS-Name: zagat.com

Am 14.05.2015 um 08:45 schrieb BeamLiu:

Hi @ganskef https://github.com/ganskef , according to https://wiki.mozilla.org/CA:Recommendations_for_Roots , I added the additional CA=true extension into the root certificate, please update the latest code and remove the old keystore and certificate in user home directory, it should work now. Thanks a lot. Later, I will also have a try in Linux, and the intecerption between android and server over https.

— Reply to this email directly or view it on GitHub https://github.com/adamfisk/LittleProxy/issues/184#issuecomment-101938727.

[BeamLiu]

hi @ganskef , yes, I tested here, the same, I believe the google certificate may have some special and lead to some potential bug in lower level library, however, the browser can recognize all the SAN. I had a testing, to publish 120+ SAN in my local, and then use this proxy, it seems everything works fine. Don't know what is special with the google certificate.

There should be some more bugs in lower level, maybe in netty, JDK, or LittleProxy itself, the reason I need MITM https interception is that I had some jobs are doing in Fiddler, and want to translate it to java program automation using LittleProxy, after my code is done, I am crazy that the original black box client can work well with Fiddler, but cannot establish SSL connection with LittleProxy, I am still doing investigation, I am also doubting it is some bugs in netty, as it failed in handshake stage, no workaround solution yet, so I believe some bugs are expected.

[ganskef]

Hi @BeamLiu, it was very hard for me to build automated tests. Most of my clients have done the handshake with the upstream proxy instead of my MITM extension, like browsers does it. This seems to be happens with a Netty based client and with Apache HC. Only an URLConnection based client works for me. It was hard to debug. I felt me a little lost in the connections :-) . Frank

Am 15.05.2015 um 05:10 schrieb BeamLiu:

hi @ganskef https://github.com/ganskef , yes, I tested here, the same, I believe the google certificate may have some special and lead to some potential bug in lower level library, however, the browser can recognize all the SAN. I had a testing, to publish 120+ SAN in my local, and then use this proxy, it seems everything works fine. Don't know what is special with the google certificate.

There should be some more bugs in lower level, maybe in netty, JDK, or LittleProxy itself, the reason I need MITM https interception is that I had some jobs are doing in Fiddler, and want to translate it to java program automation using LittleProxy, after my code is done, I am crazy that the original black box client can work well with Fiddler, but cannot establish SSL connection with LittleProxy, I am still doing investigation, I am also doubting it is some bugs in netty, as it failed in handshake stage, no workaround solution yet, so I believe some bugs are expected.

— Reply to this email directly or view it on GitHub https://github.com/adamfisk/LittleProxy/issues/184#issuecomment-102240690.

[rkkoszewski]

Hello, could somebody figure out some work around for this? I'm trying to make it run on the latest 1.0.0 beta version and the browser keeps spinning and waiting but nothing is happening. Is there any other version of LittleProxy where HTTPS with SSL is working with MITM enabled? Greetings.

[zerkz]

@rkkoszewski take a look at @ganskef 's https://github.com/ganskef/LittleProxy-mitm . Note that it does NOT support proxy chaining with MITM yet.

[ganskef]

For clarification: MITM is generally working. You simply have to start the Launcher class, for LittleProxy with the MITM parameter. LittleProxy use a self signed certificate which causes warnings (or failures) in the browser. My extension creates a root certificate to import in the browser and generate server certificates on the fly, signed with it.

This thread is a little bit mixed up. There are more comments about special cases with some exceptional sites or special test client behavior.

@rkkoszewski If you have a spinning browser, you could have requested the proxy URL directly. This causes an endless loop in the moment Issue #205. It's intended to configure the proxy in the browser settings and request your URL with the browser. (Sorry, if I misunderstand you)

[rkkoszewski]

Thanks a lot @zerkz. Just what I was looking for!! Its working just perfectly.

@ganskef direct connection works but I need to inject some JavaScript in every page, and the infinite spinning issue was just happening on SSL connections. But with the build that @zerkz suggested injection works just perfectly on SSL and non SSL pages. Thanks!

[christianleger]

I'd like to ask: what would a fully functional MITM proxy consist of, which would allow us to intercept and inspect HttpObjects?

My basic understanding is mostly based on what @ganskef has stated: LittleProxy gives us the tools to implement our own extension of the MITMManager to "customize your certificate generation as needed".

What customization is needed? What more do we need besides A) call one or more of the MITMManager interface when starting a proxy instance and B) locate the cert generated and add it to my browser's cert list? If I do those steps, will I then be able to view my HTTPS traffic the same as I can do with HTTP? My apologies if this is off-topic from this thread - I didn't want to start a new issue if it has been addressed but my parsing of the discussion didn't give me the answers I'm looking for.

Thanks!

[ganskef]

Hi Christian,

yes, it's possible to build a fully functional MITM application on top of LittleProxy. You have the choice:

https://github.com/ganskef/LittleProxy-mitm

https://github.com/lightbody/browsermob-proxy/tree/master/mitm

MITM by @jekh https://github.com/lightbody/browsermob-proxy/pull/347 is reimplemented December last year. It's released in Maven. It supports LittleProxy and the alternate Jetty based proxy in BMP. Please have a look at the README.md file.

MITM by @ganskef is in my personal use since early 2014, published in January 2015, but there's no release in Maven. You've to integrate the snapshots like it's done in ganskef/LittleProxy-parent.

I need ganskef/LittleProxy with minimal modifications to use it on Android too (but HTTPS fails with Android 5+), and to use it offline without an Internet connection. Please see: https://github.com/adamfisk/LittleProxy/pull/230, and #174. Note: the README describes how to gather the address:port portion of the URI from the CONNECT for HTTPS in filters.

Regards Frank

Am 13.01.2016 um 21:24 schrieb Christian Leger:

I'd like to ask: what would a fully functional MITM proxy consist of, which would allow us to intercept and inspect HttpObjects?

My basic understanding is mostly based on what @ganskef https://github.com/ganskef has stated: LittleProxy gives us the tools to implement our own extension of the MITMManager to "customize your certificate generation as needed".

What customization is needed? What more do we need besides A) call one or more of the MITMManager interface when starting a proxy instance and B) locate the cert generated and add it to my browser's cert list? If I do those steps, will I then be able to view my HTTPS traffic the same as I can do with HTTP? My apologies if this is off-topic from this thread - I didn't want to start a new issue if it has been addressed but my parsing of the discussion didn't give me the answers I'm looking for.

Thanks!

— Reply to this email directly or view it on GitHub https://github.com/adamfisk/LittleProxy/issues/184#issuecomment-171422058.

[christianleger]

Thank you very much Frank for taking the time to answer me.

I must confess that I still don't know what I'm doing - I don't know where certificates (such as the .jks and .cert files created by my instance of LittleProxy) need to go, and right now I'm experimenting with JBrowserDriver which I have running but which is complaining that it's getting untrusted certs (even though I've given it the 'trustanything' option for its jbd.pemfile parameter. I'm still just learning how SSL actually works, and how browsers and Java interact with certs. I'll go back to reading and googling :)

[jekh]

Since MITM with SSL is now supported, along with MITM + Chained Proxy, I'll go ahead and close this issue.