search

browserup / browserup-proxy

BrowserUp Proxy is a free utility to watch, test, and manipulate web application network traffic and performance.
https://browserup.com
Apache License 2.0
164 stars 42 forks source link

SSL Certificate - certificate_unknown - Android #137

Closed nicexavi closed 3 years ago

nicexavi commented 5 years ago

Describe the bug Hi, I'm trying to run the BrowserMobProxyServer with an Android device to monitor the traffic.

The problem, it's that I can't monitor the traffic that runs over HTTPS, ONLY for the apps. The Chrome in my Android device, works fine with the BrowserUp Proxy, but I when I try to access to any app (like Play Store) it won't work, and it throws me a "javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown" exception.

I'm using Selenium to automate the tests and I'm using the BrowserUpProxy version 1.2.1.

I've already installed the ca-certificate-rsa.cer on my Android Device. Still won't work.

To Reproduce Steps to reproduce the behavior:

  1. Create the Embedded BrowserUpProxy with setTrustAllServers to true.
  2. Start the Proxy, connect Android to the proxy.

Screenshots Throws me the error: [LittleProxy-0-ClientToProxyWorker-0] ERROR org.littleshoot.proxy.impl.ClientToProxyConnection - (NEGOTIATING_CONNECT) [id: 0x53e1bf22, L:0.0.0.0/0.0.0.0:8080 ! R:/10.10.40.77:46362]: Caught an exception on ClientToProxyConnection io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:472) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:682) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:617) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:534) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at java.base/java.lang.Thread.run(Thread.java:835) Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131) at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:307) at java.base/sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:285) at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:180) at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:164) at java.base/sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:681) at java.base/sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:636) at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:454) at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:433) at java.base/javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:634) at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:295) at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1332) at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1227) at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274) at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ... 16 more [LittleProxy-0-ProxyToServerWorker-3] INFO org.littleshoot.proxy.impl.ProxyToServerConnection - (HANDSHAKING) [id: 0x74048cdf, L:0.0.0.0/0.0.0.0:63811]: Connection to upstream server failed javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131) at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:307) at java.base/sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:285) at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:180) at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:164) at java.base/sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:681) at java.base/sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:636) at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:454) at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:433) at java.base/javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:634) at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:295) at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1332) at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1227) at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274) at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:682) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:617) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:534) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at java.base/java.lang.Thread.run(Thread.java:835)

ericbeland commented 5 years ago

Can you try installing ca-certificate-ec.cer ?

nicexavi commented 5 years ago

I've installed it.

Throws the same error and in ChromeWebBrowser on Android, says me that the connection it's not private.

ericbeland commented 5 years ago

You wrote that you installed "ca-certificate-rsa.cer" not "ca-certificate-ec.cer" which are two different files, so I just wanted to clarify. We ship with both.

nicexavi commented 5 years ago

Sorry about the confusion.

First, I tried to install the RSA and didn't work.

Now, you said me to try the EC and I tried it with the same result.

ericbeland commented 5 years ago

Can you point me to the steps you followed to install it?

nicexavi commented 5 years ago
  1. I downloaded the certificates from: https://github.com/browserup/browserup-proxy/tree/master/browserup-proxy-core/src/main/resources/sslSupport
  2. I sended the certificates via Telegram to my Android
  3. I opened it in my Android and the Android asked me to confirm my pattern
  4. I named the certificate with name "rsa-cert-littleproxy"
  5. In the "Credential use" i selected "VPN and apps"
  6. The certificate it's fully installed into the phone.

(Same steps for the "ca" certificate

Then I started the embebed proxy in the java. The ChromeBrowser in android works well. When I use the apps, throws me the "certificate_unknown" error.

nicexavi commented 5 years ago

Any updates on this @ericbeland ?

Do you know what it could be?

Thanks!

ericbeland commented 5 years ago

I'm sorry @nicexavi I don't know off-hand. It sounds like the android apps are maybe separate from the certificate acceptance for chrome. I don't really have any ideas how to fix that, though--I haven't worked on it with chrome apps as that isn't our use-case. We focus on browser based testing. If anyone else has ideas, please chime in.

Ardesco commented 4 years ago

I'm going to guess it's related to Androids new cert policies, you now need to add an exclusion in the app codebase for custom certificates:

https://android-developers.googleblog.com/2016/07/changes-to-trusted-certificate.html

Also look at the Android section here, it gives you all the info you need to fix general Android 7+ certificate errors:

https://www.charlesproxy.com/documentation/using-charles/ssl-certificates/

nicexavi commented 4 years ago

Hi,

It's strange because with CharlesProxy there isn't any problem.

I mean... I can decrypt SSL and do Network Throttling with Charles certificate and all seems be working well.

With Browserup nope.

Where is the difference...?

Ardesco commented 4 years ago

Charles proxy and and browserup proxy use different certificates. I would guess that the exclusion has been added for the Charles proxy certificate, but not for the browserup certificate.

Ardesco commented 4 years ago

You could always try making browserup use your Charles proxy certs and see if that works. Go to Help -> SSL Proxying -> Export Charles root certificate and private key... and then use it like below:

https://github.com/browserup/browserup-proxy/blob/master/browserup-proxy-mitm/src/test/java/com/browserup/bup/mitm/example/CustomCAKeyStoreExample.java

nicexavi commented 4 years ago

Solution:

Include the directive and generate my own certificates.

Network security config: https://android-developers.googleblog.com/2016/07/changes-to-trusted-certificate.html

Generate certs: https://github.com/browserup/browserup-proxy/blob/master/browserup-proxy-mitm/src/test/java/com/browserup/bup/mitm/example/SaveGeneratedCAExample.java

Use certs: https://github.com/browserup/browserup-proxy/blob/master/browserup-proxy-mitm/src/test/java/com/browserup/bup/mitm/example/CustomCAKeyStoreExample.java

Thanks a lot folks!

ericbeland commented 4 years ago

@nicexavi Would you be willing to consider making a README on this topic we can ship and link to from the main readme? It seems like something we could document to leave future breadcrumbs for ourselves and others. Would you be interested in creating a PR with that info, some description of the problem and what you did?

nicexavi commented 4 years ago

Hey!

Yes, I'll do a README to help the community.

I'll to the PR with my other github account named "suitux". It's my personal account instead of the enterprise account.

Thanks!

akondas commented 3 years ago

If someone is not bothered by an unsecured certificate, there is one more solution (it took me a while to find it, so I share):

ChromeOptions chromeOptions = new ChromeOptions();
chromeOptions.setAcceptInsecureCerts(true);