search

browserup / browserup-proxy

BrowserUp Proxy is a free utility to watch, test, and manipulate web application network traffic and performance.
https://browserup.com
Apache License 2.0
164 stars 41 forks source link

Error ProxyToServerConnection (DISCONNECTED) #303

Closed Gordiievskyi closed 4 years ago

Gordiievskyi commented 4 years ago

Describe the bug In my logs I often see next error message: 

[ERROR 2020-08-17T16:03:14,857 org.littleshoot.proxy.impl.ProxyToServerConnection] (LittleProxy-0-ProxyToServerWorker-7) (DISCONNECTED) [id: 0x3ad57cb0, L:0.0.0.0/0.0.0.0:59843 ! R:ml314.com/172.26.155.36:443]: Caught an exception on ProxyToServerConnection io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 3c21444f43545950452048544d4c205055424c494320222d2f2f494554462f2f4454442048544d4c20322e302f2f454e223e0a3c68746d6c3e3c686561643e0a3c7469746c653e3430302042616420526571756573743c2f7469746c653e0a3c2f686561643e3c626f64793e0a3c68313e42616420526571756573743c2f68313e0a3c703e596f75722062726f777365722073656e7420612072657175657374207468617420746869732073657276657220636f756c64206e6f7420756e6465727374616e642e3c6272202f3e0a3c2f703e0a3c68723e0a3c616464726573733e4170616368652f322e322e33202843656e744f532920536572766572206174203137322e32362e3135352e333620506f72742038303c2f616464726573733e0a3c2f626f64793e3c2f68746d6c3e0a

To Reproduce Steps to reproduce the behavior: Steps to reproduce the behavior:

  1. Create proxy server on some port, in my case 8080: ./browserup-proxy-2.0.1/bin/browserup-proxy -port 8080
  2. Create new proxy curl --location --request POST '0.0.0.0:8080/proxy?trustAllServers=true&port=8081'
  3. Start trace the traffic : curl --location --request PUT '0.0.0.0:8080/proxy/8081/har?captureHeaders=true&captureCookies=true&captureContent=true&captureBinaryContent=true&initialPageRef=MyPage%20112&initialPageTitle=My%20Page' \ --header 'Content-Type: application/x-www-form-urlencoded' \ --data-raw ''
  4. Use user agent of your choice, in my case chrome, configure it to use created proxy( either throught system proxy or with args if user agent support it, chrome supports it so I will rpovide here command for OSX) open -a "Google Chrome" --args --proxy-server="http://0.0.0.0:8081" --profile-directory=Default5
  5. Open this web page https://www.wsj.com/ wait for content to be loaded.

Expected behavior In BUP proxy logs there no exception

Please complete the following information:

Additional context

[INFO  2020-08-28T16:39:42,560 org.littleshoot.proxy.impl.ProxyToServerConnection] (LittleProxy-0-ProxyToServerWorker-3) (HANDSHAKING) [id: 0x525a8f5f, L:0.0.0.0/0.0.0.0:63948 ! R:ml314.com/172.26.155.36:443]: Connection to upstream server failed io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 3c21444f43545950452048544d4c205055424c494320222d2f2f494554462f2f4454442048544d4c20322e302f2f454e223e0a3c68746d6c3e3c686561643e0a3c7469746c653e3430302042616420526571756573743c2f7469746c653e0a3c2f686561643e3c626f64793e0a3c68313e42616420526571756573743c2f68313e0a3c703e596f75722062726f777365722073656e7420612072657175657374207468617420746869732073657276657220636f756c64206e6f7420756e6465727374616e642e3c6272202f3e0a3c2f703e0a3c68723e0a3c616464726573733e4170616368652f322e322e33202843656e744f532920536572766572206174203137322e32362e3135352e333620506f72742038303c2f616464726573733e0a3c2f626f64793e3c2f68746d6c3e0a
    at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1203) [netty-all-4.1.39.Final.jar:4.1.39.Final]
    at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1271) [netty-all-4.1.39.Final.jar:4.1.39.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:505) [netty-all-4.1.39.Final.jar:4.1.39.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:444) [netty-all-4.1.39.Final.jar:4.1.39.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283) [netty-all-4.1.39.Final.jar:4.1.39.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-all-4.1.39.Final.jar:4.1.39.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-all-4.1.39.Final.jar:4.1.39.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-all-4.1.39.Final.jar:4.1.39.Final]
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1421) [netty-all-4.1.39.Final.jar:4.1.39.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-all-4.1.39.Final.jar:4.1.39.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-all-4.1.39.Final.jar:4.1.39.Final]
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) [netty-all-4.1.39.Final.jar:4.1.39.Final]
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-all-4.1.39.Final.jar:4.1.39.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:697) [netty-all-4.1.39.Final.jar:4.1.39.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:632) [netty-all-4.1.39.Final.jar:4.1.39.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:549) [netty-all-4.1.39.Final.jar:4.1.39.Final]
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:511) [netty-all-4.1.39.Final.jar:4.1.39.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918) [netty-all-4.1.39.Final.jar:4.1.39.Final]
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-all-4.1.39.Final.jar:4.1.39.Final]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_231]

[ERROR 2020-08-28T16:39:42,562 org.littleshoot.proxy.impl.ProxyToServerConnection] (LittleProxy-0-ProxyToServerWorker-3) (DISCONNECTED) [id: 0x525a8f5f, L:0.0.0.0/0.0.0.0:63948 ! R:ml314.com/172.26.155.36:443]: Caught an exception on ProxyToServerConnection io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 3c21444f43545950452048544d4c205055424c494320222d2f2f494554462f2f4454442048544d4c20322e302f2f454e223e0a3c68746d6c3e3c686561643e0a3c7469746c653e3430302042616420526571756573743c2f7469746c653e0a3c2f686561643e3c626f64793e0a3c68313e42616420526571756573743c2f68313e0a3c703e596f75722062726f777365722073656e7420612072657175657374207468617420746869732073657276657220636f756c64206e6f7420756e6465727374616e642e3c6272202f3e0a3c2f703e0a3c68723e0a3c616464726573733e4170616368652f322e322e33202843656e744f532920536572766572206174203137322e32362e3135352e333620506f72742038303c2f616464726573733e0a3c2f626f64793e3c2f68746d6c3e0a
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:475) ~[netty-all-4.1.39.Final.jar:4.1.39.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283) ~[netty-all-4.1.39.Final.jar:4.1.39.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-all-4.1.39.Final.jar:4.1.39.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-all-4.1.39.Final.jar:4.1.39.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-all-4.1.39.Final.jar:4.1.39.Final]
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1421) [netty-all-4.1.39.Final.jar:4.1.39.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-all-4.1.39.Final.jar:4.1.39.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-all-4.1.39.Final.jar:4.1.39.Final]
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) [netty-all-4.1.39.Final.jar:4.1.39.Final]
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-all-4.1.39.Final.jar:4.1.39.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:697) [netty-all-4.1.39.Final.jar:4.1.39.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:632) [netty-all-4.1.39.Final.jar:4.1.39.Final]
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:549) [netty-all-4.1.39.Final.jar:4.1.39.Final]
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:511) [netty-all-4.1.39.Final.jar:4.1.39.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918) [netty-all-4.1.39.Final.jar:4.1.39.Final]
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-all-4.1.39.Final.jar:4.1.39.Final]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_231]
Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 3c21444f43545950452048544d4c205055424c494320222d2f2f494554462f2f4454442048544d4c20322e302f2f454e223e0a3c68746d6c3e3c686561643e0a3c7469746c653e3430302042616420526571756573743c2f7469746c653e0a3c2f686561643e3c626f64793e0a3c68313e42616420526571756573743c2f68313e0a3c703e596f75722062726f777365722073656e7420612072657175657374207468617420746869732073657276657220636f756c64206e6f7420756e6465727374616e642e3c6272202f3e0a3c2f703e0a3c68723e0a3c616464726573733e4170616368652f322e322e33202843656e744f532920536572766572206174203137322e32362e3135352e333620506f72742038303c2f616464726573733e0a3c2f626f64793e3c2f68746d6c3e0a
    at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1203) ~[netty-all-4.1.39.Final.jar:4.1.39.Final]
    at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1271) ~[netty-all-4.1.39.Final.jar:4.1.39.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:505) ~[netty-all-4.1.39.Final.jar:4.1.39.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:444) ~[netty-all-4.1.39.Final.jar:4.1.39.Final]
    ... 16 more
ericbeland commented 4 years ago

We're rewriting to remove the netty/littleproxy dependency (where this is happening) and will instead be using MITMproxy. Closing this for now as the code where it is happening is getting bulldozed.

amjd commented 4 years ago

@ericbeland: Sorry for the off-topic question, but would appreciate if you could answer this: Would switching to MITMproxy still keep the BUP Java API and features mostly intact or there's going to be significant change in functionality?

ericbeland commented 4 years ago

That's a great question. This will be a breaking API change. Intention-wise, the goal was not to drop functionality unless forced to. Areas we've been forced to that I remember offhand might include dynamic filter limitations, slowing down requests for bandwidth simulation, allowlist/blocklist (changes, not removal). There could be more.

On the flip side, we will work with HTTP/2 and websockets correctly which is a necessity these days.

amjd commented 4 years ago

That doesn't sound too bad. I'm sure the pros outweigh the cons, and MITMproxy should make for a more stable (and actively maintained) base to build on.

Gordiievskyi commented 4 years ago

@ericbeland I use black list to prevent some overlays to load, is this functionality absent at all or just has changed endpoint and/or params ? Same with filters - I use them to simulate 'disable cache' so question is same.

ericbeland commented 4 years ago

It might be changed endpoint or params. We need allow/block for ourselves, so I'd anticipate that will be there. I'd have to look through on the disable cache question.

Gordiievskyi commented 4 years ago

@ericbeland Its a good news will wait for the filter updates

amjd commented 4 years ago

@Gordiievskyi How do you use filters to disable cache?

Gordiievskyi commented 4 years ago

@amjd https://stackoverflow.com/questions/46791043/selenium-chrome-driver-without-cache/62701856#62701856