linkedin / flashback

mock the internet
BSD 2-Clause "Simplified" License
578 stars 54 forks source link

Https requests don't work for request time out #8

Closed bigwheel closed 7 years ago

bigwheel commented 7 years ago

I read How to Record and Replay an HTTPS Transaction · linkedin/flashback Wiki and tried to proxy https request. But it didn't work correctly for time out.

Environemnt

Reproduce procedure

openssl genrsa 1024 > private-key.pem
openssl rsa -in private-key.pem -pubout -out public-key.pem
openssl req -new -key private-key.pem > my-request.csr # enter for all dialogs, then no password
openssl x509 -req -in my-request.csr -signkey private-key.pem -out public-key.crt -days 3650
openssl pkcs12 -export -in public-key.crt -inkey private-key.pem -out my-identity.p12

# add certificate to os root certificates
sudo cp public-key.crt /usr/share/ca-certificates
sudo echo "public-key.crt" >> /etc/ca-certificats.conf
sudo update-ca-certificates

./startAdminServer.sh -port 1234

# in another console
curl "http://localhost:1234/admin?action=startFlashback" --data '{
  "sceneMode": "record",
  "sceneName": "test1",
  "matchRule": "matchEntireRequest",
  "scenePath": "./scene",
  "proxyHost": "localhost",
  "proxyPort": "5564",
  "caCertPath": "absolute path of my-identity.p12",
  "caCertPwd": "",
  "caAlias": "",
  "caKeyPwd": "",
  "caCertOU": "",
  "caCertO": "",
  "caCertCN": "",
  "caCertL": "",
  "caCertCC": ""
}'

Above commands succeed entirely, however,

Result

curl --proxy localhost:5564 -L 'https://www.google.co.jp/'
curl: (28) Operation timed out after 300570 milliseconds with 0 out of 0 bytes received

And console executes "./startAdminServer.sh -port 1234" doesn't show any logs.

I think my prrocedure is not correct for flashback, but what's wrong ?

ben-oldcom commented 7 years ago

thanks for trying flashback, i will take a look, there might be something wrong with the keys that generated.

bigwheel commented 7 years ago

Thank you for reply. As this instruction ( https://jamielinux.com/docs/openssl-certificate-authority/create-the-root-pair.html ), I started following key generate command. But result was same time out.

openssl genrsa -aes256 -out private-key.pem 4096

Could you tell me a successful example for https ?

ben-oldcom commented 7 years ago

i did it a while back by following this wiki, let me redo it again tomorrow and have an example cert file for you to try. If i remember correctly, i was using the intermediate cert.

ben-oldcom commented 7 years ago

@bigwheel there is possible something wrong in the Flashback-Admin for passing in the parameters to Flashback (something that being used in ConnectionFlowFactory --> createFullHttpsConnectionFlow), I will need to talk with @sf1152 tomorrow to see what's the best way to fix it. will update you as soon as I get back from him. sorry for the inconvenience.

bigwheel commented 7 years ago

Good news! I don't need this quickly, then no need temporary solution. When problem is fixed, I must try it in my project :)

ben-oldcom commented 7 years ago

@bigwheel I followed exactly what the wiki does: https://jamielinux.com/docs/openssl-certificate-authority/create-the-root-pair.html, for root and intermediate certificates. (with only the change of "Alice Ltd" --> FLASHBACK which i don't think matters. for all password i keep using the same as in that doc "secretpassword").

then one extra step is to generate the pkcs intermediate ca openssl pkcs12 -export -out intermediate/intermediate.p12 -inkey intermediate/private/intermediate.key.pem -in intermediate/certs/intermediate.cert.pem -name "flashbackca" in your flashback folder, start the admin server ./startAdminServer.sh -port 1234

In another console, start flashback curl "http://localhost:1234/admin?action=startFlashback" --data '{ "sceneMode": "record", "sceneName": "flashbacktest1", "matchRule": "matchEntireRequest", "scenePath": "/tmp", "proxyHost": "localhost", "proxyPort": "5555", "caCertPath": "<absolute path to where you generated your own CA>/intermediate/intermediate.p12", "caCertPwd": "secretpassword", "caAlias": "flashbackca", "caKeyPwd": "secretpassword", "caCertCN": "", "caCertOU": "", "caCertO": "", "caCertL": "", "caCertCC": "" }' curl google.com curl --cacert intermediate/certs/intermediate.cert.pem --cert intermediate/intermediate.p12:secretpassword --proxy localhost:5555 -L 'https://www.google.com/' shutdown flashback curl "http://localhost:1234/admin?action=shutDownFlashback" --data '{}' check the recorded vi /tmp/flashbacktest1

Let me know if this works. I will go ahead and close this issue, feel free to reopen it if still not working.

I really appreciated you are using flashback and actually helped us find couple of issues in the admin/proxy code (everything is working with the above instruction, just enhancement and improvements), i will create new issues and ask help from @sf1152 to fix.

thestephenstanton commented 6 years ago

I went through the steps exactly as you said and suggested (but doing with Scala instead) and I get these errors. Any ideas on what might be going on?

`18/06/22 15:18:52 INFO steps.StopReadingFromClient: Stop reading from client 18/06/22 15:18:52 ERROR channel.ServerChannelHandler: Caught exception on server channel io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: General SSLEngine problem at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:346) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:229) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) at java.lang.Thread.run(Thread.java:748) Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1529) at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535) at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:813) at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781) at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1114) at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:981) at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:934) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:315) ... 11 more Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728) at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330) at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322) at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1614) at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216) at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052) at sun.security.ssl.Handshaker$1.run(Handshaker.java:992) at sun.security.ssl.Handshaker$1.run(Handshaker.java:989) at java.security.AccessController.doPrivileged(Native Method) at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467) at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1148) at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1003) ... 13 more Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397) at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302) at sun.security.validator.Validator.validate(Validator.java:260) at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281) at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136) at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1601) ... 21 more Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280) at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392) ... 27 more

Unexpected end of file from server java.net.SocketException: Unexpected end of file from server at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:851) at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678) at sun.net.www.protocol.http.HttpURLConnection.doTunneling(HttpURLConnection.java:2055) at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:183) at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:162) at scalaj.http.DefaultConnectFunc$.apply(Http.scala:471) at scalaj.http.DefaultConnectFunc$.apply(Http.scala:469) at scalaj.http.HttpRequest.scalaj$http$HttpRequest$$doConnection(Http.scala:355) at scalaj.http.HttpRequest.exec(Http.scala:335) at scalaj.http.HttpRequest.asString(Http.scala:455) at com.redventures.databroker.data.HailoClient$.ping(HailoClient.scala:13) at com.redventures.databroker.data.HailoClientSpec$$anonfun$1.apply(HailoClientSpec.scala:12) at com.redventures.databroker.data.HailoClientSpec$$anonfun$1.apply(HailoClientSpec.scala:11) at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85) at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104) at org.scalatest.Transformer.apply(Transformer.scala:22) at org.scalatest.Transformer.apply(Transformer.scala:20) at org.scalatest.FlatSpecLike$$anon$1.apply(FlatSpecLike.scala:1682) at org.scalatest.TestSuite$class.withFixture(TestSuite.scala:196) at org.scalatest.FlatSpec.withFixture(FlatSpec.scala:1685) at org.scalatest.FlatSpecLike$class.invokeWithFixture$1(FlatSpecLike.scala:1679) at org.scalatest.FlatSpecLike$$anonfun$runTest$1.apply(FlatSpecLike.scala:1692) at org.scalatest.FlatSpecLike$$anonfun$runTest$1.apply(FlatSpecLike.scala:1692) at org.scalatest.SuperEngine.runTestImpl(Engine.scala:289) at org.scalatest.FlatSpecLike$class.runTest(FlatSpecLike.scala:1692) at org.scalatest.FlatSpec.runTest(FlatSpec.scala:1685) at org.scalatest.FlatSpecLike$$anonfun$runTests$1.apply(FlatSpecLike.scala:1750) at org.scalatest.FlatSpecLike$$anonfun$runTests$1.apply(FlatSpecLike.scala:1750) at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:396) at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:384) at scala.collection.immutable.List.foreach(List.scala:381) at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:384) at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:373) at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:410) at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:384) at scala.collection.immutable.List.foreach(List.scala:381) at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:384) at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:379) at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:461) at org.scalatest.FlatSpecLike$class.runTests(FlatSpecLike.scala:1750) at org.scalatest.FlatSpec.runTests(FlatSpec.scala:1685) at org.scalatest.Suite$class.run(Suite.scala:1147) at org.scalatest.FlatSpec.org$scalatest$FlatSpecLike$$super$run(FlatSpec.scala:1685) at org.scalatest.FlatSpecLike$$anonfun$run$1.apply(FlatSpecLike.scala:1795) at org.scalatest.FlatSpecLike$$anonfun$run$1.apply(FlatSpecLike.scala:1795) at org.scalatest.SuperEngine.runImpl(Engine.scala:521) at org.scalatest.FlatSpecLike$class.run(FlatSpecLike.scala:1795) at com.redventures.flashback.FlashbackFlatSpec.org$scalatest$BeforeAndAfterAll$$super$run(FlashbackFlatSpec.scala:15) at org.scalatest.BeforeAndAfterAll$class.liftedTree1$1(BeforeAndAfterAll.scala:213) at org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:210) at com.redventures.flashback.FlashbackFlatSpec.run(FlashbackFlatSpec.scala:15) at org.scalatest.tools.SuiteRunner.run(SuiteRunner.scala:45) at org.scalatest.tools.Runner$$anonfun$doRunRunRunDaDoRunRun$1.apply(Runner.scala:1340) at org.scalatest.tools.Runner$$anonfun$doRunRunRunDaDoRunRun$1.apply(Runner.scala:1334) at scala.collection.immutable.List.foreach(List.scala:381) at org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:1334) at org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:1011) at org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:1010) at org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:1500) at org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:1010) at org.scalatest.tools.Runner$.run(Runner.scala:850) at org.scalatest.tools.Runner.run(Runner.scala) at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.runScalaTest2(ScalaTestRunner.java:131) at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.main(ScalaTestRunner.java:28)

`

sf1152 commented 6 years ago

I assume you are running in record model. In record model, there are two trust stores. If you client code is running in the same process of Flashback, you need make sure you start flashback first before you point client to your keystore. If not, you might end up using your customized keystore to authenticate with third-party certificate.

thestephenstanton commented 6 years ago

@sf1152 So I start flashback first after newing up a new FlashbackRunner.Builder() and call the method start(), do I need to import this keystore into java somehow? Similar to this post? https://blog.codecentric.de/en/2013/01/how-to-use-self-signed-pem-client-certificates-in-java/