Closed bigwheel closed 7 years ago
thanks for trying flashback, i will take a look, there might be something wrong with the keys that generated.
Thank you for reply. As this instruction ( https://jamielinux.com/docs/openssl-certificate-authority/create-the-root-pair.html ), I started following key generate command. But result was same time out.
openssl genrsa -aes256 -out private-key.pem 4096
Could you tell me a successful example for https ?
i did it a while back by following this wiki, let me redo it again tomorrow and have an example cert file for you to try. If i remember correctly, i was using the intermediate cert.
@bigwheel there is possible something wrong in the Flashback-Admin for passing in the parameters to Flashback (something that being used in ConnectionFlowFactory --> createFullHttpsConnectionFlow), I will need to talk with @sf1152 tomorrow to see what's the best way to fix it. will update you as soon as I get back from him. sorry for the inconvenience.
Good news! I don't need this quickly, then no need temporary solution. When problem is fixed, I must try it in my project :)
@bigwheel I followed exactly what the wiki does: https://jamielinux.com/docs/openssl-certificate-authority/create-the-root-pair.html, for root and intermediate certificates. (with only the change of "Alice Ltd" --> FLASHBACK which i don't think matters. for all password i keep using the same as in that doc "secretpassword").
then one extra step is to generate the pkcs intermediate ca
openssl pkcs12 -export -out intermediate/intermediate.p12 -inkey intermediate/private/intermediate.key.pem -in intermediate/certs/intermediate.cert.pem -name "flashbackca"
in your flashback folder, start the admin server
./startAdminServer.sh -port 1234
In another console, start flashback
curl "http://localhost:1234/admin?action=startFlashback" --data '{ "sceneMode": "record", "sceneName": "flashbacktest1", "matchRule": "matchEntireRequest", "scenePath": "/tmp", "proxyHost": "localhost", "proxyPort": "5555", "caCertPath": "<absolute path to where you generated your own CA>/intermediate/intermediate.p12", "caCertPwd": "secretpassword", "caAlias": "flashbackca", "caKeyPwd": "secretpassword", "caCertCN": "", "caCertOU": "", "caCertO": "", "caCertL": "", "caCertCC": "" }'
curl google.com
curl --cacert intermediate/certs/intermediate.cert.pem --cert intermediate/intermediate.p12:secretpassword --proxy localhost:5555 -L 'https://www.google.com/'
shutdown flashback
curl "http://localhost:1234/admin?action=shutDownFlashback" --data '{}'
check the recorded
vi /tmp/flashbacktest1
Let me know if this works. I will go ahead and close this issue, feel free to reopen it if still not working.
I really appreciated you are using flashback and actually helped us find couple of issues in the admin/proxy code (everything is working with the above instruction, just enhancement and improvements), i will create new issues and ask help from @sf1152 to fix.
I went through the steps exactly as you said and suggested (but doing with Scala instead) and I get these errors. Any ideas on what might be going on?
`18/06/22 15:18:52 INFO steps.StopReadingFromClient: Stop reading from client 18/06/22 15:18:52 ERROR channel.ServerChannelHandler: Caught exception on server channel io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: General SSLEngine problem at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:346) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:229) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) at java.lang.Thread.run(Thread.java:748) Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1529) at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535) at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:813) at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781) at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1114) at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:981) at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:934) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:315) ... 11 more Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728) at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330) at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322) at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1614) at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216) at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052) at sun.security.ssl.Handshaker$1.run(Handshaker.java:992) at sun.security.ssl.Handshaker$1.run(Handshaker.java:989) at java.security.AccessController.doPrivileged(Native Method) at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467) at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1148) at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1003) ... 13 more Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397) at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302) at sun.security.validator.Validator.validate(Validator.java:260) at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281) at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136) at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1601) ... 21 more Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280) at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392) ... 27 more
Unexpected end of file from server java.net.SocketException: Unexpected end of file from server at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:851) at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678) at sun.net.www.protocol.http.HttpURLConnection.doTunneling(HttpURLConnection.java:2055) at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:183) at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:162) at scalaj.http.DefaultConnectFunc$.apply(Http.scala:471) at scalaj.http.DefaultConnectFunc$.apply(Http.scala:469) at scalaj.http.HttpRequest.scalaj$http$HttpRequest$$doConnection(Http.scala:355) at scalaj.http.HttpRequest.exec(Http.scala:335) at scalaj.http.HttpRequest.asString(Http.scala:455) at com.redventures.databroker.data.HailoClient$.ping(HailoClient.scala:13) at com.redventures.databroker.data.HailoClientSpec$$anonfun$1.apply(HailoClientSpec.scala:12) at com.redventures.databroker.data.HailoClientSpec$$anonfun$1.apply(HailoClientSpec.scala:11) at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85) at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104) at org.scalatest.Transformer.apply(Transformer.scala:22) at org.scalatest.Transformer.apply(Transformer.scala:20) at org.scalatest.FlatSpecLike$$anon$1.apply(FlatSpecLike.scala:1682) at org.scalatest.TestSuite$class.withFixture(TestSuite.scala:196) at org.scalatest.FlatSpec.withFixture(FlatSpec.scala:1685) at org.scalatest.FlatSpecLike$class.invokeWithFixture$1(FlatSpecLike.scala:1679) at org.scalatest.FlatSpecLike$$anonfun$runTest$1.apply(FlatSpecLike.scala:1692) at org.scalatest.FlatSpecLike$$anonfun$runTest$1.apply(FlatSpecLike.scala:1692) at org.scalatest.SuperEngine.runTestImpl(Engine.scala:289) at org.scalatest.FlatSpecLike$class.runTest(FlatSpecLike.scala:1692) at org.scalatest.FlatSpec.runTest(FlatSpec.scala:1685) at org.scalatest.FlatSpecLike$$anonfun$runTests$1.apply(FlatSpecLike.scala:1750) at org.scalatest.FlatSpecLike$$anonfun$runTests$1.apply(FlatSpecLike.scala:1750) at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:396) at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:384) at scala.collection.immutable.List.foreach(List.scala:381) at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:384) at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:373) at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:410) at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:384) at scala.collection.immutable.List.foreach(List.scala:381) at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:384) at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:379) at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:461) at org.scalatest.FlatSpecLike$class.runTests(FlatSpecLike.scala:1750) at org.scalatest.FlatSpec.runTests(FlatSpec.scala:1685) at org.scalatest.Suite$class.run(Suite.scala:1147) at org.scalatest.FlatSpec.org$scalatest$FlatSpecLike$$super$run(FlatSpec.scala:1685) at org.scalatest.FlatSpecLike$$anonfun$run$1.apply(FlatSpecLike.scala:1795) at org.scalatest.FlatSpecLike$$anonfun$run$1.apply(FlatSpecLike.scala:1795) at org.scalatest.SuperEngine.runImpl(Engine.scala:521) at org.scalatest.FlatSpecLike$class.run(FlatSpecLike.scala:1795) at com.redventures.flashback.FlashbackFlatSpec.org$scalatest$BeforeAndAfterAll$$super$run(FlashbackFlatSpec.scala:15) at org.scalatest.BeforeAndAfterAll$class.liftedTree1$1(BeforeAndAfterAll.scala:213) at org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:210) at com.redventures.flashback.FlashbackFlatSpec.run(FlashbackFlatSpec.scala:15) at org.scalatest.tools.SuiteRunner.run(SuiteRunner.scala:45) at org.scalatest.tools.Runner$$anonfun$doRunRunRunDaDoRunRun$1.apply(Runner.scala:1340) at org.scalatest.tools.Runner$$anonfun$doRunRunRunDaDoRunRun$1.apply(Runner.scala:1334) at scala.collection.immutable.List.foreach(List.scala:381) at org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:1334) at org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:1011) at org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:1010) at org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:1500) at org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:1010) at org.scalatest.tools.Runner$.run(Runner.scala:850) at org.scalatest.tools.Runner.run(Runner.scala) at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.runScalaTest2(ScalaTestRunner.java:131) at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.main(ScalaTestRunner.java:28)
`
I assume you are running in record model. In record model, there are two trust stores. If you client code is running in the same process of Flashback, you need make sure you start flashback first before you point client to your keystore. If not, you might end up using your customized keystore to authenticate with third-party certificate.
@sf1152 So I start flashback first after newing up a new FlashbackRunner.Builder() and call the method start(), do I need to import this keystore into java somehow? Similar to this post? https://blog.codecentric.de/en/2013/01/how-to-use-self-signed-pem-client-certificates-in-java/
I read How to Record and Replay an HTTPS Transaction · linkedin/flashback Wiki and tried to proxy https request. But it didn't work correctly for time out.
Environemnt
Reproduce procedure
Above commands succeed entirely, however,
Result
And console executes "./startAdminServer.sh -port 1234" doesn't show any logs.
I think my prrocedure is not correct for flashback, but what's wrong ?