Order of magnitude performance degradation when TLS is used to connect to MySQL

jvshahid commented 6 years ago

Currently:

Turning on TLS causes significant slowdown compared with using plain sockets.

Expected:

Using TLS shouldn't be an order of magnitude slower, see below for more context.

Context

This is a follow up story to the investigation that the Diego team just finished in this story. The team noticed that the benchmark test suit doesn't pass when TLS is turned on and we had that story to investigate that behavior.

After a few days of investigating the slowness introduced by TLS we came across this article. The article mentions a bottleneck in ASIO (boost asynchronous IO library that mysql use) when used with OpenSSL. The blog post is pretty old (from August 2015), but I think pxc-release is using a very old version of boost (version 1.59.0) that predates the article. Boost 1.59.0 was released in March 2015, which makes it likely to include the issues mentioned in the post.

We tried to run the benchmark after changing the pxc packaging script to use the embedded & bundled SSL library called yassl instead of openssl and the results are very promising. The trade off of using yassl is that it doesn't support TLS 1.2 and doesn't include all cipher suites.

Given that this requires changes to the pxc-release, I am opening this issue to keep track of that change as well as transfer the context that we acquired.

Proposed solution/feature

Use yassl instead of OpenSSL
Upgrade to a more recent version of Boost, given that it fixes the problems mentioned in the article

The benchmark we are using can be found here and the Diego team is happy to XTP on setting up CI jobs to run it. The BBS and the benchmark emit datadog metrics that could be useful during the course of this investigation. We also have historical benchmark results if you would like to compare your results to ours.

Would you want to open an PR for this feature?

Yes

Deployment Context:

Please provide relevant details about your deployment. That might include:

Are you using cf-mysql-release with a service broker? No
How many nodes and proxies are deployed? single node, In fact we tried skipping the proxies and connecting to the mysql node using port 13306 and saw no significant improvement.
Release version: we tried 0.13.0 as well as current master
Other bosh releases deployed on the same vms: Nothing besides what's is in the use-pxc.yml ops file.

Reference:

Provided above in the Context section

cf-gitbot commented 6 years ago

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/159669487

The labels on this github issue will be updated when the story is started.

abg commented 2 years ago

To close this issue off, we had discovered a performance bottleneck in OpenSSL v1.0.x as provided by the ubuntu-xenial stemcells.

This issue manifests with particular high concurrency workloads with TLS enabled. For OSS consumers of this release, this can be avoided by using the ubuntu bionic stemcell.

I am going to close this issue for now.

cloudfoundry / pxc-release