search

GoogleCloudPlatform / PerfKitBenchmarker

PerfKit Benchmarker (PKB) contains a set of benchmarks to measure and compare cloud offerings. The benchmarks use default settings to reflect what most users will see. PerfKit Benchmarker is licensed under the Apache 2 license terms. Please make sure to read, understand and agree to the terms of the LICENSE and CONTRIBUTING files before proceeding.
https://googlecloudplatform.github.io/PerfKitBenchmarker/
Apache License 2.0
1.91k stars 512 forks source link

Run snapshots with full setup instead of installing all from scratch #430

Open kivio opened 9 years ago

kivio commented 9 years ago

In more complicated benchmarks like Cassandra YCSB when i want to run 20 nodes of cassandra process of installation take too much time.

My idea is to install just one VM from scratch and next make snapshot and run other from there. We can use for that https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/issues/429 for make sure installation do on each node just this steps who are required.

voellm commented 9 years ago

Improving the speed of provisioning is a great suggestion.

voellm commented 9 years ago

See refactor and lets make sure we desing for it. https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/issues/193

cmccoy commented 9 years ago

@kivio - I think snapshotting on the different cloud providers would be a bit difficult given the variation between APIs. What would you think about an image with PKB dependencies pre-installed?

Part of the issue with Cassandra and a large number of nodes is that startup can fail when multiple nodes join the cluster simultaneously (see CASSANDRA-7069). While we could improve the speed at which the nodes are provisioned and configured, I think there's still going to be a delay bringing up the cluster.

Upgrading Cassandra to 2.1 or 2.2 may improve things.

kivio commented 9 years ago

@voellm - i think it doesn't conflict with your refactor

@cmccoy - i don't known how it work in different cloud providers. This is just idea. Maybe is possible to full setup VM, next create image for them and setup another for that image?

My current problem is when i run cassandra YCSB on 46 vms with 20 seed nodes and 26 loaders nodes, process of preparing take me about 45 min, tests just 30. I looking for possibility to cut first phase to minimum (but without specify others - get my VM: pkb-cassandra-ycsb-ubuntu.raw for that test, install in your cloud and next run tests with it. Better for me is write, use ubuntu 14.04-cloudimage, and run tests).

cmccoy commented 9 years ago

Maybe is possible to full setup VM, next create image for them and setup another for that image?

During the benchmark setup? I think this is possible, but most setup currently runs in parallel across VMs, so I'm not sure that this would necessarily speed things up, and you would have the additional snapshotting / cloned VM boot times.

Opened #434 to track some low hanging fruit for the Cassandra benchmark.

voellm commented 9 years ago

It looks like the meta ask here is faster provisioning. Connor/Eric and I all discussed this for a bit yesterday. It looks like there a couple of things that can be done like Connor indicated but we can also break up the stages a little more.

So instead of prepare, execute, cleanup We can have prepare, provision_software, execute and cleanup

We can also look at having a mechanism to create a benchmark image that can be used later skipping the install of software altogether.

Just some ideas.

On Fri, Aug 21, 2015 at 7:22 AM, Connor McCoy notifications@github.com wrote:

Maybe is possible to full setup VM, next create image for them and setup another for that image?

During the benchmark setup? I think this is possible, but most setup currently runs in parallel across VMs, so I'm not sure that this would necessarily speed things up, and you would have the additional snapshotting / cloned VM boot times.

Opened #434 https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/issues/434 to track some low hanging fruit for the Cassandra benchmark.

— Reply to this email directly or view it on GitHub https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/issues/430#issuecomment-133441035 .

Anthony F. Voellm (aka Tony) Google Voice: (650) 516-7382 https://www.google.com/voice/b/0?pli=1#phones Blog: http://perfguy.blogspot.com/ Benchmark like a pro... PerfKitBenchmarker https://github.com/GoogleCloudPlatform/PerfKitBenchmarker