search

GoogleCloudPlatform / PerfKitBenchmarker

PerfKit Benchmarker (PKB) contains a set of benchmarks to measure and compare cloud offerings. The benchmarks use default settings to reflect what most users will see. PerfKit Benchmarker is licensed under the Apache 2 license terms. Please make sure to read, understand and agree to the terms of the LICENSE and CONTRIBUTING files before proceeding.
https://googlecloudplatform.github.io/PerfKitBenchmarker/
Apache License 2.0
1.9k stars 510 forks source link

Control for OS/Package/Compiler differences #150

Open timothyeharris opened 9 years ago

timothyeharris commented 9 years ago

Would it make sense to create a "canonical" .VHD (and docker image) to include in with PerfKitBenchmarker that has all the appropriate libraries, compilers, and benchmark packages pre-installed?

Since it's possible to upload and use your own VHD/Container with all the cloud providers, this would make it possible to eliminate base OS differences as a source of problems with the benchmarks, and a source of argument for benchmark variability.

A pre-req for using perfkitbenchmarker could then be to upload the canonical image into your account/subscription for use by the tool.

(There's nothing really stopping individuals from doing this today, I suppose. Just wondering if a "blessed" version makes sense)

voellm commented 9 years ago

We are staying away from custom images because we want to see the perf of the vendor provided images.

The hiccups in config are things we can communicate to Canonical to fix on their end. On Mar 6, 2015 12:53 AM, "Tim Harris" notifications@github.com wrote:

Would it make sense to create a "canonical" .VHD (and docker image) to include in with PerfKitBenchmarker that has all the appropriate libraries, compilers, and benchmark packages pre-installed?

Since it's possible to upload and use your own VHD/Container with all the cloud providers, this would make it possible to eliminate base OS differences as a source of problems with the benchmarks, and a source of argument for benchmark variability.

A pre-req for using perfkitbenchmarker could then be to upload the canonical image into your account/subscription for use by the tool.

(There's nothing really stopping individuals from doing this today, I suppose. Just wondering if a "blessed" version makes sense)

— Reply to this email directly or view it on GitHub https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/issues/150.

timothyeharris commented 9 years ago

Fair enough. The biggest hits I'm seeing to end-to-end runtime right now are coming from "getting the environment set up" -- because of packages not being installed correctly on the vendor supplied images. By "canonical" I didn't mean the vendor, rather canon in the religious sense :-)

voellm commented 9 years ago

Which in itself is interesting in a way. On Mar 7, 2015 7:00 PM, "Tim Harris" notifications@github.com wrote:

Fair enough. The biggest hits I'm seeing to end-to-end runtime right now are coming from "getting the environment set up" -- because of packages not being installed correctly on the vendor supplied images.

— Reply to this email directly or view it on GitHub https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/issues/150#issuecomment-77701096 .

voellm commented 9 years ago

Before I close this out I should mention there is a --image option for using custom images. What we dont have is the ability to ignore package install.

Should we change this to a feature request to ignore installing packages to allow truly custom images?

timothyeharris commented 9 years ago

I am planning to use the --image option to handle this for own use case, yes :-) I think there are two ways to go: 1) A flag to not install packages -- this would be fine for me. 2) Check to see if a package is already installed, and only attempt installation if it's not. -- This would also work for me, and would probably be a little more robust.

voellm commented 9 years ago

Let's go the check if already installed route. Thought we already did this.

cmccoy commented 9 years ago

We do verify that packages aren't installed multiple times during the same run, and package installs will be a noop if the system already has it. Anything compiled from source will always be installed, however.

timothyeharris commented 9 years ago

It might... (or it may just rely on the package installer to say "I already have this")

I'm looking for a way to make it easy to compare results between different instance types and know that what went into the end to end runtime is the same... i.e. I don't want to compare a result where I have the overhead of package installs on one instance and not on another....

For my use case, taking the --image route will solve this problem.