PerfKitBenchmarker roadmap

[mateusz-blaszkowski]

Do you have any list of planned features you are going to work on in the future (kind of a public roadmap)? What about the Docker support proposal (https://docs.google.com/document/d/1XEmo8OGd7tJIwR89yDkUXeXB13zXfVNUaczveAmWSLY/edit)? I'm trying to figure out in what direction PKB is going.

Thanks!

[voellm]

Mateusz,

Thanks for asking. I'll include the roadmap and ask Carlos to discuss the Docker Swarm proposal during the community meeting this month.

There are two parts to the roadmap;

• The technical part such as declaring V1, adding more Windows Workloads, moving up the Cloud Stack to PaaS and SaaS services, scoring, etc.
• The stewardship path. Google committed to the first year to run it. What I'd like to do is figure out should we do it for another year or look to move the project to its own foundation or Apache so we can open up to more committers after year one.

We are 7 months after the first open release and the pickup and community growth has been pretty awesome. I'd like to keep that up because Openness around Cloud Performance really matters.

Tony

On Thu, Oct 8, 2015 at 4:21 AM, Mateusz Blaszkowski < notifications@github.com> wrote:

Do you have any list of planned features you are going to work on in the future (kind of a public roadmap)? What about the Docker support proposal ( https://docs.google.com/document/d/1XEmo8OGd7tJIwR89yDkUXeXB13zXfVNUaczveAmWSLY/edit)? I'm trying to figure out in what direction PKB is going.

Thanks!

— Reply to this email directly or view it on GitHub https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/issues/517.

Anthony F. Voellm (aka Tony) Google Voice: (650) 516-7382 https://www.google.com/voice/b/0?pli=1#phones Blog: http://perfguy.blogspot.com/ Benchmark like a pro... PerfKitBenchmarker https://github.com/GoogleCloudPlatform/PerfKitBenchmarker

[voellm]

I'll resolve this after https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/issues/507 is complete.

[meteorfox]

@voellm @mateusz-blaszkowski

I will be more than happy to talk about the Docker Swarm provider proposal on the community meeting.

But in summary, the Docker Swarm proposal at the moment is just that, a proposal, which I'm requesting comments from the PerfKit community to see if there's interest in such feature. There are really 2 parts to the proposal, which I should probably split them in two proposal since they are independent.

The first part, I think is non-controversial and trivial to implement, it would just add Docker Swarm as a provider, and it would work just like the Kubernetes provider.

The second part would require several changes, but it would not change how you use PKB Today. The way I envision it is that the container features will be orthogonal to the current features, essentially, just like PKB talks to a cloud provider to build a VM, it would instead talk directly to a container 'cluster manager' (e.g. AWS ECS, Docker-swarm, Kubernetes, etc) to create/delete the containers, and manage their configuration (e.g. exposing ports, ambassador containers, attach data containers as volumes, etc). Ideally, it would use Docker images that have been pre-built from Dockerfiles, and they will try to bake-in containers' best practices where possible. (e.g. No SSH, distinguish privileged vs non-privileged, data containers, etc).

[mateusz-blaszkowski]

Thank you guys for answers. That clarifies some of my thoughts.

@voellm - we (Intel) have some ideas for extending PKB (especially for private cloud usage) but this may require some help from the community to achieve this. I will try to gather a list of items that we see could be valuable. Let's see if these items coincide with your needs.

@meteorfox - that would be great to see your changes. I agree that containers support should not change the way PKB is used currently. It may be delivered as a separate feature. We plan to work on Mesos&Marathon provider and we will again face the same issues. SSH is not the best candidate for containers but it is the easiest way with the current implementation of PKB. The first part you propose should be pretty straightforward, but the second part might require a lot of work in Perkit framework structure. However, I'm definitely looking forward to the changes :) Having these pre-built and configured Docker images would be awesome!

[voellm]

@Mateusz One thing we have talked about doing is factoring out the transport (communications) layer to make it more extensible. For example where SSH is not an option or available you could tunnel commands over an object store or filesytem in a message passing manner. For system that can inject messages over serial we can use that, etc. We also have the need in some circumstances.

Feel free to put your ideas into the up coming community deck at the end of your Kubernetes section.

On Thu, Oct 15, 2015 at 6:51 AM, Mateusz Blaszkowski < notifications@github.com> wrote:

Thank you guys for answers. That clarifies some of my thoughts.

@voellm https://github.com/voellm - we (Intel) have some ideas for extending PKB (especially for private cloud usage) but this may require some help from the community to achieve this. I will try to gather a list of items that we see could be valuable. Let's see if these items coincide with your needs.

@meteorfox https://github.com/meteorfox - that would be great to see your changes. I agree that containers support should not change the way PKB is used currently. It may be delivered as a separate feature. We plan to work on Mesos&Marathon provider and we will again face the same issues. SSH is not the best candidate for containers but it is the easiest way with the current implementation of PKB. The first part you propose should be pretty straightforward, but the second part might require a lot of work in Perkit framework structure. However, I'm definitely looking forward to the changes :) Having these pre-built and configured Docker images would be awesome!

— Reply to this email directly or view it on GitHub https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/issues/517#issuecomment-148393174 .

Anthony F. Voellm (aka Tony) Google Voice: (650) 516-7382 https://www.google.com/voice/b/0?pli=1#phones Blog: http://perfguy.blogspot.com/ Benchmark like a pro... PerfKitBenchmarker https://github.com/GoogleCloudPlatform/PerfKitBenchmarker

[mateusz-blaszkowski]

As I said on the community meeting - I'm copying the list of improvements which are important for us:

• Possibility to run at scale (up to 1000 nodes with thousands of instances). Current potential bottlenecks:
  • SSH (https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/issues/550) - generating dozens of thousands of SSH request would be very costly (if not impossible)
  • logs (800 mb of logs when running redis with 100 redis client instances)
  • CPU consumption on the host which is running the benchmarks (Load average ~1.1 when running redis with 100 redis client instances)
• Accept failures
  • Don't stop the run if one of the benchmarks failed (#573)
  • Present failure rate at the end of a run
• Synchronize multiple benchmarks to be run on the same time (overlapping workloads - not sure if there's an issue for this)
• Leverage scheduling policies (spreading, stacking, random, round-robin, etc.)
  • Misrepresented results when instances are scheduled on the same host (for i.e. SCP - when we copy the data between two instances running on the same host we get extraordinary results - much higher than the network throughput ;) )
• Spawn new instances during benchmark execution
  • One of the interesting test scenarios would be to verify how many instances we are able to spawn without affecting other instances
• Generate cluster noise (benchmarks running in a loop or for a specified period of time)
• Gather only a subset of metrics
• Repeat the test several times
• Pluginable mechanism for storing the results
  • Adding another publisher should be as simple as adding a new class and changing a flag parameter. Curently we would need also changes in a core (in publusher.py)

[voellm]

Great feedback.

Laid out the next quarters idea here - https://docs.google.com/presentation/d/1MkMPxYN1rgIPXy0576UUVGGnUIPaJavFnY1aWfJqh14/edit

In terms of commitments #550 high on my list and we'll look to do that Jan - Mar - Q1 for Google if not before.

573 is currently being worked by @skschneider. He has a good model for how to handle it and working code.

Added #601 for the high CPU load you mention. Added #602 for the log size issue.

436 is the tracking issue for multiple benchmarks

For other issues high on your list please open GitHub issues so we can track them individually and its easier for the community to see what is needed.

[voellm]

In terms of tracking when things will happen we should use the "Milestone" flags on issue. When something is being worked or planned the target release can be set. Sound like a good idea?

[mateusz-blaszkowski]

Thank you @voellm for the update.

For me "Milestone" flags will be perfect as long as they are updated regularly.

[voellm]

Sounds good on the Milestone. I've move bugs to target them.

To help lay things out here is an architecture diagram - https://drive.google.com/file/d/0B5iICDqrB_E9eE1pbTJDSGhiUlU/view

Here is a also a guesstimate on changes laid out on a timeline... https://drive.google.com/file/d/0B5iICDqrB_E9OFlpYU0zaFNlT1k/view

Based on this I'm going to close the issue.