search

kubernetes / perf-tests

Performance tests and benchmarks
Apache License 2.0
892 stars 526 forks source link

Re-implement scheduling-throughput measurement #1027

Open mm4tt opened 4 years ago

mm4tt commented 4 years ago

Currently, we have a very crude logic for computing scheduling-throughput. We list all pods every 5s and see how many new pod very scheduled in that time. It doesn't work well in small tests where number of pods is low as we only get 1-2 representative 5s windows.

We should rewrite it to make it more reliable and accurate, e.g. use prometheus query to monitor rate of pod bindings

fejta-bot commented 4 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

wojtek-t commented 4 years ago

/remove-lifecycle stale

fejta-bot commented 4 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

wojtek-t commented 4 years ago

/remove-lifecycle stale

fejta-bot commented 3 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

wojtek-t commented 3 years ago

/remove-lifecycle stale

/assign @marseel

Since this is already happening - @marseel can you please update the issue with info what work is remaining to do?

k8s-ci-robot commented 3 years ago

@wojtek-t: GitHub didn't allow me to assign the following users: marseel.

Note that only kubernetes members, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see the contributor guide

In response to [this](https://github.com/kubernetes/perf-tests/issues/1027#issuecomment-720615008): >/remove-lifecycle stale > >/assign @marseel > >Since this is already happening - @marseel can you please update the issue with info what work is remaining to do? Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
marseel commented 3 years ago

/assign

marseel commented 3 years ago

Currently, new prometheus metric is already implemented and data is collected for all tests in master branch. It's worth to mention that previous SchedulingThroughput allowed us to gather metric with interval 1s, based on Prometheus we are only able to gather with interval 5s. The only thing remaining is turning on alerting for specific tests and removing old SchedulingThroughput. I've postponed this step, because it might have been conflicting with merging density with load test.

wojtek-t commented 3 years ago

only able to gather with interval 5s.

Unless we start scraping metrics more/less often. We may consider scraping more often potentially too.

I've postponed this step, because it might have been conflicting with merging density with load test.

Which is the right decision :) Though we're almost done with this merging, so we will be able to get back to it.

fejta-bot commented 3 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale

wojtek-t commented 3 years ago

/remove-lifecycle stale

fejta-bot commented 3 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale

wojtek-t commented 3 years ago

/remove-lifecycle stale

k8s-triage-robot commented 3 years ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

wojtek-t commented 3 years ago

/remove-lifecycle stale

dmatch01 commented 2 years ago

Hi @marseel Do you know if the old SchedulingThroughput was ever removed or is it still active? I still the source code in the main branch.

The only thing remaining is turning on alerting for specific tests and removing old SchedulingThroughput. I've postponed this step, because it might have been conflicting with merging density with load test.

marseel commented 2 years ago

Old SchedulingThroughput wasn't ever removed unfortunately. Currently, we are scraping metrics every 5 seconds for Prometheus in new measurement and old SchedulingThroughput has a better precision. I'm pretty sure it's possible to replace it, we just need to be really careful. Unfortunately I didn't have much time to work on it.

dmatch01 commented 2 years ago

Thanks @marseel for the quick response. Just to make sure I understand, the scheduling_throughput.go code is still being used to produce SchedulingThroughput metrics, is that correct? Regarding Prometheus, could you provide more details on how, if any, Prometheus is being used to produce the current SchedulingThroughput metrics? Thank you

Old SchedulingThroughput wasn't ever removed unfortunately. Currently, we are scraping metrics every 5 seconds for Prometheus in new measurement and old SchedulingThroughput has a better precision. I'm pretty sure it's possible to replace it, we just need to be really careful. Unfortunately I didn't have much time to work on it.

marseel commented 2 years ago

Currently, our load test measures scheduling throughput using both methods . For example for test run https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-scale-performance/1446158765446402048/ you can find both measurements' results:

dmatch01 commented 2 years ago

I see. It is interesting to see that max is different between the two reports. I have a pretty good understanding of the "Old" SchedulingThroughput after walking through the code. I'll look at the "New" one from Prometheus to understand the details. If you would like for me to share my findings with you please let me know the forum you prefer, e.g. here in this Issue or in Slack, DM or SIG channel. Thanks!

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

wojtek-t commented 2 years ago

/remove-lifecycle stale

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

marseel commented 2 years ago

/remove-lifecycle stale

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

wojtek-t commented 2 years ago

/remove-lifecycle stale

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

wojtek-t commented 1 year ago

/remove-lifecycle rotten

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

wojtek-t commented 1 year ago

/remove-lifecycle rotten

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

wojtek-t commented 1 year ago

/remove-lifecycle rotten

k8s-triage-robot commented 8 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

wojtek-t commented 8 months ago

/remove-lifecycle stale

k8s-triage-robot commented 5 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

wojtek-t commented 5 months ago

/remove-lifecycle stale

k8s-triage-robot commented 2 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

wojtek-t commented 2 months ago

/remove-lifecycle stale