search

kubernetes / sig-release

Repo for SIG release
Apache License 2.0
544 stars 392 forks source link

Define and collect input metrics about Kubernetes releases #1527

Open saschagrunert opened 3 years ago

saschagrunert commented 3 years ago

Target is to define a set of metrics around Kubernetes releases to elaborate on the reduced release cadence. This discussion came up during the KEP implementation phase. The planned survey outcome can help us to interpret the raw data later on, too.

The following questions have to be resolved before starting to collect the metrics:

/priority important-longterm /cc @aojea /help

aojea commented 3 years ago

We have to start defining which areas we want to measure, then define the metrics and the actions associated to each metrics, (I'm making up the examples), i.e.:

Quality:

ideally, everything should be automated and you just can build a dashboard with that, per example, the exception requests instead of an email can be an issue or a PR, or a form, so you can automate it.

fejta-bot commented 3 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale

saschagrunert commented 3 years ago

@aojea are you still planning to work on this? :)

aojea commented 3 years ago

@aojea are you still planning to work on this? :)

Sorry, I was just sharing some experience and things that worked for me in response to the KEP about the release cadence. I really don't have time ...

k8s-triage-robot commented 3 years ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot commented 3 years ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

k8s-ci-robot commented 3 years ago

@k8s-triage-robot: Closing this issue.

In response to [this](https://github.com/kubernetes/sig-release/issues/1527#issuecomment-921603581): >The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. > >This bot triages issues and PRs according to the following rules: >- After 90d of inactivity, `lifecycle/stale` is applied >- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied >- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed > >You can: >- Reopen this issue or PR with `/reopen` >- Mark this issue or PR as fresh with `/remove-lifecycle rotten` >- Offer to help out with [Issue Triage][1] > >Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community). > >/close > >[1]: https://www.kubernetes.dev/docs/guide/issue-triage/ Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
tpepper commented 3 years ago

Would it make sense to title this "Define and collect input metrics about Kubernetes releases"? Currently it feels scoped in that way. It would be very useful to additionally have output usage metrics (eg: download counts of different artifacts during specific time spans).

aojea commented 3 years ago

It would be very useful to additionally have output usage metrics (eg: download counts of different artifacts during specific time spans).

we should be careful with those metrics to avoid falling into the marketing/populism, downloads of artifacts is commonly abused by CI systems ... if you just can differentiate users from bots :thinking: , This problem is similar as counting number of contributors and including people that fixed a typo in one year :/

tpepper commented 3 years ago

https://github.com/kubernetes/sig-release/issues/1372#issuecomment-767108296 represents a way to get at less game-able data perhaps.

Simple stats gaming bots are possible, but I wouldn't consider CI as inherently a problem. I'm actually interested in seeing the bulk CI usage too. If people are still doing a lot of CI on 1.9 that might be informative.

And I'm really interested in the zeros. Which of our RPM and deb variants have zero downloads?

aojea commented 3 years ago

agree

saschagrunert commented 3 years ago

Sounds good, created https://github.com/kubernetes/sig-release/issues/1723 for elaboration.

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

saschagrunert commented 2 years ago

/remove-lifecycle stale

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

saschagrunert commented 2 years ago

/remove-lifecycle stale

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

palnabarun commented 2 years ago

/remove-lifecycle stale

evankanderson commented 2 years ago

I'm not sure what statistics you get from your artifact registry, but here are a few thoughts on the usage side

For within-release management:

jberkus commented 2 years ago
  1. Number of Enhancements dropped from the release, with a reason category for why they were dropped. Defer to enhancements team to come up with 2-6 categories for dropping enhancements (e.g. feature cancelled permanently, author incommunicado, etc.)
  2. Number/percent of release team shadows who completed the release work (i.e. didn't drop out).
  3. Number of issues reported based on alpha and beta releases, by day (i.e. is anyone actually testing these?)
parispittman commented 2 years ago

+1 to calls for metrics on # of regressions, kep status’ (current and future releases; eg how many KEPs not GA yet, how many went to GA, etc), and team shadow metrics

saschagrunert commented 2 years ago

The new project board for enhancements would help us to gather the metrics in a more automated way. I think we should give the board more time to evolve while keeping the necessity for those metrics in mind.

cc @leonardpahlke

Referring discussion: https://kubernetes.slack.com/archives/C2C40FMNF/p1662537798532949?thread_ts=1662477434.202879&cid=C2C40FMNF

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale