search

kubernetes / sig-testing

Home for SIG Testing discussion and documents.
Apache License 2.0
6 stars 8 forks source link

Describe and write down the CI policy #9

Closed cpanato closed 2 years ago

cpanato commented 4 years ago

Describe the CI policy for jobs, like:

We have a policy for blocking and informing jobs, https://github.com/kubernetes/sig-release/blob/master/release-blocking-jobs.md . If we compare this policy with what is proposed in https://github.com/kubernetes/test-infra/issues/18599, what would we add? what would we change? We should evaluate what changes we need to make to help ensure we are acting on useful information and check tht CI jobs are maintained and well.

/area release-eng /area ci /kind documentation /priority important-soon /milestone v1.20

k8s-ci-robot commented 4 years ago

@cpanato: The label(s) area/ci cannot be applied, because the repository doesn't have them

In response to [this](https://github.com/kubernetes/sig-release/issues/1217): >Describe the CI policy for jobs, like: > >- having contact information >- resources defined > >Parent Issue: https://github.com/kubernetes/sig-release/issues/1215 > >/area release-eng >/area ci >/kind documentation >/priority important-soon >/milestone v1.20 Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
cpanato commented 4 years ago

will close because it is a duplicate of https://github.com/kubernetes/test-infra/issues/18551

/close

k8s-ci-robot commented 4 years ago

@cpanato: Closing this issue.

In response to [this](https://github.com/kubernetes/sig-release/issues/1217#issuecomment-687189268): >will close because it is a duplicate of https://github.com/kubernetes/test-infra/issues/18551 > >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
alejandrox1 commented 4 years ago

Reopening this to evaluate how the current release blocking and release informing policy compares with whats proposed in https://github.com/kubernetes/test-infra/issues/18599 .

/reopen

k8s-ci-robot commented 4 years ago

@alejandrox1: Reopened this issue.

In response to [this](https://github.com/kubernetes/sig-release/issues/1217#issuecomment-690294967): >Reopening this to evaluate how the current release blocking and release informing policy compares with whats proposed in https://github.com/kubernetes/test-infra/issues/18599 . > > >/reopen Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
spiffxp commented 4 years ago

I would recommend we strive to make the criteria something that can be enforced via tests or automation. I still think the final decision should come down to humans, but it's not clear to me how often people really check against adherence to these criteria.

Taking a look at release-blocking criteria

Have the average of 75% percentile duration of all runs for a week finishing in 120 minutes or less

This used to be charted; p75_duration in http://storage.googleapis.com/k8s-metrics/job-health-latest.json is daily not weekly

Run at least every 3 hours

If every job is a prowjob, we could statically check the job configs that use interval; if not we could approximate by using runs from http://storage.googleapis.com/k8s-metrics/job-health-latest.json and alert if it's less than 8

Be able to pass 3 times in a row against the same commit

We don't measure this currently, is it possible for us to do so? Or should we use some other measure?

Be Owned by a SIG, or other team, that is responsive to addressing failures, and whose alert email is configured in the job.

Ownership is enforced via static checks against the testgrid config. "That is responsive" though, I'm not sure how we measure that?

Have passed 75% of all of its runs in a week, and have failed for no more than 10 runs in a row

This used to be charted; failure_rate in http://storage.googleapis.com/k8s-metrics/job-health-latest.json is daily not weekly

I think testgrid's summary page shows how many out of 10 recent columns passed, but I'm not sure if we measure this over time?

fejta-bot commented 3 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

cpanato commented 3 years ago

/remove-lifecycle stale

cpanato commented 3 years ago

added this to my next week backlog

saschagrunert commented 3 years ago

Took my freedom to rename the issue to optically match with others.

fejta-bot commented 3 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale

spiffxp commented 3 years ago

/remove-lifecycle stale

spiffxp commented 3 years ago

I would like to move this over to kubernetes/sig-testing with the intent of tackling this in v1.22, any objections?

saschagrunert commented 3 years ago

I would like to move this over to kubernetes/sig-testing with the intent of tackling this in v1.22, any objections?

Sounds good! Thank you for catching up with this :pray:

LappleApple commented 3 years ago

/sig testing

fejta-bot commented 3 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale

k8s-triage-robot commented 3 years ago

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community. /lifecycle rotten

k8s-triage-robot commented 3 years ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

k8s-ci-robot commented 3 years ago

@k8s-triage-robot: Closing this issue.

In response to [this](https://github.com/kubernetes/sig-release/issues/1217#issuecomment-913024485): >The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. > >This bot triages issues and PRs according to the following rules: >- After 90d of inactivity, `lifecycle/stale` is applied >- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied >- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed > >You can: >- Reopen this issue or PR with `/reopen` >- Mark this issue or PR as fresh with `/remove-lifecycle rotten` >- Offer to help out with [Issue Triage][1] > >Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community). > >/close > >[1]: https://www.kubernetes.dev/docs/guide/issue-triage/ Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
spiffxp commented 3 years ago

/reopen /remove-lifecycle rotten /lifecycle frozen

k8s-ci-robot commented 3 years ago

@spiffxp: Reopened this issue.

In response to [this](https://github.com/kubernetes/sig-release/issues/1217#issuecomment-915391961): >/reopen >/remove-lifecycle rotten >/lifecycle frozen Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
k8s-ci-robot commented 3 years ago

@cpanato: The label(s) area/release-eng, area/ci cannot be applied, because the repository doesn't have them.

In response to [this](https://github.com/kubernetes/sig-testing/issues/9): >Describe the CI policy for jobs, like: > >- having contact information >- resources defined > >We have a policy for blocking and informing jobs, https://github.com/kubernetes/sig-release/blob/master/release-blocking-jobs.md . >If we compare this policy with what is proposed in https://github.com/kubernetes/test-infra/issues/18599, what would we add? what would we change? >We should evaluate what changes we need to make to help ensure we are acting on useful information and check tht CI jobs are maintained and well. > >/area release-eng >/area ci >/kind documentation >/priority important-soon >/milestone v1.20 Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
spiffxp commented 3 years ago

/sig testing

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

k8s-ci-robot commented 2 years ago

@k8s-triage-robot: Closing this issue.

In response to [this](https://github.com/kubernetes/sig-testing/issues/9#issuecomment-1030662470): >The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. > >This bot triages issues and PRs according to the following rules: >- After 90d of inactivity, `lifecycle/stale` is applied >- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied >- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed > >You can: >- Reopen this issue or PR with `/reopen` >- Mark this issue or PR as fresh with `/remove-lifecycle rotten` >- Offer to help out with [Issue Triage][1] > >Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community). > >/close > >[1]: https://www.kubernetes.dev/docs/guide/issue-triage/ Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.