search

kubernetes / test-infra

Test infrastructure for the Kubernetes project.
Apache License 2.0
3.83k stars 2.65k forks source link

Cleanup and update `releng/test_config.yaml` #32340

Open xmudrii opened 7 months ago

xmudrii commented 7 months ago

releng/test_config.yaml contains definitions for kubernetes/kubernetes tests (mainly periodics). We use those definitions to generate jobs for release branches when cutting a new release branch.

Over the time, this file has become unmaintained which is a huge risk for having jobs that are not properly testing release branches.

Here are some problems that I observed; this is a non exhaustive list, in other words, there might be other issues that I didn't observe:

Once these issues are addressed, we should verify if arguments are up to date for remaining test options.

cc @kubernetes/release-engineering @dims

xmudrii commented 7 months ago

/sig release /area release-eng

xmudrii commented 7 months ago

/priority important-soon

dims commented 7 months ago

/milestone v1.31

k8s-ci-robot commented 7 months ago

@dims: The provided milestone is not valid for this repository. Milestones in this repository: [someday, v1.24, v1.25]

Use /milestone clear to clear the milestone.

In response to [this](https://github.com/kubernetes/test-infra/issues/32340#issuecomment-2031896924): >/milestone v1.31 Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
k8s-triage-robot commented 4 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

xmudrii commented 3 months ago

/remove-lifecycle stale

xmudrii commented 3 months ago

We're running into this issue again ahead of the 1.31 jobs cut. These are the high priority issues at the moment:

/priority critical-urgent cc @dims @mehabhalodiya

dims commented 3 months ago

ci-kubernetes-e2e-gce-cos-k8sbeta-default (1.30) has --env=ENABLE_CACHE_MUTATION_DETECTOR=true, do we need to keep this for 1.31 as well?

Yes, we should keep it. helps catch the scenario where anything mutates a shared informer cache (see description here)

ci-kubernetes-e2e-gce-cos-k8sbeta-alphafeatures (1.30) uses a custom testSuite called alphafeatures-ccm-eventedpleg, do we need to keep it for 1.31 as well?

Yes, please. will help give us signal on an alpha feature https://github.com/kubernetes/kubernetes/blob/9413cf204ac92711cc8aff472b1ed11ba79760ac/pkg/features/kube_features.go#L261-L267

ci-kubernetes-e2e-gce-cos-k8sstable1-default (1.29) has --env=ENABLE_POD_SECURITY_POLICY=true, however, this env is present only for 1.29. Do we need it for any other version or is it specific to only 1.29?

We can drop it, does not do much AFAICT looking through references - https://cs.k8s.io/?q=ENABLE_POD_SECURITY_POLICY%7CEnablePodSecurityPolicy&i=nope&files=&excludeFiles=vendor&repos=

BenTheElder commented 3 months ago

Can we please just use config fork annotations on normal prowjobs and stop using this other thing entirely?

I don't think SIG release should be defining these jobs, just forking them when new branches are cut, and I really doubt anyone else in the project is looking at this file. It's hard to even understand what these config fields are doing.

cc @liggitt @tallclair re: PSP

yes we want to keep the cache mutation detector enabled.

liggitt commented 3 months ago

ENABLE_POD_SECURITY_POLICY is irrelevant in 1.25+ and can be dropped

BenTheElder commented 3 months ago

documented and widely used:

https://github.com/kubernetes/test-infra/tree/master/releng/config-forker#supported-annotations

Versus ~no docs for generate_tests and the out of band config.

If the config fork annotations are not powerful enough for some reason, let's fix that.

xmudrii commented 3 months ago

Can we please just use config fork annotations on normal prowjobs and stop using this other thing entirely?

Long term, we can look into it. But short term and specially for this upcoming release, definitely no. We're happy to look into it, but this will require some time and we want to get the release branch up and running ASAP.

BenTheElder commented 3 months ago

I'm pretty sure https://kubernetes.slack.com/archives/CN0K3TE2C/p1722833073336809 is related

The problem is that job maintainers are going to PR the job definitions and expect auto-forking, and are not aware that this out of band config needs changing as well, since it is only used once per release branch.

I think we really need to switch for 1.32 or we'll keep having regressions.

cc @pohly @aojea, @kubernetes/release-engineering

xmudrii commented 3 months ago

/assign Let met look into this and if we can quickly drop this

k8s-triage-robot commented 1 day ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

xmudrii commented 1 day ago

/remove-lifecycle stale