search

kubernetes-sigs / kubetest2

Kubetest2 is the framework for launching and running end-to-end tests on Kubernetes.
Apache License 2.0
326 stars 105 forks source link

kubetest2-ginko tester overwrites local e2e.test #203

Closed mattcary closed 1 year ago

mattcary commented 1 year ago

Expected behavior: kubetest2 builds and uses a local e2e.test Actual behavior: kubetest2 builds a local e2e.test, but uses a different one.

I'm trying to test out a new e2e.test config flag (https://github.com/kubernetes/kubernetes/pull/111481). I'm running the following kubetest2 command, inspired by the cloud-provider-gcp prow job:

GOPATH=/clank/go/src kubetest2 gce -v 6  --gcp-project mattcary-e2e-test --legacy-mode --repo-root /clank/go/src/k8s.io/kubernetes --build --up --down --test=ginkgo --node-size n1-standard-4 --master-size n1-standard-8 -- --parallel=30 --test-args='--minStartupPods=8 --ginkgo.flakeAttempts=3 --enabled-volume-drivers=gcepd' --skip-regex='\[Slow\]|\[Serial\]|\[Disruptive\]|\[Flaky\]|\[Feature:.+\]' --focus-regex='\[Driver:.gcepd\]'

This is using kubetest2 at commit 2aac35a from 13 July.

The command above correctly builds e2e.test and puts it into a local runfiles. While the gce cluster is coming up, I can confirm that the e2e.test has my new flag.

However, somewhere around here in the logs it's replaced by some version pulled down from somewhere, instead of my local build:

I0728 15:11:35.330046  345616 ginkgo.go:121] Using kubeconfig at /clank/go/src/k8s.io/kubernetes/_rundir/af0111ff-bacd-4b37-92c3-e5516e3ff27e/kubetest2-kubeconfig
I0728 15:11:37.148140  345616 package.go:199] Found existing tar at /usr/local/google/home/mattcary/.cache/kubernetes-test-linux-amd64.tar.gz
I0728 15:11:39.380846  345616 package.go:202] Validated hash for existing tar at /usr/local/google/home/mattcary/.cache/kubernetes-test-linux-amd64.tar.gz
I0728 15:11:41.857031  345616 package.go:165] Found existing kubectl at /clank/go/src/k8s.io/kubernetes/_rundir/af0111ff-bacd-4b37-92c3-e5516e3ff27e/kubectl
W0728 15:11:43.508631  345616 package.go:171] sha256 does not match
Copying gs://kubernetes-release/release/v1.25.0-alpha.3/bin/linux/amd64/kubectl...
\ [1 files][ 42.0 MiB/ 42.0 MiB]                                                
Operation completed over 1 objects/42.0 MiB.                                     
I0728 15:11:46.098135  345616 ginkgo.go:91] Running ginkgo test as /clank/go/src/k8s.io/kubernetes/_rundir/af0111ff-bacd-4b37-92c3-e5516e3ff27e/ginkgo [--nodes=30 /clank/go/src/k8s.io/kubernetes/_rundir/af0111ff-bacd-4b37-92c3-e5516e3ff27e/e2e.test -- --kubeconfig=/clank/go/src/k8s.io/kubernetes/_rundir/af0111ff-bacd-4b37-92c3-e5516e3ff27e/kubetest2-kubeconfig --kubectl-path=/clank/go/src/k8s.io/kubernetes/_rundir/af0111ff-bacd-4b37-92c3-e5516e3ff27e/kubectl --ginkgo.flakeAttempts=1 --ginkgo.skip=\[Slow\]|\[Serial\]|\[Disruptive\]|\[Flaky\]|\[Feature:.+\] --ginkgo.focus=\[Driver:.gcepd\] --report-dir=/clank/go/src/k8s.io/kubernetes/_artifacts --test.timeout=24h --minStartupPods=8 --ginkgo.flakeAttempts=3 --enabled-volume-drivers=gcepd]

That is, at the log message at 15:11:35 the correct e2e.test is in _rundir/af0111fff*, and by the log message at 15:11:26 its been replaced by a different one.

I've looked through the kubetest2 code and wonder if the problem is here: https://github.com/kubernetes-sigs/kubetest2/blob/master/pkg/testers/ginkgo/package.go#L105, although I'm not really sure what's going on.

BenTheElder commented 1 year ago

Hmm I think you need another flag https://github.com/kubernetes-sigs/kubetest2/blob/c76fb417aa01874c79b5adf703f0f651897c8a9d/pkg/testers/ginkgo/ginkgo.go#L123 which seems like a bad default

mattcary commented 1 year ago

Oh god and since this a gingko flag and the build flag is a deployer flag or whatever there's no easy way to coordinate the flag or give any kind of useful error message or help to the user.

Infinite sadness.

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

k8s-ci-robot commented 1 year ago

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to [this](https://github.com/kubernetes-sigs/kubetest2/issues/203#issuecomment-1364835727): >The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. > >This bot triages issues according to the following rules: >- After 90d of inactivity, `lifecycle/stale` is applied >- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied >- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed > >You can: >- Reopen this issue with `/reopen` >- Mark this issue as fresh with `/remove-lifecycle rotten` >- Offer to help out with [Issue Triage][1] > >Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community). > >/close not-planned > >[1]: https://www.kubernetes.dev/docs/guide/issue-triage/ Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.