search

kubernetes-sigs / kubetest2

Kubetest2 is the framework for launching and running end-to-end tests on Kubernetes.
Apache License 2.0
331 stars 106 forks source link

Unable to run kubetest2 <deployer> --test more than once from same path #176

Closed Rajalakshmi-Girish closed 1 year ago

Rajalakshmi-Girish commented 2 years ago

Summary: We are unable to use kubetest2 to test kubernetes more than once from a same path.

metadata.json generated by kubetest2 command is objecting to run kubetest2 second time from same path stating the following error: Example using gingko tester:

root@kubetest2:/workspace# kubetest2 tf --test=ginkgo -- --flake-attempts 2 --focus-regex='ariable Expansion should verify that a failing subpath expansion can be modified during the lifecycle of a container'
F1210 11:00:28.745660   14208 ginkgo.go:205] failed to run ginkgo tester: key tester-version already exists in the metadata
Error: exit status 255
root@kubetest2:/workspace#

Expected result: Should be able to run kubetest2 any number of times from a given path. There should ether be a flag to ignore metadata.json or overwrite the metadata with latest kubetest2 run

In order to run k8s conformance tests by following option B at https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md#running-conformance-tests-with-kubetest, we are required to trigger kubetest2 command twice. 

failed to run ginkgo tester: key tester-version already exists in the metadata
Error: exit status 255

is the error thrown when run from same path.

Note: This started happening after the below commit to fail if key is already present in metadata.json. https://github.com/kubernetes-sigs/kubetest2/pull/162/commits/ac36e3203cb49dce808f5bc211e1decc63956d93 Expect this check to be more liberal https://github.com/kubernetes-sigs/kubetest2/blob/master/pkg/metadata/metadata.go#L49

Sample job that threw this error: https://prow.ppc64le-cloud.org/view/gs/ppc64le-kubernetes/logs/test-periodic-kubernetes-conformance-test-ppc64le/1468899640811196416

Rajalakshmi-Girish commented 2 years ago

@mkumatag ^^

mkumatag commented 2 years ago

/cc @amwat

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

Rajalakshmi-Girish commented 2 years ago

/remove-lifecycle stale

BenTheElder commented 2 years ago

fixed by your pending PR https://github.com/kubernetes-sigs/kubetest2/pull/183, right?

Rajalakshmi-Girish commented 2 years ago

fixed by your pending PR #183, right?

This seems to be fixed by https://github.com/kubernetes-sigs/kubetest2/pull/162 Closing this issue as we no longer encounter this after upgrading to latest kubetest2

Rajalakshmi-Girish commented 2 years ago

This error started coming back after moving metadata.json from RunDir to ARTIFACTS via https://github.com/kubernetes-sigs/kubetest2/pull/221

IMO, if a second kubetest2 command is not allowed to run from the same path as the first kubetest2, The error message has to suggest what can be done.

BenTheElder commented 2 years ago

IMHO: metadata.json should be part of ARTIFACTS, it's expected to be uploaded in CI and read by the CI tools.

We should probably just overwrite or amend the existing file.

In the meantime, kubetest2 should probably be used with unique $ARTIFACTS, maybe we should instead use a tempdir for the ARTIFACTS location if it's not explicitly set.

I've stepped down to emeritus_approver in this project though, I switched roles / transferred teams earlier this year and this was on my "I am still doing too many things" list.

Rajalakshmi-Girish commented 1 year ago

Unable to reproduce this now! Seems some change to the metadata package fixed this.