search

minio / mint

Collection of tests to detect overall correctness of MinIO server.
Apache License 2.0
78 stars 50 forks source link

Mint - Run subsequent system tests even if a test fails #325

Open iternity-dotcom opened 3 years ago

iternity-dotcom commented 3 years ago

Is your feature request related to a problem? Please describe. Whenever the Mint test suite is executed, a single failing test in minio-java or minio-go causes all the remaining tests from the same "test package" to be skipped. The disadvantage is, that you don't see all failing tests but only the first failing test (at least in minio-java and minio-go tests). Ok the advantage is, that you don't need to run the whole test suite to see if "something" is broken.

Describe the solution you'd like We suggest to introduce an optional runtime variable RUN_ON_FAIL. By setting the variable RUN_ON_FAIL=1 all tests from all packages should be executed even if a preceeding tests is failing.

Describe alternatives you've considered Changing the default behavior to ALWAYS run every test independent of failing tests. This would change the default behavior of the Mint test suite.

Additional context We implemented this feature for minio-go and minio-java and we will create appropriate PRs and link them to this issue, so that there is a single place for discussions for this.

iternity-dotcom commented 3 years ago

Also refering to a discussion in slack: https://minio.slack.com/archives/C3NDUB8UA/p1615891367010200 @kannappanr @GarimaKapoor

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 15 days if no further activity occurs. Thank you for your contributions.

iternity-dotcom commented 3 years ago

There are still some open PRs related to this issue.

-> https://github.com/minio/minio-go/pull/1475 -> https://github.com/minio/minio-java/pull/1180

And a change in minio-java that needs to be reverted: https://github.com/minio/mint/commit/721525fa3894306c117e36f7dcea6e1af369f133

We will rebase those PRs and add a new one for minio-java.

andrebruch commented 3 years ago

The two PRs are now up to date: https://github.com/minio/minio-go/pull/1475 https://github.com/minio/minio-java/pull/1180

The change in the minio-java call (721525f) does not need to be reverted. I passed the variable RUN_ON_FAIL as environment variable and additionally (that was the error) as parameter to minio-java.