The conformance tests are not all independent of one another. That is, when there's a failure, isolating that test (for example by changing Specify to FSpecify) and re-running the test suite will not yield the same test failure, because the tests have interdependencies.
To take one example, I am currently seeing this failure:
/home/rogpeppe/other/oci-distribution-spec/conformance/00_conformance_suite_test.go:12
Push
/home/rogpeppe/other/oci-distribution-spec/conformance/02_push_test.go:15
Blob Upload Chunked
/home/rogpeppe/other/oci-distribution-spec/conformance/02_push_test.go:152
Get on stale blob upload should return 204 with a range and location [It]
/home/rogpeppe/other/oci-distribution-spec/conformance/02_push_test.go:215
Expected
<int>: 405
to equal
<int>: 204
/home/rogpeppe/other/oci-distribution-spec/conformance/02_push_test.go:220
If I try to run that test on its own to see exactly what HTTP requests are being sent and received, so to try to understand why this failure is happening and how to fix it, I'd expect to be able to focus on the test by changing line 215 to use FSpecify.
However, that doesn't work because on line 217, the test refers toprevResponse, a variable outside of that test, that has been populated by this test.
This pattern is common throughout the conformance tests.
In general, it's a good idea for tests to be self-contained. Failing that, it would be nice if tests could at least document their dependencies so running a test on its own doesn't require error-prone tracing of dependencies.
The conformance tests are not all independent of one another. That is, when there's a failure, isolating that test (for example by changing
SpecifytoFSpecify) and re-running the test suite will not yield the same test failure, because the tests have interdependencies.To take one example, I am currently seeing this failure:
The location of the failure is here.
If I try to run that test on its own to see exactly what HTTP requests are being sent and received, so to try to understand why this failure is happening and how to fix it, I'd expect to be able to focus on the test by changing line 215 to use
FSpecify.However, that doesn't work because on line 217, the test refers to
prevResponse, a variable outside of that test, that has been populated by this test.This pattern is common throughout the conformance tests.
In general, it's a good idea for tests to be self-contained. Failing that, it would be nice if tests could at least document their dependencies so running a test on its own doesn't require error-prone tracing of dependencies.