Previously, we've discussed the merits of two sorts of conformance testing:
"flow-based" end-to-end tests, like what's currently in this repo. These tests are good for checking interoperability, common cases (both failure and success), compare good for checking interoperability, common cases (both failure and success), compatibility with the infrastructure (including tests that can run against staging/dev environments). The cost is that they're heavierweight and involve network calls.
"test-vector style" tests: e.g., for verification flows. They may miss some coverage and it's harder to exercise network calls, but these are much faster and support fuzzing better.
I, for one, left that conversation convinced that it's a good idea to do both.
We're starting to pick up on test vector work (e.g., https://github.com/sigstore/protobuf-specs/issues/66 ; CC @asraa). At the Sigstore clients meeting the other day (CC @bdehamer, @codysoyland, @kommendorkapten) we discussed where these (and related harness code etc.) should live. While previously, we had planned to put them in sigstore/protobuf-specs, it seems like sigstore/conformance-tests is a quite natural fit.
If you all agree, we can add those in here, update the README to articulate the overall test strategy, and use this as the home for test vector testing. If you think they'd be better off elsewhere, we can discuss where too.
Previously, we've discussed the merits of two sorts of conformance testing:
I, for one, left that conversation convinced that it's a good idea to do both.
We're starting to pick up on test vector work (e.g., https://github.com/sigstore/protobuf-specs/issues/66 ; CC @asraa). At the Sigstore clients meeting the other day (CC @bdehamer, @codysoyland, @kommendorkapten) we discussed where these (and related harness code etc.) should live. While previously, we had planned to put them in sigstore/protobuf-specs, it seems like sigstore/conformance-tests is a quite natural fit.
If you all agree, we can add those in here, update the README to articulate the overall test strategy, and use this as the home for test vector testing. If you think they'd be better off elsewhere, we can discuss where too.