lidel
commented
10 months ago I've been thinking about this today while looking at https://github.com/ipfs/gateway-conformance/pull/190 and if we modify utility helpers MustGetCid, MustGetCidWithCodec, MustGetDescendantsCids to take expected CIDs as argument, we have bidirectional verification of the test harness:
- Hardcoded CIDs will make test cases explicit rather than implicit.
- We keep the perk of programmatically loading fixtures AND we detect when the content of a fixture changes – CIDs will no longer match.
This will catch both software errors (CAR fixture parser library bugs) and human ones (someone swapping fixture file with something that has different CIDs).
For rare cases when we have a lot CIDs or not care about having exact blocks, we could have helpers which do not require explicit CID in argument.
See the changes in https://github.com/ipfs/gateway-conformance/pull/87
We used fixtures + libraries to generate expectation values (like an expected payload or an expected CIDs) But in some (most?) cases, this relies on the fact that the library itself is correct.
Take a test that verifies the ordering of blocks in a CAR produced by a gateway, if we use go-ipld-xyz to load the car and generate the array of blocks, then we test against the library, not against the spec.
The other option is to hardcode results (like arrays of CIDs) which is okay up to a point where we have dozens of long lists of random strings and one test breaks or we have to update it.
Ideally we'd have some way to describe CIDs & other payloads like documentation and load these in the code, similar to how we do DNS link configurations.