How do we ensure that implementations comply with the spec?

[mattsb42-aws]

We need to find some way to check that an implementation complies with the specification.

One tool in this might be expanding the test vector framework[1] and test vectors.

Each specification document SHOULD be involved in these checks.

[1] https://github.com/awslabs/aws-crypto-tools-test-vector-framework [2] https://github.com/awslabs/aws-encryption-sdk-test-vectors

[robin-aws]

The plan I've arrived at for this is:

1. Make https://github.com/awslabs/aws-encryption-sdk-test-vectors a submodule of each ESDK implementation
   1. This is the preferred way to depend on the test vectors when actually driving the tests anyway. The Javascript ESDK already does this for example: https://github.com/aws/aws-encryption-sdk-javascript
2. Add https://github.com/awslabs/aws-encryption-sdk-specification as a submodule of https://github.com/awslabs/aws-encryption-sdk-test-vectors
   1. This is more of a semantic dependency since the specification is not machine-readable, but it provides a mechanism for tracking when the test vectors need to be updated to cover new features/behaviour
3. Apply dependabot to the test vectors and ESDK repositories to automate tracking these pending updates

The overall picture for a specification change flowing through all the affected projects will look like this:

1. Squishy human merges a spec change to spec feature X, bumping version from 1.0 to 1.1
2. Dependabot notices this and opens a test vectors PR
   1. It won't understand what piece of the specification has changed, only that the head of master has advanced
3. Squishy human deals with this (adds test cases or just a string to a list that implementations have to acknowledge), test vectors change is merged
4. Depandabot notices this and opens a PR on each ESDK implementation
5. Squishy humans eventually deal with this in each implementation
   1. Sometimes just verifying tests still pass if the implementation is already aligned, or augmenting the test vector client code for that implementation, or changing the implementation itself

Some specification changes will not need any actual test vector or implementation changes (for example, when a squishy human makes a spec change just to enter “Y implementation now meets specification feature X version 1.1"), in which case steps 3 and 5 will just be approving and merging the Dependabot-created PRs. This shouldn't be a huge pain since Dependabot will continuously update the PRs it cuts to point to the latest version, so these updates can be easily handled in batches.

The test vectors will need to be expanded to cover more kinds of tests, but this flow will still help track the required tasks even when the specification change is not unit-testable, and we can apply the fully-manual approach while the test vectors catch up.

[mattsb42-aws]

Am I correctly reading an implied "the spec repo and test vectors repo need to start publishing release tags"? If not, what commitish will the submodules point to and how will dependabot track/update that change?

[robin-aws]

No need for release tags - the submodules will just point to literal commit hashes for the current state, and dependabot should be triggered on every new commit to master. I'm experimenting with that now to confirm, however.