w3c-ccg / http-signatures-test-suite

The HTTP Signatures test suite
Other
8 stars 3 forks source link

Explicit rejection mode vs simple exceptions to reduce false positives #12

Open liamdennehy opened 5 years ago

liamdennehy commented 5 years ago

Throwing my v10-compliant implementation at this suite has generated a lot of false positives in terms of compliance. The phrase "MUST produce an error" appears to indicate any failure (non-zero exit) is an appropriate exception, while my implementation simply doesn't understand the requested operation e.g. hs2019 - correct for v10.

Any "MUST produce an error" case should also require the application to emit an explicit message stating this is for protocol compliance reasons, e.g. starting with Refused:, so that these false positives are not counted as compliant.

aljones15 commented 5 years ago

One idea we had was exit codes actually. Currently we have 2 exit codes 0 and 1. I think we could expand it so we have 4:

0 for success 1 for expected error 2 for unexpected error 3 for not implemented