Add staging to selftest

sigstore / sigstore-conformance

Conformance testing for Sigstore clients

https://sigstore.dev

7 stars 10 forks source link

Add staging to selftest #127

Closed jku closed 6 months ago

jku commented 6 months ago

This might be enough to close #121.

Summary

Do several small changes required to support staging tests in self-test

the enable-staging action input defaults to false: we may want to reconsider before releasing -- maybe opt-out is better?
only a few tests are marked as compatible with staging at this point: I may have missed some but since there are loads of false negatives in the remaining tests (tests that seemingly have the correct result but only because they fail for the wrong reasons on staging) I didn't want to extend without research

TODO?

--staging flag is sort of documented in cli_protocol.md but it could probably be improved

Test run https://github.com/sigstore/sigstore-conformance/actions/runs/8017481217

jku commented 6 months ago

cc @javanlacerda please have a look:

did I miss anything?
does the approach of doing two runs of the test suite in a single GH action call make sense?

javanlacerda commented 6 months ago

cc @javanlacerda please have a look:

did I miss anything?

does the approach of doing two runs of the test suite in a single GH action call make sense?

Hey @jku! It seems awesome!

I am just wondering if we could have staging as default and production as optional. I mean, It could decrease the production infrastructure usage for development proposes. Does it make sense?

jku commented 6 months ago

I am just wondering if we could have staging as default and production as optional. I mean, It could decrease the production infrastructure usage for development proposes. Does it make sense?

I think once we're sure this works the default should be to test everything (and not testing with production seems like a decision that could bite a client developer in the behind). Whether there is a use case to ever test only staging I don't know...

woodruffw commented 6 months ago

Whether there is a use case to ever test only staging I don't know...

I personally can't think of one 🙂 -- IMO we can save ourselves a state here and make staging tests "additive" rather than dealing with different combinations.