Closed javanlacerda closed 6 months ago
rather than enabling individual tests to be run against staging, maybe we should allow the entire suite to be flipped between production and staging at a time?
We can try that... but it may make sense to still enable just a subset of tests (the end-to-end ones) in the beginning: otherwise it might be quite a bit of work to re-create all of the test assets for staging (I assume that will be needed at least: I haven't created any of the assets)
If all (or many) tests can be run against staging, then we need to provide clients a way to enable/disable staging testing: xfail for individual tests is not viable... Some potential solutions:
staging
option to the GH action: this way clients can add a whole new action call to to their workflow when they are ready to test stagingI suppose the second one makes more sense?
Waving hands about what tests should be executed by pytest:
@pytest.mark.signing
to skip tests: we could do something similar: if pytest gets "--staging", it could skip tests with @pytest.mark.no_staging_assets
make_materials*()
functions at that timeSome of results that currently PASS when running GHA_SIGSTORE_CONFORMANCE_XFAIL="test_verify_with_trust_root test_verify_dsse_bundle_with_trust_root" pytest test --staging --skip-signing --entrypoint=$PWD/sigstore-python-conformance
are likely incorrect: if the suite uses production assets against staging infra, the client-under test is going to fail as the test expects (but it's failing for the wrong reasons) .
Yeah, getting the production/staging states right will probably require a larger refactor/discussion of how we want the CI to run the suite. I'm okay with punting on that for now 🙂
(CCing @jku for review as well, since I'm now the last pusher.)
Ah, sorry @jku I merged too quickly here, please feel free to finish your review and suggest any changes if we missed something!
Should we expect a significant increase in traffic to staging?
Should we expect a significant increase in traffic to staging?
I don't think so -- based on current integrations, this should only run a handful of times a day on the sigstore-python, Java, etc. repos. So the traffic should be roughly the same as the existing staging tests, e.g. the ones sigstore-python
does in its own CI 🙂
Closes #121
Summary
One conformance requirement for the clients is that they should be able to run the commands against
staging
. After this PR, the clients should be able to receive the--staging
token in the command line and point to staging resources.Release Note
Updated client run function to receive
staging
argument. Updated run subprocess call to pass--staging
flag to clients cli when required. Modified CLI parser to receive --staging flag Update CLI and conformance READMEs adding staging featuresDocumentation