sigstore / sigstore-conformance

Conformance testing for Sigstore clients
https://sigstore.dev
7 stars 10 forks source link

Improve unexpected success handling #108

Closed jku closed 11 months ago

jku commented 11 months ago

Summary / Release note


An unexpected success now looks like this:

_____________________________ test_verify_rejects_invalid_inclusion_proof ______________________________

client = <test.client.SigstoreClient object at 0x7f842a9066d0>
make_materials_by_type = <function make_materials_by_type.<locals>._make_materials_by_type at 0x7f842a90c0e0>

    def test_verify_rejects_invalid_inclusion_proof(
        client: SigstoreClient, make_materials_by_type: _MakeMaterialsByType
    ) -> None:
        """
        Check that the client rejects a bundle with an old inclusion proof
        """

        materials: BundleMaterials
        input_path, materials = make_materials_by_type("a.txt", BundleMaterials)
        materials.bundle = Path("a.txt.invalid_inclusion_proof.sigstore")

>       with client.raises():

/home/jkukkonen/src/sigstore-conformance/test/test_bundle.py:142: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/lib/python3.11/contextlib.py:144: in __exit__
    next(self.gen)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <test.client.SigstoreClient object at 0x7f842a9066d0>

    @contextmanager
    def raises(self):
        try:
            yield
        except ClientFail:
            pass
        else:
            assert self.completed_process
            msg = _CLIENT_ERROR_MSG.format(
                exitcode=self.completed_process.returncode,
                command=" ".join(map(str, self.completed_process.args)),
                stdout=self.completed_process.stdout,
                stderr=self.completed_process.stderr,
            )
>           raise ClientUnexpectedSuccess(msg)
E           test.client.ClientUnexpectedSuccess: 
E           Command: /home/jkukkonen/src/sigstore-conformance/sigstore-python-conformance verify-bundle --bundle a.txt.invalid_inclusion_proof.sigstore --certificate-identity https://github.com/sigstore-conformance/extremely-dangerous-public-oidc-beacon/.github/workflows/extremely-dangerous-oidc-beacon.yml@refs/heads/main --certificate-oidc-issuer https://token.actions.githubusercontent.com a.txt
E           Exit code: 0
E           
E           !!! STDOUT !!!
E           ==============
E           
E           OK: a.txt
E           
E           
E           !!! STDERR !!!
E           ==============
E           
E           None

/home/jkukkonen/src/sigstore-conformance/test/client.py:158: ClientUnexpectedSuccess

It would be even nicer if the custom raises() wasn't shown in the stack trace but I think this is an improvement:

I removed the "CLIENT FAILURE" line from output (the exception name should be very clear already: ClientFail or ClientUnexpectedSuccess).

Fixes #107