Add several bundle tests

steiza commented 1 year ago

For https://github.com/sigstore/sigstore-conformance/issues/83

Summary

This extends the bundle verification tests to exercise more failure paths, like ensuring you're using the right trust root, ensuring the signed entry timestamp is from the correct timeframe, and that the message signature and digest haven't been tampered with.

Release Note

NONE

Documentation

N/A

tetsuo-cpp commented 1 year ago

Hey @steiza, there are a few linting errors. Once you fix these, you can run make lint to check again.

./env/bin/python -m mypy action.py test/
test/test_bundle.py:67: error: Incompatible types in assignment (expression has type "str", variable has type "Path")  [assignment]
test/test_bundle.py:83: error: Incompatible types in assignment (expression has type "str", variable has type "Path")  [assignment]
test/test_bundle.py:98: error: Incompatible types in assignment (expression has type "str", variable has type "Path")  [assignment]
test/test_bundle.py:113: error: Incompatible types in assignment (expression has type "str", variable has type "Path")  [assignment]
Found 4 errors in 1 file (checked 8 source files)
make: *** [Makefile:23: lint] Error 1
Error: Process completed with exit code 2.

tetsuo-cpp commented 1 year ago

Interesting, sigstore-python is failing a few of these tests which is causing the Self-test CI to fail. Once we get this merged, I'll debug what's going on with the Python client.

@steiza, just to confirm, your library is passing these tests right?

steiza commented 1 year ago

@steiza, just to confirm, your library is passing these tests right?

That's correct.

Interesting, sigstore-python is failing a few of these tests which is causing the Self-test CI to fail.

I also noticed this.

When I was testing locally, I think test_verify_rejects_invalid_set() isn't passing because it raises an exception instead of returning FAIL / Failure reason:

sigstore-conformance$ sigstore verify github --bundle test/assets/a.txt.invalid_set.sigstore --cert-identity https://github.com/sigstore-conformance/extremely-dangerous-public-oidc-beacon/.github/workflows/extremely-dangerous-oidc-beacon.yml@refs/heads/main --offline test/assets/a.txt
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/sigstore/_internal/keyring.py", line 116, in verify
    key.verify(
  File "/usr/local/lib/python3.11/site-packages/cryptography/hazmat/backends/openssl/ec.py", line 328, in verify
    _ecdsa_sig_verify(self._backend, self, signature, data)
  File "/usr/local/lib/python3.11/site-packages/cryptography/hazmat/backends/openssl/ec.py", line 124, in _ecdsa_sig_verify
    raise InvalidSignature
cryptography.exceptions.InvalidSignature

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/sigstore", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sigstore/_cli.py", line 549, in main
    _verify_github(args)
  File "/usr/local/lib/python3.11/site-packages/sigstore/_cli.py", line 948, in _verify_github
    result = verifier.verify(materials=materials, policy=policy_)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sigstore/verify/verifier.py", line 264, in verify
    verify_set(self._rekor, entry)
  File "/usr/local/lib/python3.11/site-packages/sigstore/_internal/set.py", line 44, in verify_set
    client._rekor_keyring.verify(
  File "/usr/local/lib/python3.11/site-packages/sigstore/_internal/keyring.py", line 125, in verify
    raise KeyringSignatureError("invalid signature") from exc
sigstore._internal.keyring.KeyringSignatureError: invalid signature

However for test_verify_rejects_invalid_digest(); when I test locally it seems to pass verification (which shouldn't be the case):

sigstore-conformance$ sigstore verify github --bundle test/assets/a.txt.invalid_digest.sigstore --cert-identity https://github.com/sigstore-conformance/extremely-dangerous-public-oidc-beacon/.github/workflows/extremely-dangerous-oidc-beacon.yml@refs/heads/main --offline test/assets/a.txt
OK: test/assets/a.txt

tetsuo-cpp commented 1 year ago

@woodruffw I think we should merge this even if CI is failing and then follow up with sigstore-python fixes. Is that ok?

woodruffw commented 1 year ago

I think we should merge this even if CI is failing and then follow up with sigstore-python fixes. Is that ok?

Yeah, works for me -- merging!

tetsuo-cpp commented 1 year ago

Awesome, thanks for your contribution @steiza!

sigstore / sigstore-conformance