search

slsa-framework / slsa-verifier

Verify provenance from SLSA compliant builders
Apache License 2.0
226 stars 48 forks source link

chore: slsa-framework/slsa-github-generator@v2.0.0: add testdata #758

Closed ramonpetgrave64 closed 5 months ago

ramonpetgrave64 commented 5 months ago

https://github.com/slsa-framework/slsa-github-generator/issues/3576

Next step in https://github.com/slsa-framework/slsa-github-generator/blob/main/RELEASE.md#update-verifier

Creating new test data for slsa-github-generator@v2.0.0

Instructions:

diff to download-artifacts.sh

diff --git a/download-artifacts.sh b/download-artifacts.sh
old mode 100644
new mode 100755
index e5e218e8..49257ea6
--- a/download-artifacts.sh
+++ b/download-artifacts.sh
@@ -88,6 +88,10 @@ unzip_files() {
         rm -rf "${tmp_dir}"
         ;;

+    ./*.zip)
+        unzip -o "${zip_path}" -d "${output_path}"
+        ;;
+
     *)
         echo "unexpected file path: ${zip_path}"
         exit 1
@@ -167,7 +171,7 @@ rename_java_files "test-java-project-" "maven"
 rename_java_files "workflow_dispatch-" "gradle"

 # Files downloaded. Now copy them
-repo_path="../.."
+repo_path="/path/to/slsa-verifier"

 # Go builder files.
 copy_files "gha_go-binary-linux-amd64-" "${repo_path}/cli/slsa-verifier/testdata/gha_go/${version}"

download the artifacts

../slsa-verifier/download-artifacts.sh 8791212155 v2.0.0
../slsa-verifier/download-artifacts.sh 8791219359 v2.0.0
../slsa-verifier/download-artifacts.sh 8791219514 v2.0.0
../slsa-verifier/download-artifacts.sh 8791219607 v2.0.0

docker github auth

gh auth login --scopes=read:packages
echo `gh auth token` | docker login ghcr.io -u ramonpetgrave64 --password-stdin
cosign save \
    --dir ./cli/slsa-verifier/testdata/gha_generic_container/v2.0.0/container_workflow_dispatch \
    ghcr.io/slsa-framework/example-package.verifier-e2e.all.tag.main.default.slsa3@sha256:55aee984fd6b1d0e0a19a55265d10d40063a2212bdbabd75b202b1728236548d
ramonpetgrave64 commented 5 months ago

Test failing https://github.com/slsa-framework/slsa-verifier/actions/runs/8801653756/job/24155532842?pr=758#step:6:6

2024-04-23T14:07:06.0587137Z Verified build using builder "https://github.com/slsa-framework/slsa-github-generator/.github/workflows/generator_container_slsa3.yml@refs/tags/v1.9.0" at commit 2bcaa7495e1cbd11fbd4f598d857b3a6f18df933
2024-04-23T14:07:06.0588248Z --- FAIL: Test_runVerifyGHAArtifactImage (0.00s)
2024-04-23T14:07:06.0589075Z     --- FAIL: Test_runVerifyGHAArtifactImage/versioned_tag_no_match_empty_tag_workflow_dispatch_>_v1.9.0 (0.11s)
2024-04-23T14:07:06.0590280Z         main_regression_test.go:869:   any(
2024-04-23T14:07:06.0591552Z             -  e"expected hash 'a3e4bf251423a455ad90c3d706f95f133ed11a8e81e3f34e6fa6d056a1c15529' not found: artifact hash does not match provenance subject",
2024-04-23T14:07:06.0592595Z             +  e"tag used to generate the binary does not match provenance",
2024-04-23T14:07:06.0593057Z               )
2024-04-23T14:07:06.0593410Z         main_regression_test.go:869:   any(
2024-04-23T14:07:06.0594628Z             -  e"expected hash 'a3e4bf251423a455ad90c3d706f95f133ed11a8e81e3f34e6fa6d056a1c15529' not found: artifact hash does not match provenance subject",
2024-04-23T14:07:06.0595643Z             +  e"tag used to generate the binary does not match provenance",
2024-04-23T14:07:06.0596275Z               )
2024-04-23T14:07:06.0596608Z         main_regression_test.go:869:   any(
2024-04-23T14:07:06.0597812Z             -  e"expected hash 'a3e4bf251423a455ad90c3d706f95f133ed11a8e81e3f34e6fa6d056a1c15529' not found: artifact hash does not match provenance subject",
2024-04-23T14:07:06.0598825Z             +  e"tag used to generate the binary does not match provenance",
2024-04-23T14:07:06.0599256Z               )
2024-04-23T14:07:06.0599595Z         main_regression_test.go:869:   any(
2024-04-23T14:07:06.0600780Z             -  e"expected hash 'a3e4bf251423a455ad90c3d706f95f133ed11a8e81e3f34e6fa6d056a1c15529' not found: artifact hash does not match provenance subject",
2024-04-23T14:07:06.0601796Z             +  e"tag used to generate the binary does not match provenance",
2024-04-23T14:07:06.0602235Z               )
2024-04-23T14:07:06.0602726Z     --- FAIL: Test_runVerifyGHAArtifactImage/valid_main_branch_default (0.43s)
2024-04-23T14:07:06.0603295Z         main_regression_test.go:869:   any(
2024-04-23T14:07:06.0604502Z             -  e"expected hash 'a3e4bf251423a455ad90c3d706f95f133ed11a8e81e3f34e6fa6d056a1c15529' not found: artifact hash does not match provenance subject",
2024-04-23T14:07:06.0605250Z               )
2024-04-23T14:07:06.0605568Z         main_regression_test.go:869:   any(
2024-04-23T14:07:06.0606747Z             -  e"expected hash 'a3e4bf251423a455ad90c3d706f95f133ed11a8e81e3f34e6fa6d056a1c15529' not found: artifact hash does not match provenance subject",
2024-04-23T14:07:06.0607480Z               )
2024-04-23T14:07:06.0607799Z         main_regression_test.go:869:   any(
2024-04-23T14:07:06.0608982Z             -  e"expected hash 'a3e4bf251423a455ad90c3d706f95f133ed11a8e81e3f34e6fa6d056a1c15529' not found: artifact hash does not match provenance subject",
2024-04-23T14:07:06.0609714Z               )
2024-04-23T14:07:06.0610174Z         main_regression_test.go:869:   any(
2024-04-23T14:07:06.0611367Z             -  e"expected hash 'a3e4bf251423a455ad90c3d706f95f133ed11a8e81e3f34e6fa6d056a1c15529' not found: artifact hash does not match provenance subject",
2024-04-23T14:07:06.0612102Z               )
2024-04-23T14:07:06.0612719Z     --- FAIL: Test_runVerifyGHAArtifactImage/tag_no_match_empty_tag_workflow_dispatch_>_v1.9.0 (0.12s)
2024-04-23T14:07:06.0613367Z         main_regression_test.go:869:   any(
2024-04-23T14:07:06.0614543Z             -  e"expected hash 'a3e4bf251423a455ad90c3d706f95f133ed11a8e81e3f34e6fa6d056a1c15529' not found: artifact hash does not match provenance subject",
2024-04-23T14:07:06.0615546Z             +  e"tag used to generate the binary does not match provenance",
2024-04-23T14:07:06.0615975Z               )
2024-04-23T14:07:06.0616326Z         main_regression_test.go:869:   any(
2024-04-23T14:07:06.0617525Z             -  e"expected hash 'a3e4bf251423a455ad90c3d706f95f133ed11a8e81e3f34e6fa6d056a1c15529' not found: artifact hash does not match provenance subject",
2024-04-23T14:07:06.0618545Z             +  e"tag used to generate the binary does not match provenance",
2024-04-23T14:07:06.0618980Z               )
2024-04-23T14:07:06.0619307Z         main_regression_test.go:869:   any(
2024-04-23T14:07:06.0620496Z             -  e"expected hash 'a3e4bf251423a455ad90c3d706f95f133ed11a8e81e3f34e6fa6d056a1c15529' not found: artifact hash does not match provenance subject",
2024-04-23T14:07:06.0621497Z             +  e"tag used to generate the binary does not match provenance",
2024-04-23T14:07:06.0621926Z               )
2024-04-23T14:07:06.0622252Z         main_regression_test.go:869:   any(
2024-04-23T14:07:06.0623429Z             -  e"expected hash 'a3e4bf251423a455ad90c3d706f95f133ed11a8e81e3f34e6fa6d056a1c15529' not found: artifact hash does not match provenance subject",
2024-04-23T14:07:06.0624434Z             +  e"tag used to generate the binary does not match provenance",
2024-04-23T14:07:06.0625117Z               )
2024-04-23T14:07:06.0625598Z     --- FAIL: Test_runVerifyGHAArtifactImage/wrong_branch_master (0.58s)
2024-04-23T14:07:06.0626149Z         main_regression_test.go:869:   any(
2024-04-23T14:07:06.0627514Z             -  e"expected hash 'a3e4bf251423a455ad90c3d706f95f133ed11a8e81e3f34e6fa6d056a1c15529' not found: artifact hash does not match provenance subject",
2024-04-23T14:07:06.0628542Z             +  e"branch used to generate the binary does not match provenance",
2024-04-23T14:07:06.0628986Z               )
2024-04-23T14:07:06.0629314Z         main_regression_test.go:869:   any(
2024-04-23T14:07:06.0630504Z             -  e"expected hash 'a3e4bf251423a455ad90c3d706f95f133ed11a8e81e3f34e6fa6d056a1c15529' not found: artifact hash does not match provenance subject",
2024-04-23T14:07:06.0631527Z             +  e"branch used to generate the binary does not match provenance",
2024-04-23T14:07:06.0631974Z               )
2024-04-23T14:07:06.0632299Z         main_regression_test.go:869:   any(
2024-04-23T14:07:06.0633487Z             -  e"expected hash 'a3e4bf251423a455ad90c3d706f95f133ed11a8e81e3f34e6fa6d056a1c15529' not found: artifact hash does not match provenance subject",
2024-04-23T14:07:06.0634516Z             +  e"branch used to generate the binary does not match provenance",
2024-04-23T14:07:06.0634967Z               )
2024-04-23T14:07:06.0635293Z         main_regression_test.go:869:   any(
2024-04-23T14:07:06.0636473Z             -  e"expected hash 'a3e4bf251423a455ad90c3d706f95f133ed11a8e81e3f34e6fa6d056a1c15529' not found: artifact hash does not match provenance subject",
2024-04-23T14:07:06.0637498Z             +  e"branch used to generate the binary does not match provenance",
2024-04-23T14:07:06.0637941Z               )
2024-04-23T14:07:06.0638404Z     --- FAIL: Test_runVerifyGHAArtifactImage/valid_main_branch_set (0.52s)
2024-04-23T14:07:06.0638951Z         main_regression_test.go:869:   any(
2024-04-23T14:07:06.0640306Z             -  e"expected hash 'a3e4bf251423a455ad90c3d706f95f133ed11a8e81e3f34e6fa6d056a1c15529' not found: artifact hash does not match provenance subject",
2024-04-23T14:07:06.0641055Z               )
2024-04-23T14:07:06.0641381Z         main_regression_test.go:869:   any(
2024-04-23T14:07:06.0642599Z             -  e"expected hash 'a3e4bf251423a455ad90c3d706f95f133ed11a8e81e3f34e6fa6d056a1c15529' not found: artifact hash does not match provenance subject",
2024-04-23T14:07:06.0643335Z               )
2024-04-23T14:07:06.0643654Z         main_regression_test.go:869:   any(
2024-04-23T14:07:06.0644836Z             -  e"expected hash 'a3e4bf251423a455ad90c3d706f95f133ed11a8e81e3f34e6fa6d056a1c15529' not found: artifact hash does not match provenance subject",
2024-04-23T14:07:06.0645566Z               )
2024-04-23T14:07:06.0645882Z         main_regression_test.go:869:   any(
2024-04-23T14:07:06.0647058Z             -  e"expected hash 'a3e4bf251423a455ad90c3d706f95f133ed11a8e81e3f34e6fa6d056a1c15529' not found: artifact hash does not match provenance subject",
2024-04-23T14:07:06.0647794Z               )
ramonpetgrave64 commented 5 months ago

Now passing. I had recorded the incorrect hash.

ramonpetgrave64 commented 5 months ago

@laurentsimon @kpk47