kingdonb
commented
1 year ago It is definitely testable in-place. I have tested it and found:
-
8
which should be resolved before we try deploying it to fluxcd/website. (It should also be tagged, and I'd suggest creating some additional tests created for the ruby files.)
Anyway, we can see from the test output here:
It runs, fails with exit code 2 (it's buried a little bit, but if you dig through the output for 'exit code 2' you find it):
Summary:
--------
Total issues in main site: 67
Total issues in preview site: 81
Resolved issues: 29
New issues: 43
Fail: The preview site has introduced new issues!
Top 3 problematic links introduced in the PR:
Link: https://deploy-preview-1573--fluxcd.netlify.app/flux-e2e/#fluxs-default-configuration-for-networkpolicy
Found on: https://deploy-preview-1573--fluxcd.netlify.app/flux-e2e/#fluxs-default-configuration-for-networkpolicy
---------
Link: https://deploy-preview-1573--fluxcd.netlify.app/flux/components/image/imagepolicies/#filtertags
Found on: https://deploy-preview-1573--fluxcd.netlify.app/flux/components/image/imagepolicies/#filtertags
---------
Link: https://deploy-preview-1573--fluxcd.netlify.app/flux/components/kustomize/kustomization/#garbage-collection
Found on: https://deploy-preview-1573--fluxcd.netlify.app/flux/components/kustomize/kustomization/#garbage-collection
---------
Please check pr-summary.csv for the full list of new issues.
make: *** [Makefile:39: summary] Error 1
Error: Process completed with exit code 2.Then it attaches the summary PR csv (this explains what did this PR do to trigger the failing check!)
If the test passes, we move onto the baseline report which is also interesting, but shouldn't fail the check. That one gets uploaded, only if it contains any useful records.
Finally, we should be able to see that when no more issues are triggered in either report, the CSV uploads will stop.
One should only need these CSVs when there are really a lot of errors (like right now.)
No way we can close #3 without adding some formal tests.