Adding a fast fail option when fetching and/or checking allows for the stop abruptly when an error is encountered during execution rather than continuing on with processing. This feature allows a user to ensure that reports are generated, fixers executed, remote locker is updated and notifications are sent ONLY if all tests (fetchers and checks) have not errored. Failing (not erroring) checks are still valid.
Requirements
[ ] Add a "fast fail" option that aborts the current execution immediately (upon first encountered/unexpected error)
[ ] When "fast fail" is invoked evidence is not committed or pushed to the remote evidence locker
[ ] When "fast fail" is invoked notifiers will not fire
[ ] The "fast fail" option will work on a fetcher execution
[ ] The "fast fail" option will work on a check execution
[ ] The "fast fail" option will work on a combined fetcher/check execution
[ ] Current functionality will not be compromised
[ ] Include documentation for "fast fail"
Approach
Add --fastfail to CLI options.
Update runner logic to immediately stop execution when an unexpected error is encountered during a test suite execution.
Add a fast fail result class for both fetcher and check execution that will stop execution upon first unexpected error.
Leverage unittest.TextTestResult functionality to stop execution upon unexpected error. One possible route for a "fast fail" fetcher test result class would be to set shouldStop to True inside of a stopTest method when a non-dependency chaining error is encountred during the fetch. Similar can be done for a "fast fail" check test result class although here you would not need to guard against a dependency chaining error.
Provide different "fast fail" result classes to runners when --fastfail option is used.
Update CLI execution process to bypass check execution if an error is encountered during a fetcher processing. This is only applicable when fetchers and checks are executed as part of a single command execution.
Update documentation by providing content about the "fast fail" option. Content should include "fast fail" behavior for fetchers, checks and fetchers/checks combined. Provide details on the state of the local locker when execution is aborted due to "fast fail".
Security and Privacy
N/A
Test Plan
Test fetcher only execution
Test check only execution (If a check errors further processing (reports, fixers, notifiers) will not run)
Test combined fetcher/check execution
If a fetcher errors then no checks are to run
If a check errors further processing (reports, fixers, notifiers) will not run
Ensure current functionality is not changed when not using the --fastfail option
Overview
Adding a fast fail option when fetching and/or checking allows for the stop abruptly when an error is encountered during execution rather than continuing on with processing. This feature allows a user to ensure that reports are generated, fixers executed, remote locker is updated and notifications are sent ONLY if all tests (fetchers and checks) have not errored. Failing (not erroring) checks are still valid.
Requirements
Approach
--fastfailto CLI options.Trueinside of a stopTest method when a non-dependency chaining error is encountred during the fetch. Similar can be done for a "fast fail" check test result class although here you would not need to guard against a dependency chaining error.--fastfailoption is used.Security and Privacy
N/A
Test Plan
--fastfailoption