Checkbox is a testing framework used to validate device compatibility with Ubuntu Linux. It’s the testing tool developed for the purposes of the Ubuntu Certification program.
Currently if a template generates a unit that is invalid, a message is logged (error for some, debug for some others) and the unit is skipped. This message is visible using Checkbox local but it burried in the Agent log in remote. This is of course sub-optimal as we have no recollection of anything happening after the fact but this can lead to tests being... not executed, without an explanation or even a way for the user to notice.
This PR moves the validation at run_job time, reporting the failure to the user with a clear text when the job would be executed with an explanation as to why the job wasn't executed. This also marks the job as failed (and the test run as failed), making the failure easier to find and to audit.
Description
Currently if a template generates a unit that is invalid, a message is logged (error for some, debug for some others) and the unit is skipped. This message is visible using Checkbox local but it burried in the Agent log in remote. This is of course sub-optimal as we have no recollection of anything happening after the fact but this can lead to tests being... not executed, without an explanation or even a way for the user to notice.
This PR moves the validation at
run_jobtime, reporting the failure to the user with a clear text when the job would be executed with an explanation as to why the job wasn't executed. This also marks the job as failed (and the test run as failed), making the failure easier to find and to audit.Resolved issues
Fixes: https://warthogs.atlassian.net/browse/CHECKBOX-1629
Documentation
Pending implementation approval, I think this should be mentioned in both the extended tutorial and the normal tutorial
Tests
This implements a new metabox scenario