There are often incoming reports that are clearly not good-faith effort to test the distribution: A misconfigured reporter, an accidental "Yes" answer to the question "Prereqs failed to install, try to continue anyway?", or other things.
There are some ways that a test report will warn about these things:
When we find one of these things, we should process the report in a different way:
If the report is a pass, it means that either the distribution dependencies can be relaxed, or the distribution's tests do not cover the cases where the dependency is needed.
We can add a note to the report explaining this
We could grade the report as INVALID or UNKNOWN or even FAIL
If the report is a fail, it means nothing: The distribution said it required libraries that are not present
There are often incoming reports that are clearly not good-faith effort to test the distribution: A misconfigured reporter, an accidental "Yes" answer to the question "Prereqs failed to install, try to continue anyway?", or other things.
There are some ways that a test report will warn about these things:
When we find one of these things, we should process the report in a different way:
If we are changing the report given to us by the reporter, we need to record the changes so we can track down issues with reporters in the future. See https://github.com/cpan-testers/cpantesters-schema/issues/31