To help identify and eliminate unintentional vulnerabilities in generated test suites, we can add a debug feature that detects vulnerabilities found in the test suite by tools outside of the intentional vulnerabilities. I can envision two modes during scorecard generation:
1) Normal Debug: Generate a list of CWEs in the current expected results file, and then for each tool, each time a CWE in that list is identified outside of an expected test case for that CWE, dump out the CWE, and location found.
This is the most useful as it eliminates extraneous vulns of the type we actually count so tools don't 'strangely' find more vulns of a given type than expected.
2) Go Hard Debug: Report ANY CWEs outside of expected locations. Some tools find all kinds of extraneous stuff, so eliminate any findings mapped to 0 or -1. We might have to tune other parsers to map more extraneous findings to 0/-1, or add a filter her to eliminate CWEs we just don't care about.
This would all be done in BenchmarkScore, so would work across ALL the tools.
To help identify and eliminate unintentional vulnerabilities in generated test suites, we can add a debug feature that detects vulnerabilities found in the test suite by tools outside of the intentional vulnerabilities. I can envision two modes during scorecard generation:
1) Normal Debug: Generate a list of CWEs in the current expected results file, and then for each tool, each time a CWE in that list is identified outside of an expected test case for that CWE, dump out the CWE, and location found.
This is the most useful as it eliminates extraneous vulns of the type we actually count so tools don't 'strangely' find more vulns of a given type than expected.
2) Go Hard Debug: Report ANY CWEs outside of expected locations. Some tools find all kinds of extraneous stuff, so eliminate any findings mapped to 0 or -1. We might have to tune other parsers to map more extraneous findings to 0/-1, or add a filter her to eliminate CWEs we just don't care about.
This would all be done in BenchmarkScore, so would work across ALL the tools.