search

mitre / hipcheck

Automatically assess and score software repositories for supply chain risk.
https://mitre.github.io/hipcheck/
Apache License 2.0
74 stars 5 forks source link

Adding english-language description of final policy expr boolean evaluation #383

Open j-lanson opened 1 month ago

j-lanson commented 1 month ago

With RFD#0004, Hipcheck can handle an unlimited number of analyses, but we lose the hard-coded semantic information that made the report human-readable (i.e. the knowledge of concern format / output units and meaning was embedded into the code). When a plugin analysis fails by comparison, all we do currently is say "X was the output, the policy was Y". Printing these policies raw in the report output is not very ergonomic.

By necessity, the top of the AST for a policy expression must be an operation that returns a boolean. Investigate to what extent we can provide english language descriptions for this final operation, combined with the default_query_explanation() that each plugin provides, to give a generated explanation of why the policy fails. To start, I think limiting this to just the top-level boolean-returning op and its immediate operands is a good goal.

e.g. entropy, (eq 0 (count (filter (gt 8.0) $))) --> "expected the number of commit entropies passed through (count (filter (gt 8.0) $)) to equal 0, was X" review (lte $ 0.05) --> "expected the percent of commits receiving review to be less than 0.05, was X"

j-lanson commented 1 month ago

This will likely rely on #371 , as we need a structured JSON-pointer type to know which operand of the top-level boolean expression contains the metric data.

Also, should be able to "collapse" any portion of the expression that does not rely on JSON-pointer data. So when we handle (eq (sub 10 5) ..) we should be able to evaluate the entire first operand