search

oasis-tcs / sarif-spec

OASIS SARIF TC: Repository for development of the draft standard, where requests for modification should be made via Github Issues
https://github.com/oasis-tcs/sarif-spec
Other
169 stars 47 forks source link

Add version to the reportingDescriptor object #535

Open davidknise opened 2 years ago

davidknise commented 2 years ago

Hey SARIF team!

We would like to propose adding a version field to the reportingDescriptor object, which should adhere to Semantic Versioning.

Analyzer teams make updates to their rule implementations, but ultimately, don't have a mechanism to communicate changes at the rule level.

As a result reader wanting to know if a rule change, we:

  1. Can assume every rule implementation is updated with every new tool version (too broad)
  2. Ask SARIF writers to adhere to writing semantic versions in the reportingDescriptor's property bag

Adding the field would cement Option 2 and be a dependable communication mechanism.

As a result reader, we hope to be able to differentiate between new result findings from existing functionality and new result findings from updated implementations.

Thanks, Dave

michaelcfanning commented 1 year ago

Hey, David! It's been a while since you filed this. We are spinning up an incremental SARIF 2.2. revision.

I am sympathetic to this request for sure. As a tools developer, in my day job at Microsoft, I notice we spend an inordinate amount of time maintaining a change manifest that attempts to call out rule updates.

I am not sure that a per-rule semver value would help us. My first concern would be how/whether this would be maintained. A second question would be whether the literal versioning updates can drive consumer scenarios sufficiently. Possibly! I guess nearly any change in rule version might be useful as a clue to you to provide special handling of analysis delta.

But! Even with rule reversioning, you won't be able to be absolutely sure that analysis delta results from rule change rather than code change (assuming that both your tool version and your rules have changed).

Our philosophy in analysis infrastructure that I'm involved in is that this scenario requires automation to have two execution + baselining cases. The first scenario is 1) scan targets have changed but the tool has not (since last run), 2) scan targets have not changed (since last run) but the tool has. By controlling for changes only on one side, you can implement precise handling for analysis delta.

With this result management approach, the current versioning information at the tool component level is arguably sufficient. I.e., if you see a change, any change, you initiate a rebaselining run against the previous run's scan targets, using the updated tool.

I can appreciate, of course, that this approach could be readily implemented in a green field scenario and very difficult to retrofit if you weren't already designed for it.

Please let me know your thoughts.