There doesn't seem to be a way to embed ids into FI reports

[evverx]

I'm not sure how exactly FI is supposed to be used but it seems to me that in general building at least two reports and comparing them would be a reliable way to figure out whether anything has been improved. To make it easier to tell reports apart it would be great if there was a way to embed SHAs and/or dates into reports. Dates and SHAs would be useful on OSS-Fuzz too to make it clear that for example https://storage.googleapis.com/oss-fuzz-introspector/systemd/inspector-report/20220422/fuzz_report.html is out of date and shouldn't be relied on (The date is included in the link there but it took me a while to find it and I think it would be better if it was in the report itself).

More generally I think it would be ideal if FI could generate diffs of some kind to make it easier to compare two reports by analogy with how for example https://coveralls.io/github/systemd/systemd?branch=main works but I think for the time being SHAs/dates should at least make it easier to figure out where reports come from.

[MarahGamdou]

Hi, I'm wondering if the commit SHAs have been included in the reports or if it is possible to include them. If not, do you think the date of report generation can be reliable to deduce commit SHAs? Thank you!

[DavidKorczynski]

We have dates in the report now, which can be used for tracking.

We can add sha's of commits to e.g. the metadata section. Not all projects rely on git or version control systems as such (xpdf if I remember correctly) so I think it's smartest to keep it as an optional.

Diffing two reports is useful -- perhaps a way to start this is to include more summary and easy-to-digest data at the top of the report. A more thorough diffing is also interesting but will take some more effort.

[MarahGamdou]

@DavidKorczynski Thanks! I agree that this may not be useful for all projects, so it's better to keep it optional as you mentioned. I also think sha's of commits can be very useful to track the exact code that was used to generate the reports. Sha's of commits are also integrated into OSS-Fuzz reports now (in the fuzzer logs and bug reports).

[evverx]

@MarahGamdou I'm not sure if it helps but as a stopgap (assuming the projects you're interested in use git) it should be possible to embed SHAs into FI build logs by adding something like

printf 'SHA: %s\n' $(git rev-parse HEAD)

to OSS-Fuzz build scripts (they are usually named build.sh in the OSS-Fuzz repository). This way at least the last seven FI reports can be reliably mapped onto their source by going to the "Introspector Builds" tab at https://oss-fuzz-build-logs.storage.googleapis.com/index.html and looking for the "SHA" string in the logs. It can't be easily automated unfortunately so use cases like attesting that certain reports come from particular commits for example or keeping track of coverage automatically can't be covered by that stopgap.

[MarahGamdou]

@evverx Thanks for suggesting this stopgap! My use case is actually attesting that certain FI reports come from particular commits because I use the reports as fuzzing evidence.

[evverx]

I think to fully cover this use case OSS-Fuzz should provide provenance of some kind that can be processed automatically without having to parse HTML reports and apart from SHAs it should probably include links to repositories, raw data like https://github.com/ossf/fuzz-introspector/issues/485 and some other metadata. Having said that it's an advanced feature and I don't think there are a lot testing services built with use cases like that in mind (and it's understandable because this level of quality assurance where testing artifacts are used like that can't be even reached for various reasons in most places).

I think at this point it's possible to semi-automatically gauge whether the latest snapshots of projects are fuzzed or not using links to the last successful builds but that's mostly useful in environments where packages always roll forward on a regular basis without any downstream patches.