willmurphyscode
commented
2 months ago From my initial reading, the best approach is to add a max_year field to the result_set config object. Then, yardstick-based quality gates can have a 2021 result set and a 2022 result set, and we can just move images from one to another as enough 2022 labels are added to the image.
Other options considered
Make a per-image config
I could change the images field on the result_set config to be a list[ImageConfig] instead of a list[str] and could be the year there, and write some custom parsing so that a bare string in the config gets inflated to ImageConfig(img: <the string>, max_year: cfg.default_max_year).
Override in the individual quality gates
Yardstick quality gates are typically implemented as little python scripts themselves (see #126, grype's exampe, vunnel's example), and I could put code in these gate.py files to check which image is being scanned and update ask for a yardstick comparison with that max year instead of the default max year.
This option probably requires the least design and might be easiest, but it amounts to hard-coding around the lack of this feature everywhere we want it. It will be cleaner and easier long term to make this a first-class feature in yardstick.
What would you like to be added:
I would like to be able to tell yardstick that the default max year for a given image is later (or earlier) than the default_max_year specified in the top level of the yardstick config.
Why is this needed:
Some images don't have enough vulnerabilities to compare if we exclude CVEs after 2021. This difficulty is especially true when adding new distros or versions of distros that GA'ed long after 2021.
Additional context:
Discovered trying to add quality gates for anchore/vunnel#569.