Inferred metrics (e.g. infNDCG) are only available for gold standards that contain strata information. This was not the case for CT 2017.
When we create an aggregated gold standard containing CT 2017, we ended up losing this information, even though it was available for CT 2018. However, the stratum for CT 2017 is only one, 1.
Consider using this heuristic to be able to calculate inferred metrics for aggregated gold standards even if only a subset of it has more than one strata.
I'm not sure what would happen here; worst case scenario I expect regular metrics for 2018 and nulls for 2017. That's still good, since currently we get nulls for everything, thus losing information.
Inferred metrics (e.g. infNDCG) are only available for gold standards that contain strata information. This was not the case for CT 2017.
When we create an aggregated gold standard containing CT 2017, we ended up losing this information, even though it was available for CT 2018. However, the stratum for CT 2017 is only one, 1.
Consider using this heuristic to be able to calculate inferred metrics for aggregated gold standards even if only a subset of it has more than one strata.
I'm not sure what would happen here; worst case scenario I expect regular metrics for 2018 and nulls for 2017. That's still good, since currently we get nulls for everything, thus losing information.