pleary
commented
4 months ago I'm going to close this as what was done for the Common Ancestor task was essentially an instance of this - optimizing precision and accuracy of "leaf rollup" or score aggregation to achieve an end goal. In that ticket it was to optimize a common ancestor. We still don't have an end goal in mind for what we've called "leaf rollup". So until we can clearly define what we want, I think we have what we can do for now. The parameters for aggregation are configurable, and we can incorporate precision/recall/F1 into test results. If we can define what the end goal of "leaf roll up" is, let's make another ticket to customize the aggregation parameters for that goal to maximize F1
Currently our model implementations that return multiple suggestions only ever return leaf nodes in the model. Sometimes these leaf nodes have a very low likelihood of being the correct taxon. If we could combine multiple leaf suggestions and offer one of their ancestors as a suggestion instead, that ancestor will have a higher likelihood of being accurate, while being less precise.
This task is to determine the ideal thresholds of when to roll up suggestions to a higher taxon (e.g. if combined_score is below a certain value) and when to stop rolling up (e.g. if a suggestion is at a certain rank_level, stop rolling up). We should consider things like, when rolling-up scores, should we use an iconic taxon filter of available as we do now, or never filter by iconic taxon.
Ideally these thresholds could be determined given only the modes and associated data, and confirmed observations to test against. That way we could determine new thresholds for every model, and model version - e.g. use different thresholds on mobile devises that use a different export of the trained models