The scripts, and Animl in general, doesn't know when a model was deployed, renamed, or automation rules applied, and we currently do not store inference request data at the image level (though we should), so it's up to the user to ensure the model was used for the entire date range.
Currenlty, if Animl never requested inference for the model being analyzed for some image(s) in the date range, but there are validating labels in those images, those images will be counted as false negatives, which will significantly skew the results (model will appear to to have worse recall than it actually does).
We could minimize this as much as possible by adding a step early on in the scripts that queries the Project for all images that have ml-generated Labels from whatever model were analyzing, and using the first instance of the model's use and last to adjust the START_DATE and END_DATE.
We'd definitely want to notify the user that this was happening in the console.
In scripts/analyzeMLObjectLevel.js and scripts/analyzeMLSequenceLevel.js it is assumed that the model being analyzed was used for the entire duration of the date range.
The scripts, and Animl in general, doesn't know when a model was deployed, renamed, or automation rules applied, and we currently do not store inference request data at the image level (though we should), so it's up to the user to ensure the model was used for the entire date range.
Currenlty, if Animl never requested inference for the model being analyzed for some image(s) in the date range, but there are validating labels in those images, those images will be counted as false negatives, which will significantly skew the results (model will appear to to have worse recall than it actually does).
We could minimize this as much as possible by adding a step early on in the scripts that queries the Project for all images that have ml-generated Labels from whatever model were analyzing, and using the first instance of the model's use and last to adjust the
START_DATEandEND_DATE.We'd definitely want to notify the user that this was happening in the console.
Related: #76