microsoft / AISkillsForWindows

Contains samples for implementing Windows Skills by extending the preview base API and using exsting skill packages
https://docs.microsoft.com/en-us/windows/ai/windows-vision-skills/
MIT License
166 stars 46 forks source link

Output ObjectDetectorPreview Confidence Level with Matches #64

Closed mobilewares closed 4 years ago

mobilewares commented 4 years ago

Request for ObjectDetectorPreview : Is it possible to output the confidence level with each match - like what is returned for Windows ML / Custom Vision ONNX files? The Evaluations are returning some poor results in some cases (ie. mixing up leaves on a tree branch as a 'Person' from a 4k security camera is one ongoing issue) - I'm pretty sure ability to filter out low confidence matches would make a big difference here.

LPBourret commented 4 years ago

Thank you for the feedback, let me thinker on this. We do filter out low probabilities before exposing the results. In this case, the leaves may have a person-like shape, which is hard to discern, but I agree that exposing the confidence could help at least debug this issue. We could also expose the confidence threshold we use to filter out results as an optional input parameter that you can set (i.e. only report very high confidence results).

LPBourret commented 4 years ago

Hi @mobilewares please take a look at the new release of the ObjectDetector Windows Skill nuget package that exposes 2 new input features for you to filter both by type of object and by the confidence reported, This confidence is also now newly reported by the ObjectDetectorResult class.

mobilewares commented 4 years ago

Thanks for adding these features - much appreciated. I will have a play in next couple of days and see if it can help improve results (either way, having the confidence coming back was also on our wish list for other projects).

mobilewares commented 4 years ago

Hi,

I just tried out the updated sample app & it appears the invalid object detection matches are returning very high confidence - hence filtering out based on confidence is not going to work in this specific case unfortunately.

I'm guessing this would likely indicate a problem with the model (rather than the code/library) - see excerpt below (from security camera) - it's 95.6% convinced that this tree is a person.

image

thanks Niall