search

MicrosoftDocs / azure-docs

Open source documentation of Microsoft Azure
https://docs.microsoft.com/azure
Creative Commons Attribution 4.0 International
10.27k stars 21.45k forks source link

Why is there a difference between the Iteration Prediction and Quick Test Prediction #21818

Closed bl00apple closed 5 years ago

bl00apple commented 5 years ago

I was looking through the Training images in my Iteration page. An image was supposed to be predicted as 'check' but it predicted it as 'striped'(the 'striped' probability was 83%) So, I copied the image url of that image, and ran it trough the 'Quick Test'.

On the 'Quick Test', it correctly predicted the image as 'check' and the probability was 99%. Why is it that even if I use the same image, the probability differs depending on which part the image is trained on?

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ChiragMishra-MSFT commented 5 years ago

@bl00apple Thanks for the feedback. We are currently investigating into the issue and will update you shortly.

ChiragMishra-MSFT commented 5 years ago

@bl00apple Firstly, After training, the Performance is displayed. The precision and recall indicators tell you how good your classifier is, based on automatic testing. Custom Vision Service uses the images that you submitted for training to calculate these numbers, by using a process called k-fold cross validation.

The classifier uses all the images to create a model that identifies each tag. To test the quality of the model, the classifier tries each image on the model to see what the model finds.

So basically the result that you see in the Iteration Prediction is the one that was predicted by the model. But after training, the trained model now has this image used as a "tag", so it knows that it belongs to a particular tag. Hence, you get accurate results in the quick test. Usually this happens in corner cases where a particular image is difficult to be categorized. Having that said, it's good for the model to tag(categorize) such corner cases like different camera angles, lighting, background, types, styles, groups, sizes, etc. It is recommended to use a variety of photo types to ensure that your classifier is not biased and can generalize well.

Hope this helps.

bl00apple commented 5 years ago

I have one more question. Most of the images that were predicted wrong in the Iteration Prediction were predicted correct when I retried it in the Quick Test. But there are a few images still predicting it wrong even in the Quick Test area. You said that after the training, the image knows that it belongs to a particular tag. Is this particular tag the one that I've classified before the test? I'm kind of confused here, if it now knows that it belongs to a particular tag, why does it still incorrectly predict the image in the Quick Test area?

ChiragMishra-MSFT commented 5 years ago

@bl00apple As mentioned in the documentation here, the algorithm trains to this data and calculates its own accuracy by testing itself on that same data. Once the model is trained, you can test, retrain, and eventually use it to classify new images according to the needs of your app.

How it works is that the algorithm identifies characteristics from the test data for a particular tag - for example for an apple, the characteristics would be the shape, color, etc. So it's not actually a 1 to 1 mapping of the test images with a particular tag. Once you have your trained model ready, you can do a quick test on an image using the trained model.

Hence, even the trained model may predict some images wrong. To improve the accuracy of the model, it is recommended that you train your classifier using a series of images (with a minimum of 30 images per tag).

For more information, please refer :

Azure Custom Vision Doc

Build a classifier

YutongTie-MSFT commented 5 years ago

@bl00apple We will now proceed to close this thread. If there are further questions regarding this matter, please respond here and @YutongTie-MSFT and we will gladly continue the discussion.