Core ML predictions include confidence score

[atomicguy]

Would it be possible to add confidence score to label predictions from a Core ML model?

I have a Core ML model to predict object detection bounding boxes. I ran it on a set of images I have, but some low confidence predictions are included in the results.

Could it be possible to include the confidence in the predictions so I can filter them after they have been run? Or an interface to choose confidence threshold before prediction?

[ryouchinsa]

Thanks for writing the issue.

For object detection models, you can set the 'confidence_threshold'. https://rectlabel.com/help#load_coreml https://github.com/ryouchinsa/Rectlabel-support/blob/master/coreml_yolov3.py

coreml_model.user_defined_metadata['confidence_threshold'] = '0.25'

[ryouchinsa]

To edit the 'confidence_threshold' for existing models, you can use this code.

import coremltools
import sys

model_path = sys.argv[1]
coreml_model = coremltools.models.MLModel(model_path)
coreml_model.user_defined_metadata['confidence_threshold'] = '0.25'
coreml_model.save(model_path)

Currently If you need our support to fix this problem, please let us know.

[atomicguy]

I tried the code you suggested. I also tried adjusting the nonMaximumSuppression.confidenceThreshold value for my model. Neither seemed to make any change on the results when predicting with RectLabel

[ryouchinsa]

Thanks for your feedback.

Both ways are working for our Turi Create and YOLO models.

How did you train your model? Please tell us the version of Turi Create or Create ML. https://apple.github.io/turicreate/docs/userguide/object_detection/export-coreml.html

[atomicguy]

I trained the model with Create ML 2.0 (53.1). It's an object detection model, which I think means it's YOLO under the hood.

Predictions in Create ML show three 100% and one 60% prediction. When I run it in RectLabel after setting the confidence to 0.90, all 4 predictions are captured.

[ryouchinsa]

Thanks for the details.

We found that 'confidence_threshold' works when include_non_maximum_suppression=False. model.export_coreml('MyDetector.mlmodel', include_non_maximum_suppression=False)

We improved so that 'confidence_threshold' works when include_non_maximum_suppression=True. model.export_coreml('MyDetector.mlmodel', include_non_maximum_suppression=True)

We think that Create ML saves the Core ML model as include_non_maximum_suppression=True. https://apple.github.io/turicreate/docs/api/generated/turicreate.object_detector.ObjectDetector.export_coreml.html

We submitted the new update to Apple. When the new update is released, we will let you know. Thank you.

[ryouchinsa]

The new update version 3.04.2 was released. To show the new update on Mac App Store, press command + R to reload.

• Fixed so that 'confidence_threshold' is used for the model exported with non-maximum suppression.
• Added undo/redo for "Process the image using Core ML model".
• Improved so that the spinner is shown during processing.
• Improved so that backing up images/annotations is done in async.

We trained raccoon dataset using Create ML and confirmed that we can change the 'confidence_threshold' using the python script above. https://public.roboflow.com/object-detection/raccoon

Let us know your feedback.

[atomicguy]

Now when I try to load a model it says "Model Metadata description is empty."

Attached is my model and an example picture.

example.zip

[ryouchinsa]

Thanks for your feedback.

It is not an error. Literally just the Model Metadata description is empty in the model. RectLabel shows the description when finished loading the model.

1. You can set the description through python script.

import coremltools
import sys

model_path = sys.argv[1]
coreml_model = coremltools.models.MLModel(model_path)
coreml_model.short_description = "PairDetectorThresh.mlmodel"
coreml_model.user_defined_metadata['confidence_threshold'] = '0.9'
coreml_model.save(model_path)

2. You can set the description on Create ML when you create the project.

Let us know your feedback.

[atomicguy]

Awesome, that worked! I think I missed the message before because it didn't show up in dark mode prior to this update.