adding NMS to an .mlpackage detection model

[ily-R]

❓Question

I have a detection keras model EfficientDetLite2 That I converted to mlprogram as follows:

ml_model = ct.convert(keras_model, inputs=[ct.ImageType(name='args_0', shape=(1, 448, 448, 3), scale=1 / 128.0, bias=[-127.0/128.0, -127.0/128.0, -127.0/128.0], channel_first=False)], convert_to = "mlprogram")
ml_model.save("efficientDetLite2.mlpackage")

it has two outputs:

• Identity representing the boxes_coordinates of shape (1, 37629, 4)
• Identity_1 representing the scores of shape (1, 37629)

✅ The model is converted successfully.

Now I wanted to add NMS to the model like follows:

coreml_model = ct.models.MLModel("efficientDetLite2.mlpackage")
spec = coreml_model.get_spec()

spec.description.output[0].type.multiArrayType.shape.append(1)
spec.description.output[0].type.multiArrayType.shape.append(37629)
spec.description.output[0].type.multiArrayType.shape.append(4)
spec.description.output[1].type.multiArrayType.shape.append(1)
spec.description.output[1].type.multiArrayType.shape.append(37629)
coreml_model = ct.models.MLModel(spec, weights_dir = coreml_model.weights_dir)

nms_spec = ct.proto.Model_pb2.Model()
nms_spec.specificationVersion = 6

for i in range(2):
    model_out = spec.description.output[0 if i == 1 else 1].SerializeToString()
    nms_spec.description.input.add()
    nms_spec.description.input[i].ParseFromString(model_out)
    nms_spec.description.output.add()
    nms_spec.description.output[i].ParseFromString(model_out)

nms_spec.description.output[0].name = 'confidence'
nms_spec.description.output[1].name = 'coordinates'

nms = nms_spec.nonMaximumSuppression
nms.confidenceInputFeatureName = "Identity_1"
nms.coordinatesInputFeatureName = "Identity"
nms.confidenceOutputFeatureName = "confidence"
nms.coordinatesOutputFeatureName = "coordinates"
nms.iouThresholdInputFeatureName = "iouThreshold"
nms.confidenceThresholdInputFeatureName = "confidenceThreshold"
nms.pickTop.perClass = True
nms.stringClassLabels.vector.extend(np.array(['object']))
nms.iouThreshold = 0.3
nms.confidenceThreshold = 0.3

output_shapes = [1, 4]
for i in range(2):
    ma = nms_spec.description.output[i].type.multiArrayType
    ma.shapeRange.sizeRanges.add()
    ma.shapeRange.sizeRanges[0].lowerBound = 0
    ma.shapeRange.sizeRanges[0].upperBound = -1
    ma.shapeRange.sizeRanges.add()
    ma.shapeRange.sizeRanges[1].lowerBound = output_shapes[i]
    ma.shapeRange.sizeRanges[1].upperBound = output_shapes[i]
    del ma.shape[:]

model_nms = ct.models.MLModel(nms_spec)
model_nms.save('nms.mlmodel')

✅ NMS model was saved successfully

Now I wanted to combine the models into a pipeline as follows:

input_features = [
    ('args_0', ctdt.Array(448, 448, 3)),
    ('iouThreshold', ctdt.Double()),
    ('confidenceThreshold', ctdt.Double())
]
output_features = ['coordinates', 'confidence']
pipeline = ct.models.pipeline.Pipeline(input_features, output_features)
pipeline.spec.specificationVersion = 6

pipeline.add_model(coreml_model)
pipeline.add_model(model_nms)

pipeline.spec.description.input[0].ParseFromString(spec.description.input[0].SerializeToString())
pipeline.spec.description.output[0].ParseFromString(nms_spec.description.output[0].SerializeToString())
pipeline.spec.description.output[1].ParseFromString(nms_spec.description.output[1].SerializeToString())

Now i want to save the pipeline as mlprogram

model_pipeline = ct.models.MLModel(pipeline.spec, weights_dir = coreml_model.weights_dir)
model_pipeline.save("pipeline.mlpackage")

This raises the following warning :

/Users/user1/miniconda3/envs/coreml_conv/lib/python3.9/site-packages/coremltools/models/model.py:144: RuntimeWarning: You will not be able to run predict() on this Core ML model. Underlying exception message was: Error compiling model: "compiler error:  compiler error:  Encountered an error while compiling a neural network model: at unknown location: Could not open /var/folders/8r/t63s22hs3fn653kxc82_38gc0000gn/T/weights/weight.bin".
  _warnings.warn(

running the convertion script with sudo gives the following warning:

/Users/user1/miniconda3/envs/coreml_conv/lib/python3.9/site-packages/coremltools/models/model.py:144: RuntimeWarning: You will not be able to run predict() on this Core ML model. Underlying exception message was: Error compiling model: "compiler error:  compiler error:  Encountered an error while compiling a neural network model: at unknown location: Could not open /tmp/weights/weight.bin".
  _warnings.warn(

In both cases I cant open the model with Xcode 14.1, i get the error missingMetadataField(named: "inputSchema")

I tried again with this saving medthod instead:

save_spec(pipeline.spec,"pipeline.mlpackage", weights_dir = coreml_model.weights_dir)

✅ I get No warnings ✅ I can load the model in Xcode 14.1 ❌ I get no detections even with confidenceThreshold = 0.0 . ❌ I can't load the model into Netron.app

I dont know what I did wrong, I am guessing im no passing the weight correctly, but even with wrong weights, I should get detection with 0.0 threhsold.

[TobyRoseman]

I think it's going to be a lot easier to add NMS to your Keras model, then convert that updated model to Core ML. I would recommend taking that approach.

[ily-R]

will it be slower ? Also coremltools doesnt support https://www.tensorflow.org/api_docs/python/tf/image/non_max_suppression

[TobyRoseman]

The converted model should not be any slower. For TensorFlow we support both NonMaxSuppression and NonMaxSuppressionV5.

[telemetrieTP23]

But does this mean that the "old" way of making a Pipeline with merging two or more mlmodel files (Model, (Postprocessing Model) and NMS) don't work with mlpackage files instead of mlmodel anymore ? Because starting with minimum_deployment_target=ct.target.iOS15 it is only possible to create a mlpackage with coremltools.convert. So a pipeline is only possible with minimum_deployment_target=ct.target.iOS14 ? Or is it somehow possible to get a "Standonly" mlmodel file after converting a pytorch model with minimum_deployment_target=ct.target.iOS15 & iOs16 ?

[aseemw]

Hi, thank you for filing this issue. I can confirm that this is unfortunately a bug, possibly an issue both in coremltools and CoreML framework , which doesn't initialize a pipeline model correctly, that contains an mlprogram.

I am going to mark this a bug, and we will be investigating it.

[otmb]

Has this issue been resolved? The sample below is an mlprogram with NMS added.

https://gist.github.com/otmb/72cac326837bff4b4ad1dda508a91428

Thank.