Core ML Model performance far lower on iOS 17 vs iOS 16 (iOS 17 not using Neural Engine)

[mrdbourke]

🐞Describing the bug

Hello,

I'm not sure if this is 100% coremltools related or an iOS level issue.

But I've found since iOS 17 release that none of my Core ML models running on devices with iOS 17 use the neural engine, thus resulting in far slower performance.

The following screenshots show the performance of the same model (a PyTorch computer vision model) on an iPhone SE 3rd gen and iPhone 13 Pro (both use the A15 Bionic).

iOS 16 - iPhone SE 3rd Gen

iOS 16 uses the ANE and results in fast prediction, load and compilation times.

iOS 17 - iPhone 13 Pro

iOS 17 doesn't seem to use the ANE, thus the prediction, load and compilation times are all slower.

To Reproduce

The following is my code I'm using to export my PyTorch vision model.

I've used the same code for the past few months with sensational results on iOS 16.

# Convert to Core ML using the Unified Conversion API
coreml_model = ct.convert(
    model=traced_model,
    inputs=[image_input],
    outputs=[ct.TensorType(name="output")],
    classifier_config=ct.ClassifierConfig(class_names),
    convert_to="neuralnetwork",
    # compute_precision=ct.precision.FLOAT16,
    compute_units=ct.ComputeUnit.ALL
)

System environment:

• Xcode version: 15.0
  • coremltools version: 7.0.0
  • OS (e.g. MacOS version or Linux type): Linux Ubuntu 20.04 (for exporting), macOS 13.6 (for testing on Xcode)
  • Any other relevant version information (e.g. PyTorch or TensorFlow version): PyTorch 2.0

Additional context

• This happens across "neuralnetwork" and "mlprogram" type models, neither use the ANE on iOS 17 but both use the ANE on iOS 16

If anyone has a similar experience, I'd love to hear more.

Otherwise, if I'm doing something wrong for the exporting of models for iOS 17+, please let me know.

Thank you!

[aseemw]

Can you please share the CoreML model which runs on NE with iOS16 but not with iOS17 (on the same device). If everything else is same (model, device) and the only change is the iOS version , this would be a bug in the Core ML Framework (not coremltools since the model is same). You can either share the model here or file a bug report and attach it there.

[mrdbourke]

Hi @aseemw ,

Thank you for the response.

Please find the attached model:

2023-09-26_food_not_food_2_classes.mlmodel.zip

Yeah I think it may be a Core ML issue, see how you go with the model and if it doesn't work, I'll go ahead and try filing a bug report.

[mrdbourke]

Update: also posted this on the Apple Developer Forums, I'll report any further updates here too - https://developer.apple.com/forums/thread/739286

[Moeflon]

I faced a similar issue with iOS 17. Not only was my performance way worse, the output of the model was way off. In my case, the confidence column in a YOLOv8 model contained integer values orders of magnitude above 1.

I currently restored my model's performance (both in terms of speed and accuracy) on iOS 17 by downgrading coremltools to 6.2 and exporting to a .mlmodel instead of a .mlpackage. Perhaps this also solves your issues.

[TobyRoseman]

@Moeflon - Please create a new GitHub issue with steps to reproduce your problem that was solved by downgrading coremltools.

[mrdbourke]

I faced a similar issue with iOS 17. Not only was my performance way worse, the output of the model was way off. In my case, the confidence column in a YOLOv8 model contained integer values orders of magnitude above 1.

I currently restored my model's performance (both in terms of speed and accuracy) on iOS 17 by downgrading coremltools to 6.2 and exporting to a .mlmodel instead of a .mlpackage. Perhaps this also solves your issues.

@Moeflon glad to hear downgrading worked for you.

Is all that you changed the coremltools version?

Perhaps it may be coremltools package related then?

[Moeflon]

@Moeflon - Please create a new GitHub issue with steps to reproduce your problem that was solved by downgrading coremltools.

I will attempt to reproduce the issue with a non-confidential model tonight

[SupenBook]

same on ios 18