CoreML on Arm64 compilation error

[barakugav]

First I want to thank you for your support and effort developing and maintaining this library, its amazing!

Im trying to run on my MacBook with arm64 M3 and MPS, and im getting an error while compiling with the coreml feature:

  = note: Undefined symbols for architecture arm64:
            "___isPlatformVersionAtLeast", referenced from:
                -[CoreMLExecution predict:outputs:getOutputTensorDataFn:] in libonnxruntime.a[51](model.mm.o)
          ld: symbol(s) not found for architecture arm64
          clang: error: linker command failed with exit code 1 (use -v to see invocation)

This is the dependency in Cargo.toml:

ort = { version = "2.0.0-rc.2", features = ["coreml"] }

Am I doing something wrong? Am I missing some feature flags? Thanks again for the support

[decahedron1]

https://ort.pyke.io/perf/execution-providers#coreml

[barakugav]

Thanks, that indeed solve it. I have an M3 with 16-core Neural Engine and 'Metal Performance Shaders' (MPS), but i see running with CoreML is slower that running with CPU using ort, any idea why? @decahedron1 If instead of using ort I use pyo3 and regular torch with mps as device, I get x3 speedup

[decahedron1]

The CoreML EP does have some limitations with regard to what operators are supported. Unsupported operators will fall back to running on the CPU: https://onnxruntime.ai/docs/execution-providers/CoreML-ExecutionProvider.html#supported-operators

[barakugav]

And does CoreML know to use MPS? Is there any way else I can hint ort to use MPS? Is there anyway to further optimize the model other than .with_optimization_level(Level3)? Also out of curiosity, are optimizations performed on the graph done by the Rust code or by the underlying Microsoft cpp onnxruntime?

[decahedron1]

And does CoreML know to use MPS? Is there any way else I can hint ort to use MPS?

By the looks of things, no.

Is there anyway to further optimize the model other than .with_optimization_level(Level3)?

onnxsim

Also out of curiosity, are optimizations performed on the graph done by the Rust code or by the underlying Microsoft cpp onnxruntime?

All ort does is bind the C++ API to Rust. All the inference and optimization occurs exclusively on the C++ side.

[barakugav]

Thanks a lot! I appreciate the quick responses and support :) Love the library