Closed barakugav closed 2 months ago
Thanks, that indeed solve it.
I have an M3 with 16-core Neural Engine and 'Metal Performance Shaders' (MPS), but i see running with CoreML is slower that running with CPU using ort
, any idea why? @decahedron1
If instead of using ort
I use pyo3
and regular torch
with mps
as device, I get x3 speedup
The CoreML EP does have some limitations with regard to what operators are supported. Unsupported operators will fall back to running on the CPU: https://onnxruntime.ai/docs/execution-providers/CoreML-ExecutionProvider.html#supported-operators
And does CoreML know to use MPS? Is there any way else I can hint ort to use MPS?
Is there anyway to further optimize the model other than .with_optimization_level(Level3)
?
Also out of curiosity, are optimizations performed on the graph done by the Rust code or by the underlying Microsoft cpp onnxruntime?
And does CoreML know to use MPS? Is there any way else I can hint ort to use MPS?
By the looks of things, no.
Is there anyway to further optimize the model other than
.with_optimization_level(Level3)
?
Also out of curiosity, are optimizations performed on the graph done by the Rust code or by the underlying Microsoft cpp onnxruntime?
All ort
does is bind the C++ API to Rust. All the inference and optimization occurs exclusively on the C++ side.
Thanks a lot! I appreciate the quick responses and support :) Love the library
First I want to thank you for your support and effort developing and maintaining this library, its amazing!
Im trying to run on my MacBook with arm64 M3 and MPS, and im getting an error while compiling with the
coreml
feature:This is the dependency in
Cargo.toml
:Am I doing something wrong? Am I missing some feature flags? Thanks again for the support