Core ML perf regression observed on non-genai models

[guangy10]

🐛 Describe the bug

We are running exercise on the newly launched benchmarking infra with the in-tree enabled models under examples/models. You can follow the instruction to extract and inspect the benchmark_results.json. (Connecting the results to a dashboard is still WIP)

The preliminary data points collected from the benchmarking infra can be found here: https://github.com/pytorch/executorch/tree/main/extension/benchmark#preliminary-benchmark-results. It shows some previously enabled models, e.g. MobileNetV2/3, InceptionV3/4, are running expected slower with Core ML delegates.

The models are exported using the coreml export scripts (code pointer), and the app that loads and runs the exported model is located under extension/apple/benchmark.

Versions

From latest executorch main

[guangy10]

cc: @cbilgin @kimishpatel @shoumikhin

[cymbalrush]

Something weird is going on, the numbers are too high. Investigating.

[guangy10]

Here is the profiling of mv3 running on the executor_runner binary:

...
│ 186 │ Execute            │ DELEGATE_CALL                                                       │  86.8285      │  86.8285      │  86.8285      │  86.8285      │  86.8285      │  86.8285      │ ['aten.convolution.default', 'aten._native_batch_norm_legit_no_training.default' ... 'aten.addmm.default'] (317 total) │ False             │ CoreMLBackend           │ nan               │           nan           │ nan                       │ nan                        │
├─────┼────────────────────┼─────────────────────────────────────────────────────────────────────┼───────────────┼───────────────┼───────────────┼───────────────┼───────────────┼───────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼───────────────────┼─────────────────────────┼───────────────────┼─────────────────────────┼───────────────────────────┼────────────────────────────┤
│ 187 │ Execute            │ Method::execute                                                     │  86.8298      │  86.8298      │  86.8298      │  86.8298      │  86.8298      │  86.8298      │ []                                                                                                                     │ False             │                         │ nan               │           nan           │ nan                       │ nan                        │
╘═════╧════════════════════╧═════════════════════════════════════════════════════════════════════╧═══════════════╧═══════════════╧═══════════════╧═══════════════╧═══════════════╧═══════════════╧════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╧═══════════════════╧═════════════════════════╧═══════════════════╧═════════════════════════╧═══════════════════════════╧════════════════════════════╛

The avg of Method::execute is 86.829ms, far different from the numbers get from the benchmark app. @shoumikhin Is it same as how test_forward is measured in the app?

[YifanShenSZ]

Did you see any graph breaks?