search

microsoft / onnxruntime

ONNX Runtime: cross-platform, high performance ML inferencing and training accelerator
https://onnxruntime.ai
MIT License
14.71k stars 2.93k forks source link

[Performance] fp16 support and performance #22242

Open cbingdu opened 1 month ago

cbingdu commented 1 month ago

Describe the issue

FP16 model inference is slower compared to FP32. Does FP16 inference require additional configuration or just need to convert the model to FP16

To reproduce

convert onnx model from fp32 to fp16 using onnxmltools onnxruntime c++ liblary inference（convert inputs and outputs data format from fp32 to fp16）

Urgency

No response

Platform

Android

OS Version

34

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

1.18.0

ONNX Runtime API

C++

Architecture

ARM64

Execution Provider

Default CPU

Execution Provider Library Version

No response

Model File

No response

Is this a quantized model?

Yes

wejoncy commented 1 month ago

Please share more information of your question. Such as which platform, what EP do you use? what's the model looks like?

DakeQQ commented 1 month ago

I've encountered the same issue: when running inference on *.onnx Float16 models (such as: LLM, YOLO, VAE, Unet, Bert...) directly with the CPU, there is no noticeable speedup. This is a significant problem because one would expect a performance gain from using Float16.

Moreover, if you convert the model to the *.ort format, the conversion tool automatically inserts a Cast operator that converts FP16 back to FP32. This automatic conversion completely negates any potential acceleration benefits we might have gained by using the NNAPI runtime with FP16 after converting to *.ort.

Given that CPUs supporting Arm64-v8.2 and later versions do indeed support FP16 computations, I would greatly appreciate it if ONNX Runtime could prioritize the implementation of ARM-CPU-FP16 support. This feature would be highly beneficial for many users and would significantly improve the efficiency of mobile models.

skottmckay commented 1 month ago

We're working on adding more fp16 support on arm64 as well as gpu support (which would handle fp16 models as well).

github-actions[bot] commented 2 weeks ago

This issue has been automatically marked as stale due to inactivity and will be closed in 30 days if no further activity occurs. If further support is needed, please provide an update and/or more details.

leigaol commented 2 weeks ago

+1 to this