search

microsoft / onnxruntime

ONNX Runtime: cross-platform, high performance ML inferencing and training accelerator
https://onnxruntime.ai
MIT License
13.52k stars 2.76k forks source link

Execution Provider bridge for TFLite Delegates for Coral Edge TPUs #10248

Open rgov opened 2 years ago

rgov commented 2 years ago

ONNX Runtime cannot yet target Coral Edge TPU accelerators.

Coral normally uses Tensorflow Lite with a loadable TFLite Delegate that communicates with the hardware called libedgetpu.

It appears that the goals of ONNX Runtime Execution Providers and TFLite Delegates are similar so perhaps it would be possible to create a wrapper to bridge the APIs (docs). If such a bridge existed it would be possible to target Edge TPU devices.

TFLite can also target some of the same accelerators as ONNX Runtime (NNAPI, CoreML, etc.). Having the bridge could also be useful for debugging issues by reproducing them on an alternative code path.

ankitmaurya001 commented 1 year ago

Plus 1 for the feature.

mszsorondo commented 1 year ago

Hi! One question: By "bridge" you mean a bridge between Coral's Edge TPU TFLite Delegate and a new ONNX EP? or directly between ONNX EP and libedgetpu, inspired in the Edge TPU TFLite Delegate?

rgov commented 1 year ago

I would guess that adding TFLite as an ONNX EP, or creating an EP that implements the TFLite delegate API, would be better choices than talking to libedgetpu itself.

It appears that the TFLite delegate is the supported method of using the Edge TPU, rather than using libedgetpu directly. Talking to the delegate instead insulates you from changes in the (semi-undocumented, potentially unstable) libedgetpu API.

Secondly, this would make it possible to use other TFLite delegates in the future, either to introduce support for new hardware that becomes available, or just for testing purposes.

As a practical matter though, if the libedgetpu is significantly easier to integrate with, that may be better. TFLite has a complex build system, and the set of ops it supports is determined at build time, etc. Also, converting a model from ONNX to TFLite requires using TensorFlow to do an intermediary conversion; that may require calling out to Python.

Some time ago I ran an experiment previously to see which of the ONNX Model Zoo could be converted to TFLite. Results are here.

SpiderUnderUrBed commented 1 week ago

+1, this would be useful as it would help immich, which I self host, work sooner! (with Coral TPU's)

chelming commented 1 week ago

+1, this would be useful as it would help immich, which I self host, work sooner! (with Coral TPU's)

it might not work with immich, regardless, because of the size of the model they use. coral is limited to ~6mb model.