Support passing GPU buffer directly to GLDelegate

asus4 commented 4 years ago

The current redundant workflow 🤦 Unity GPU → Unity CPU → TFL CPU → TFL GPU Delegate → TFL CPU → Unity CPU → Unity GPU

TFL supports binding GPU buffer to the GPU Delegate.

https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/g3doc/performance/gpu_advanced.md

WHSnyder commented 4 years ago

Hi there, I wish I had seen this project a couple weeks ago! I've been working on a fork of tensorflow allowing for easy integration with rendering pipelines https://github.com/WHSnyder/tensorflow/tree/general_opengl, so pretty much this exact issue. So far I've just gotten the delegates to build for MacOS and Ubuntu and decoupled OpenGL from OpenGLES. The GL delegate works on Ubuntu now (given you have created a context beforehand) and the Metal delegate no longer crashes when used on MacOS (CoreGraphics wasn't linked in the /lite/delegates/gpu BUILD file). I also added some code to expose the Metal/GL delegates in the Python API (the pyexposure files).

However, I have not been able to get any of these delegates to work with this style transfer example https://www.tensorflow.org/lite/models/style_transfer/overview or this one https://github.com/tensorflow/examples/tree/master/lite/examples/style_transfer I get the exact same error as discussed here https://github.com/tensorflow/tensorflow/issues/32101 I've been using the Python API to create & call delegates, since that hasn't worked I've dedicated today to testing with the C++ API. Sorry to bombard you with links, once I get this worked out I'll see if I can merge it cleanly into this repo.

The GPU buffer binding you mention is only for Metal, but once I get everything working with the style transfer example I'll write a couple functions to expose the buffers for the GL delegate.

EDIT: Didn't notice the TfLiteGpuDelegateBindBufferToTensor function. Nobody seems to mention it, I'll see if I can get it working and implement a BindTextrueToTensor function for render target usage.

asus4 commented 4 years ago

Hi @WHSnyder, Thank you for your interest. This repository does not contain any TensorFlow's code. Please make PR to TensorFlow repository then I'll build the shared lib 😃

WHSnyder commented 4 years ago

@asus4 Wow I jumped the gun on that one, took a quick glance at the description and thought it depended on missing functionality in TFLite itself. Guess I got too excited that my project might be directly relevant :weary:. I suppose there's still a need in TFLite for a bindTensorToGLTexture function to avoid the texture to buffer copy, though it's not expensive in the grand scheme of things... I'm still absorbing the backend structure of TFLite GPU delegates, there's undocumented functionality I keep finding like the CLDelegate's BindBufferToTensor method. I'll post here again if I push any GLDelegate changes to the main TF repo

liamwalsh commented 4 years ago

For those working with ARFoundation you can indeed skip a fair few of the steps as outlined above.

mathtasatlime commented 3 years ago

@WHSnyder do your have any updates on that issue? atm I'm facing the problem that the readPixels functions blocks my main thread and reduces the performance to measly 15 fps... -> #96

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.