search

tensorflow / tensorflow

An Open Source Machine Learning Framework for Everyone
https://tensorflow.org
Apache License 2.0
185.4k stars 74.17k forks source link

Allow reusing existing tensors for outputs of TF_SessionRun() #46711

Open shoelzer opened 3 years ago

shoelzer commented 3 years ago

In the C API, TF_SessionRun() allocates new output tensors on every call. This leads to frequent allocation/deallocation and memory copies in some situations. It would be useful to be able to reuse existing tensors for output.

The current API takes TF_Tensor** output_values and will update that variable to point to newly allocated output tensors. I propose that if output_values points to existing tensors, those tensors should be used for output instead of allocating new tensors.

DamRsn commented 2 years ago

Hello,

Any news on this?

I also believe that the overhead of memory allocation should be avoided when calling TF_SesssionRun() multiple times. I'm using the Tensorflow C API (2.6) for inference in the context of real-time audio processing, and malloc is not a real-time safe operation. Having a way to avoid this memory allocation is crucial for I think numerous use cases.

This issue has been pointed more than 2 years ago for Tensorflow 1.13 (https://github.com/tensorflow/tensorflow/issues/29733).

The solution suggested by @shoelzer seems good to me. There might also be other ways to implement it.

hoyden commented 2 years ago

Same problem in C++ API, did anyone solve it ?