deqiangc
commented
2 years ago @petewarden @advaitjain
To make sure that we are on the same page. The request is: the memory address for input buffer could be changed externally by the application before each Invoke. The typical use case is double buffering of input data so that the first invoke use buffer 0, the second invoke use buffer 1, and so on.
We don't have a straightforward easy solution for now. It seems to require at least two new api exposed: one for application to tell which tensor that the TFLM shall not allocate memory, but still needs to track for the OPs; and the other for application to set buffer address for tensor. In addition to those api and complexity associated with them, the application also need to know which tensors are input tensor (aka, they would need to know the detail inside tflite file), which in my view could make things fragile ( a small change in model could lead to hard to debug issues).
I am wondering how much benefit that this feature can bring. Do you have a ballpark estimate in terms of memory saving (in percentage) and cycles (if applicable)?
advaitjain
mansnils
Alex-EEE
github-actions[bot]
Supply input data and consume output data located outside the main arena. Instead, the input/output data would remain in an arbitrary memory buffer, i.e. another memory arena. The idea is to avoid copying input/output data to/from the main arena, thus saving cycles and potentially memory. The new memory planner, NonPersistentMemoryPlannerShim partly solves this as input data can be placed in an external memory arena. However in the audio/video streaming case the input could not be changed since the memory addresses based on the offsets in the memory plan would need to be recalculated during every update (before every invoke).