search

cpetig / tflite_micro_compiler

generate tflite micro code which bypasses the interpreter (directly calls into kernels)
Apache License 2.0
77 stars 26 forks source link

Inplace operator planning #53

Open cpetig opened 4 years ago

cpetig commented 4 years ago

Dear Rafael,

I thought more about reshape operators (and their kind), what if ... if an operator's output has the same size as its first input and this input is only used by this single node (the one we investigate here), we assume that this operator can be done in-place and extend the lifetime of the tensor in the planning stage to the lifetime of the output tensor. I don't know whether this saves memory, but it will for sure make some operations more cache friendly (and enable reshape to be a no-op under the right conditions). What do you think?

(should we use a list of invalid operators or a list of valid operators, the default being the other case)

andrewstevens-infineon commented 4 years ago

Hi Christoph,

This was quite high on our list too. Especially translation from onnx import we get graphs quite heavily burdened with Reshape's. Implemented as copy's they're ridiculously wasteful of memory.

I'd vote for a valid list (nice and safe in the case of upstream changes etc). We should provide for the list being target-specific (I'm thinking here of limited accelerator HW where possibly in-place might not work for implementation reasons).