hotg-ai / rune

Rune provides containers to encapsulate and deploy edgeML pipelines and applications
Apache License 2.0
134 stars 15 forks source link

Variable sized tensor inputs/outputs #299

Open Michael-F-Bryan opened 3 years ago

Michael-F-Bryan commented 3 years ago

Some models (e.g. YAMNet) support variable sized inputs. TensorFlow represents this by having a 1x1 input that gets resized manually at runtime to fit the desired input.

What this will probably require:

Michael-F-Bryan commented 3 years ago

@Mohit0928 is currently stuck on this when implementing YAMNet.

Michael-F-Bryan commented 3 years ago

Inside the Rune, variable length tensors would be like any other tensor. If a proc block requires a certain size (e.g. because they are doing FFT) then they will enforce that by using runtime assertions.

We might run into some issues with models because we insert checks to make sure the tensor being passed as a model's input/output has the right element type and shape. As far as I am aware, TensorFlow Lite models don't have any builtin mechanism to say "you can use whatever size you like for my input/output", so what happens in practice is the model will pick a random shape (e.g. 1x1) and the user writes code that will manually resize tensors to whatever they want.

When loading a model we'll need to have a way to tell the loader that the model's inputs/outputs are variable sized (e.g. using the u8[_, 256, 256, 3] shape from above) and then update librunecoral to include the tensor resizing code (@saidinesh5 do you know how easy/hard that would be?).

Mohit0928 commented 2 years ago

I'm attaching some pics of how a TFlite interpreter takes input

Screenshot 2021-11-10 at 12 16 43 PM

Screenshot 2021-11-10 at 12 17 09 PM

Here is the link to the Slack message.

You can also find some more details about index used in the interpreter (see the TfLiteTensor* tensors in TfLiteContext)