Supporting dynamic shape of tensors

[Hakuyume]

I want a feature that users can specify the shape of tensors after the model is built. Currently, the shape of tensors are determined when the model is built (more accurately, when the variable table profile is built). However, this is inconvenient in some cases.

• When processing a stream of data, the batchsize is not constant. Users need to wait enough data or add dummy data for padding.
• Some networks do not resize images into a constant size. For example, Faster R-CNN does not.

In order to allocate memory, I think we can assume the maximum shape of tensors are specified when building a model.

[msakai]

Just FYI: Glow adopted fixed shape of tensors.

Glow: Graph Lowering Compiler Techniques for Neural Networks

We have considered the idea of developing some kind of parametric tensor types to support features such as varying batch sizes. However, we have decided to implement a simple strict type system instead and let the high-level machine learning framework specialize the computation before constructing the Glow graph. We evaluated the mechanisms that the modern programming languages use to implement generics and concluded that most hardware accelerators do not support some of these mechanisms. Production systems that use Glow may generate multiple Glow graphs for different batch sizes, or recompute the graph just-in-time.