Add operator to support dynamic neural networks

[EnfangCui]

Dynamic neural network is an emerging technology in deep learning. Compared to static models which have fixed computational graphs and parameters at the inference stage, dynamic networks can adapt their structures or parameters to different inputs, leading to notable advantages in terms of accuracy, computational efficiency, adaptiveness, etc.

In general, dynamic neural network has the following advantages: 1) Adaptiveness. Dynamic models are able to achieve a desired trade-off between accuracy and efficiency for dealing with varying computational budgets on the fly. Therefore, they are more adaptable to different hardware platforms and changing environments, compared to static models with a fixed computational cost.

2) Representation power. Due to the data-dependent network architecture/parameters, dynamic networks have significantly enlarged parameter space and improved representation power.

3) Compatibility. Dynamic networks are compatible with most advanced techniques in deep learning, such as model compression, neural architecture search (NAS) .

Demo: 1) Model early-exiting with BranchyNet. A real industrial case is here. 2) Model layer skipping with ResNet.

[ugvddm]

Hi CEF， thanks for your contribution, it is seem like a good idea, but we need find some scenaro to evaluate it more better.

[EnfangCui]

Hi CEF， thanks for your contribution, it is seem like a good idea, but we need find some scenaro to evaluate it more better.

Yes, we need to find some real-world application scenarios. I will continue to follow up on the progress of this technology, and when I have a valuable scenario, I will make a detailed proposal.

[ugvddm]

we look for it together :)