jolibrain / deepdetect

Deep Learning API and Server in C++14 support for Caffe, PyTorch,TensorRT, Dlib, NCNN, Tensorflow, XGBoost and TSNE
https://www.deepdetect.com/
Other
2.52k stars 561 forks source link

Support for Spatial Transformer Networks #40

Open beniz opened 8 years ago

beniz commented 8 years ago

Spatial Transformers act as a warping / attention mechanism useful for a variety of tasks.

There's a handful of repositories implementing STN for Caffe, see https://github.com/BVLC/caffe/issues/3114

However, STN have not made it into official Caffe yet, so it would need to be added to dd Caffe backend https://github.com/beniz/caffe/tree/master_dd_integ and rebased as Caffe recently changed the distribution of layers in C++ files.

Original paper: http://arxiv.org/abs/1506.02025

Implementations of interest:

Applications of interest:

beniz commented 8 years ago

This now appears to be readily available for Caffe, see https://github.com/matthieudelaro/caffeBVLCplus/tree/stn and https://github.com/BVLC/caffe/issues/3114#issuecomment-225789373

This is not part of official Caffe, so there's need for it to be merged in a branch of our Caffe backend, then test on a variety of architectures.

ayumiymk commented 7 years ago

Do you know some examples about STN used as attention mechanism?

beniz commented 7 years ago

This task has remained unattended, so no.

beniz commented 7 years ago

https://github.com/daerduoCarey/SpatialTransformerLayer appears to be an almost up to date version, the CPU code should be replaced with the more optimized version from the BVLCplus repository above.

Useful blog post: http://christopher5106.github.io/big/data/2016/04/18/spatial-transformer-layers-caffe-tensorflow.html