search

BVLC / caffe

Caffe: a fast open framework for deep learning.
http://caffe.berkeleyvision.org/
Other
34.03k stars 18.7k forks source link

Binary Weights #5270

Closed Alexivia closed 7 years ago

Alexivia commented 7 years ago

Issue summary

Recent developments have shown good results of Binarized Neural Networks, which are a new kind of CNN that works with binary weights instead of the typical 32-bit floats (refer to this paper). To train these BNNs they use Theano. Is there any way I can make a Caffe layer type that works with 1-bit weights instead of 32-bit FP?

Your system configuration

Operating system: Fedora 21 Compiler: gcc (GCC) 4.9.2 20150212 CUDA version (if applicable): V7.5.17 BLAS: blas.x86_64 3.5.0-10.fc21 Python: python.x86_64 2.7.8-15.fc21

shelhamer commented 7 years ago

Regular Caffe does not handle lower or variable precision, but check out Ristretto!

Alexivia commented 7 years ago

Thanks @shelhamer In the mean time I already moved towards to TensorFlow and I managed to emulate low-precision training.

shelhamer commented 7 years ago

Happy to hear you found a way forward for your project—if you ever compare to Ristretto it might be interesting to hear about any differences (but on the mailing list, and not the issue tracker). Good luck with your research 🔬

ddfabbro commented 6 years ago

What is the idea behind binarized weights? At first I though I would speed up the forward time, so I checked Ristretto.

But according to the owner:

"Ristretto's purpose is to quantize existing CNNs. Ristretto simulates a hardware accelerator with reduced word width. Ristretto is (slightly) slower than Caffe since the simulation of reduced word width adds overhead."

Reading their paper I kind of get the idea of simulation of reduced word width for running Caffe on low power embedded devices.

With this in mind is it actually possible to speed up the CPU forward time?