search

peterwittek / qml-rg

Quantum Machine Learning Reading Group @ ICFO
GNU General Public License v3.0
131 stars 63 forks source link

Pretrain networks with RBM #62

Closed PatrickHuembeli closed 5 years ago

PatrickHuembeli commented 6 years ago

According to the question today: Why is pretraining with RBM necessary and why don't we have to do that anymore today? I have found the following paper:

http://www.jmlr.org/papers/volume11/erhan10a/erhan10a.pdf

Let's use this 'Issue' to figure this out and discuss a bit.

peterwittek commented 6 years ago

You needed them in 2006 (first deep learning paper) because RBMs and DBNs did not suffer from the vanishing gradient problem. Today we have ReLUs and ResNets/DenseNets, Adam, and what-not, so backpropgation largely overcame this problem. Training is expensive for RBMs and DBNs, plus it can give funny results with CD/PCD, so they took a backseat for now. Until quantum-enhanced sampling revives these architectures, that is.