Closed PatrickHuembeli closed 5 years ago
You needed them in 2006 (first deep learning paper) because RBMs and DBNs did not suffer from the vanishing gradient problem. Today we have ReLUs and ResNets/DenseNets, Adam, and what-not, so backpropgation largely overcame this problem. Training is expensive for RBMs and DBNs, plus it can give funny results with CD/PCD, so they took a backseat for now. Until quantum-enhanced sampling revives these architectures, that is.
According to the question today: Why is pretraining with RBM necessary and why don't we have to do that anymore today? I have found the following paper:
http://www.jmlr.org/papers/volume11/erhan10a/erhan10a.pdf
Let's use this 'Issue' to figure this out and discuss a bit.