Addition of noise to break symmetry

soumith / net2net.torch

Implementation of http://arxiv.org/abs/1511.05641 that lets one build a larger net starting from a smaller one.

160 stars 37 forks source link

Addition of noise to break symmetry #3

Open VinF opened 8 years ago

VinF commented 8 years ago

Hi, When you replicate the randomly chosen units, it seems that you don't add noise to break symmetry after the replication? If so, I think this may lead to not actually increasing the network capacity.

soumith commented 8 years ago

yes, i dont do that. kinda surprising that no one else noticed. However, because of precision limits, symmetry will slowly break..., after an epoch or so.