tensorflow / probability

Probabilistic reasoning and statistical analysis in TensorFlow
https://www.tensorflow.org/probability/
Apache License 2.0
4.26k stars 1.1k forks source link

Should Flipout use Rademacher or Uniform Distribution? #1554

Closed a76yyyy closed 2 years ago

a76yyyy commented 2 years ago

In this paper, The original content is "r and s are random vectors whose entries are sampled uniformly from ±1"

I think this should refer to uniform distribution rather than Rademacher distribution.

Is it my understanding errors, or which article explains the effectiveness of the Rademacher Distribution?

a76yyyy commented 2 years ago

Who can answer my questions?

SiegeLordEx commented 2 years ago

The flipout paper uses the rademacher distribution, that is core to the derivation of the algorithm. E.g. in Observation 1:

Let E be a random sign matrix that is independent of ∆dW. Then ∆W = ∆dW ◦ E is identically distributed to ∆dW.

This would not work with a uniform distribution.