danielkelshaw
commented
1 year ago Hey, thanks for reaching out! It's been quite a while since I've looked at the paper but I'll do my best to clarify any points.
I had a quick question: in section 3.2 of the paper, the authors provide an optimization procedure. I understand steps 1−3 where they reparameterize the Diagonal Gaussian. Then in 4 they let f(ww,θ)=logq(ww,θ)−logP(ww)P(D|ww). I understand this to be a different form of (2) but for a single sample ww. Am I correct in thinking this?
As you said steps 1-3 are simply the re-parameterisation trick, allowing us to compute the gradient wrt. the parameters and alleviating the issues of differentiating wrt. a random sample. You're correct in saying that step 4 is the same as equation (2) but for a single sample -- they state in the paper: "Since each additive term in the approximate cost in (2) uses the same weight samples, the gradients of (2) are only affected by the parts of the posterior distribution characterised by the weight samples." I take this to mean that, since we are ultimately only interested in computing gradients of the ELBO, a single sample is sufficient.
Then in 5−6 they define the gradient with respect to the variational parameter μ as ∇μ=∂f(ww,θ)∂ww+∂f(ww,θ)μ and then also a similar expression for ρ. I see ∂f(ww,θ)∂ww as the gradient of the loss (2) with respect to the sampled weight ww.
My question is then, where do you do the calculations in equation (3) of the paper? It seems to me you just take the gradient of the loss function with respect to μ?
Let me first provide the code of the forward pass for convenience:
We see in L84 that the log posterior is a function of w_log_posterior, b_log_posterior; computed as per steps 1-3. In L51-L52 we instantiate instances of the GaussianVariational class which in turn registers parameters self.mu, self.rho; these are used to produce the weights and biases used for the linear layer. When we call .backward() in the training loop the gradients of the loss wrt. these parameters of the network will be computed via autograd. So while steps 5-6 in the algorithm are providing analytical formulations for the gradients $\nabla\mu, \nabla\rho$, these are being computed internally and we do not need to write this explicitly.
Looking through the code again I can see that this might be unclear. Please let me know if I've cleared things up - let me know if you need a better explanation. While I'm unlikely to update the repo myself at this time I would welcome a pull request if you think you can make it easier to understand in the code.
PaulScemama
Hey I really enjoyed going through your implementation of the paper, so thank you!
I had a quick question: in section 3.2 of the paper, the authors provide an optimization procedure. I understand steps $1-3$ where they reparameterize the Diagonal Gaussian. Then in $4$ they let $f(\pmb{w}, \theta) = \text{log} q(\pmb{w}, \theta) - \text{log} P(\pmb{w})P(\mathcal{D}|\pmb{w})$. I understand this to be a different form of $(2)$ but for a single sample $\pmb{w}$. Am I correct in thinking this?
Then in $5-6$ they define the gradient with respect to the variational parameter $\mu$ as $\nabla_{\mu} = \frac{\partial f(\pmb{w}, \theta)}{\partial \pmb{w}} + \frac{\partial f(\pmb{w}, \theta)}{\mu}$ and then also a similar expression for $\rho$. I see $\frac{\partial f(\pmb{w}, \theta)}{\partial \pmb{w}}$ as the gradient of the loss $(2)$ with respect to the sampled weight $\pmb{w}$.
My question is then, where do you do the calculations in equation $(3)$ of the paper? It seems to me you just take the gradient of the loss function with respect to $\mu$?
I'm sure I'm not understanding something, so thanks for any light you can shed on my problem! Thanks!