Stochastic Computation Graphs

[emilemathieu]

Hi all !

First thanks for your awesome package ! Otherwise, do you plan handling gradients in stochastic computation graph, i.e. graph with conditional probability distributions such as

using Distributions
w = ones(5); x = rand(5);
p = 1 / (1 + exp(-vecdot(w, x)))
y = rand(Bernoulli(p, 1))
loss = (y == 1)

In Schulman, J., Heess, N., Weber, T., & Abbeel, Gradient Estimation Using Stochastic Computation Graphs., it is described how to convert the stochastic computation graph into a deterministic computation graph, to which the backpropagation algorithm can be applied to a surrogate loss function which results in an unbiased gradient estimator for our stochastic loss.

Would you know how (and how much work is required) this could be implemented with(in) your package ?

Best, Emile

[CarloLucibello]

sorry for the very very late answer. I guess that the answer is that you will find the building blocks for creating the stochastic nodes described in the paper in these examples: 1) in the variational auto-encoder example, for the reparametrization trick (called pathways derivative in the paper) https://github.com/denizyuret/Knet.jl/blob/master/examples/variational-autoencoder/vae_mnist.jl

2) for the score function / reinforce estimator, have a look at the policy gradient actor critic implemenation in https://github.com/denizyuret/Knet.jl/blob/master/examples/reinforcement-learning/pg/actor_critic_discrete.jl. You can find on my github a repo with a different implementation