wellecks
commented
7 years ago See section 4.2 second paragraph, and equation 11 of the Inverse Autoregressive Flows paper (on my phone otherwise I'd post it) - it looks like the mu and sigma functions take z0 as the inputs
I can train IAF and it gets c. 120 loss. The images are a little blurry but are definitely numbers.
I went back to your method of using an autoencoder for each set of flow parameters.
The structure is like this. Suppose we are using one step of IAF, so we need to generate a single mu and a single sigma that are autoregressive.
We have a neural net F operating on x to produce an intermediate hidden representation h (this is not z, NB):
h = F(x)
Then we apply another fully connected layer G to get an initial mu_0. (Sigma is the same, so I'll leave it out.)
mu_0 = G(h) = G(F(x)) = mu_0(x)
Now MADE kicks in. We use the single-layer autoencoder of MADE to transform mu_0 --> Enc(mu_0) --> mu = Dec(Enc(mu_0)). This last value is the actual flow parameter mu.
Since we used MADE to transform mu_0 to mu, we will be guaranteed that mu itself is autoregressive. Do the same for sigma.
If we need K steps of IAF, then we transform the same mu_0 using K single-layer MADEs to mu_1, ..., mu_K. Note that these mu are all decoupled from one another, and depend on each other only through their shared initial representation mu_0(x).