search

naturomics / DLF

Code for reproducing results in "Generative Model with Dynamic Linear Flow"
https://arxiv.org/abs/1905.03239
70 stars 13 forks source link

Partitions K #4

Open xc-7984 opened 5 years ago

xc-7984 commented 5 years ago

In figure2 of your paper,you show K=2 is the better choice of K,so is there any different of your model with Glow when K=2? And when k=4 or 6,what's the result of inverse dynamic linear transformation.

naturomics commented 5 years ago

Even when K=2, our dynamic linear transformation is different from affine coupling layer, discussed in Section 3.1.

We found K=4, 6 for inverse dynamic linear transformation is also worse than K=2 of inverse dynamic linear transformation, so we didn't discussed it in our paper due to space constraints.

Conform it by following test if you're interested: python main.py --results_dir results/cifar10_noCond_4parts --num_parts 4 --width 308 --decomposition 1 python main.py --results_dir results/cifar10_noCond_6parts --num_parts 6 --width 256 --decomposition 1

xc-7984 commented 5 years ago

So the best K is 2?When k=2,Glow is h(x1)=x1,while yours is h(x1) = s1*x1+u1.Only changing this can make the results better than Glow on the Imagenet dataset?I amd confused about that.

naturomics commented 5 years ago

Yeah, it turns out our best results are obtained by changing y1 = x1 in affine coupling layer to y1 = s1*x1 + u1 (Actnorm layer likes). This is reasonable. In affine coupling layer, there always a half remains unchanged, it could be a bias.

xc-7984 commented 5 years ago

So if i replace the dynamic linear transform with a affine coupling layer and a actnorm layer,the result should be better.Glow consists of a affine coupling layer and a actnorm layer each step.I still don't understand why your model better than Glow on the Imagenet dataset.

lukemelas commented 5 years ago

Hello, I just wanted to follow up on this.

I feel as if I'm missing something important here. When K=2, is your model exactly the same as Glow, except for the fact that in the affine coupling layer, you have h(x_1) = s_1*x_1+u_1 instead of h(x_1)=x_1 in Glow?

naturomics commented 5 years ago

@lukemelas The changes in our best case (K=2) compared to Glow can be concluded as three points:

  1. in the affine coupling layer, we choose h(x_1) = s_1*x_1+u_1 instead of h(x_1)=x_1. We found any simple invertible h() can improve the model very significantly, there are more choices such as invertible 1x1 conv, invertible activation function (we didn't discuss these choices because we had not yet tested them at the time publishing our DLF paper, we will discuss it together with other important contributions in our next publication).
  2. we removed actnorm layer between the 1x1 conv layer and the coupling layer, as it is the special case of dynamic linear transformation without data-parameterization.
  3. we changed the NN structure slightly for training stability (and some optimization details such as learning rate).

I think our other novel contributions are also important:

  1. conceptually we connected the affine coupling layer and the AR/IAR transformations, as they are the extreme forms of dynamic linear transformation.
  2. conditional DLF allows us to control the mapping between the latent and the observation space. I can see some excellent applications utilizing this property.
lukemelas commented 5 years ago

Thanks for the quick and thorough response!

yuffon commented 5 years ago

Your response also helps me.