search

tdeboissiere / DeepLearningImplementations

Implementation of recent Deep Learning papers
MIT License
1.81k stars 650 forks source link

problem of function usage #34

Closed yoookoo closed 7 years ago

yoookoo commented 7 years ago

In DeepLearningImplementations/pix2pix/src/model/models.py, what is the function of minb_disc() ? It seems that no matter what is inputted, the output always be 1. So MBD layer is useless.

tdeboissiere commented 7 years ago

That's the mini batch normalization. I think I got this implementation from OpenAI. If you get only ones, it probably means your features are close to 0 (exp(0) == 1).

yoookoo commented 7 years ago

diffs = K.expand_dims(x, 3) - K.expand_dims(K.permute_dimensions(x, [1, 2, 0]), 0) the former is equal to the succedent, so diffs= 0 , no matter what is inputted. Maybe this is a bug.

yoookoo commented 7 years ago

Can you give me a link to OpenAI implementation? Thanks a lot.

tdeboissiere commented 7 years ago

https://github.com/openai/improved-gan/blob/master/mnist_svhn_cifar10/nn.py

tdeboissiere commented 7 years ago

I don't think diffs should always be 0 unles the transpose of x is equal to x somehow.

yoookoo commented 7 years ago

How can K.permute_dimensions(x, [1, 2, 0]) realize the function of transpose of x? Can you expand it in detail? Thank you very much. This confuses me a lot!

tdeboissiere commented 7 years ago

It just switches the axes. In 2D it is the transpose function you're used to, a nxm matrix turns into an mxn matrix. In >2D it is harder to visualize but in this case, an (a, b, c) matrix becomes a (b, c, a) matrix.

Then, broadcasting rules are applied in diffs (i.e. the matrices are repeated along the axes with dimension = 1 to allow the subtraction operation)

yoookoo commented 7 years ago

Genius you are! Is there any formula which can prove the differ between x and tranpose of x calculates mini-batch distance?

tdeboissiere commented 7 years ago

You should have a look at the paper (cf. OpenAI Improved techniques to train GAN). They define a way to measure distance within samples. There are of course other ways to do so.