pytorch code for Improved Training of Wasserstein GANs

[aosokin]

Hi, do you plan to provide a pytorch implementation of the recent paper on "Improved Training of Wasserstein GANs"? Is there an easy way to compute the gradient w.r.t. of the gradient norm?

[atgambardella]

I think on your input variable, you can just set requires_grad=True, and then take torch.norm(your_input.grad, 2)

[ccurro]

https://github.com/igul222/improved_wgan_training

edit: The early morning getting to me, you wanted pytorch 😛

[aosokin]

@atgambardella One needs the gradient of the gradient norm w.r.t. parameters, not the gradient w.r.t. the input

@ccurro This is a tensorflow code. I'm not aware of any pytorch implementation

[atgambardella]

@aosokin Section 4 of the paper, "Sampling along straight lines says":

The gradient term ||∇xˆD(xˆ)||2 is with respect to the points xˆ, not the parameters of D

[aosokin]

@atgambardella agreed, my bad. However this does not solve the problem. One needs to differentiate this norm, not simply compute it.

[soumith]

Hi Anton,

This cannot be implemented in pytorch until we merge the PR https://github.com/pytorch/pytorch/pull/1016 which enables gradients of gradients

[aosokin]

@soumith Thanks a lot for your reply! Looking forward to see that happen!

[NickShahML]

Is there a way to know when this merger has occurred?

[apaszke]

@NickShahML you can subscribe to the PR

[fonfonx]

Is it possible to implement it in Torch? How can we compute the gradient of the norm of the gradient of D?

[AjayTalati]

Might be possible to implement improved training in PyTorch now, the merge was done yesterday :+1:

[jfsantos]

This repository has an implementation of it using the new gradients of gradients feature: https://github.com/caogang/wgan-gp.

[rafaelvalle]

@jfsantos gradients of gradients? what do you mean by that?

[jfsantos]

@rafaelvalle For normalizing w.r.t. the norm of the gradients of the discriminator, you need to be able to backprop through that gradient, so you need to compute the gradient of a gradient. This is a recent feature in PyTorch's autograd.

[rafaelvalle]

@jfsantos yes! I got confused. w.r.t. the norm of the gradients of the discriminator on the interpolated points between Pr and Pg!

[LukasMosser]

@rafaelvalle @jfsantos but convolutions are not supported yet I believe.

[pclucas14]

Hi, does anyone know when convolutions will be supported ?

[jfsantos]

@pclucas14 Please follow this PR for info on double backward for convolutions: pytorch/pytorch#1643