Open ityer82 opened 6 years ago
thuster
commented
5 years ago I'm seeing the same bug... took me a while to track down what was happening. This is important in, e.g., adversarial image generation. As a simple workaround, I added these lines to dagnn.BatchNorm after the vl_nnbnorm call:
dydx = reshape(params{1}./params{3}(:,2),1,1,[],1); derInputs{1} = derOutputs{1}.*dydx;
ityer82
commented
5 years ago Hi Thanks for taking the time to respond. Can you specify what params represent? And why do you think this is the needed workaround. I’m not working with dag, but rather wrote the main pipe (e.g the backprop), and using only the function. So I could see how to work it around? Thanks, Itay
ב-17 ביוני 2019, בשעה 17:42, thuster notifications@github.com כתב/ה:
I'm seeing the same bug... took me a while to track down what was happening. This is important in, e.g., adversarial image generation. As a simple workaround, I added these lines to dagnn.BatchNorm after the vl_nnbnorm call:
dydx = reshape(params{1}./params{3}(:,2),1,1,[],1); derInputs{1} = derOutputs{1}.*dydx;
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.
Hi, I check the vl_nnbnorm derivative (using small changes in the input) and got correct derivative of the input (=der_data) only if the moments weren't inserted using 'moments' option. Looking at the code nnbnorm.cu seems that that 'batch_normalize_backward' is called at both BatchNormBackward and BatchNormForwardWithMoment structs, which cause the problem, since the close-form of the der_data output is true only if moments are a function of the input. When they are fixed (e.g using 'moments' option) the der_data expresssion should be different (since dloss/data doesn't depend of dloss/dmoments) Am I right?