Shape mismatch in adv_loss?

[puzzler10]

Hi there,

Firstly, thank you for the work and the code - it is very interesting to go through and to look at.

I was looking at the calculation of the adversarial loss under the condition "cw" (which I assume stands for Carlini and Wagner and is margin loss). I am guessing this refers to the following loss function from your paper:

The line of code looks like this: adv_loss = (pred[:, label] - pred.gather(1, indices) + args.kappa).clamp(min=0).mean().

I ran the code with a batch size of 10 for the Gumbel-Softmax samples. Here are the shapes of the terms:

• the first term pred[:, label] has shape torch.Size([10]).
• the second term pred.gather(1, indices) has shape torch.Size([10, 1]).
• when you minus the second term from the first, the resulting shape is torch.Size([10, 10]).

This doesn't seem right (I'd expect a vector of length 10, not a matrix of 10x10), but perhaps I am misunderstanding something.

[cg563]

Sorry for taking so long to respond!

What you found is indeed a bug in the code. The pred.gather(1, indices) term should be pred.gather(1, indices).squeeze(). I have pushed the bug fix.

The effect of the bug is that the CW loss is not being faithfully optimized as it was previously also encouraging cross-sample margins to be large. I expect all the results in the paper to improve slightly after this bug fix. Thank you for catching this bug!