Closed puzzler10 closed 1 year ago
Sorry for taking so long to respond!
What you found is indeed a bug in the code. The pred.gather(1, indices)
term should be pred.gather(1, indices).squeeze()
. I have pushed the bug fix.
The effect of the bug is that the CW loss is not being faithfully optimized as it was previously also encouraging cross-sample margins to be large. I expect all the results in the paper to improve slightly after this bug fix. Thank you for catching this bug!
Hi there,
Firstly, thank you for the work and the code - it is very interesting to go through and to look at.
I was looking at the calculation of the adversarial loss under the condition "cw" (which I assume stands for Carlini and Wagner and is margin loss). I am guessing this refers to the following loss function from your paper:
The line of code looks like this:
adv_loss = (pred[:, label] - pred.gather(1, indices) + args.kappa).clamp(min=0).mean()
.I ran the code with a batch size of 10 for the Gumbel-Softmax samples. Here are the shapes of the terms:
pred[:, label]
has shapetorch.Size([10])
.pred.gather(1, indices)
has shapetorch.Size([10, 1])
.torch.Size([10, 10])
.This doesn't seem right (I'd expect a vector of length 10, not a matrix of 10x10), but perhaps I am misunderstanding something.