What is the necessity of using mse loss instead of cross entropy?

[CoinCheung]

Hi, Thanks for the excellent paper and the generous share of this code base. I am trying to learn some details from this repo. Please allow me to ask a few questions:

Would you please explain why did you use mse loss rather than the normal cross entropy (or kl diversity loss) to train on the unlabeled data ? What is the philosophy behind this choice please?

By the way, I noticed other issues discussing about interleave the labeled and unlabeled batches. Do you think whether it would work if we first concatenate the labeled and unlabeled batch to one single big batch, and than run forward with this big batch. In this way, we could compute the labeled loss with the first batch size number of samples and compute the unlabeled loss with the rest samples? What I mean is roughly like this:

inten = concat([x, u1, u2])
logits = model(inten)
loss_x = cross_entropy(logits[:batchsize], label_x)
loss_u = mse(logits[batchsize:], label_u)
loss = loss_x + lam * loss_u
...

Would you please tell me why you used the interleave method rather than the above method?

[david-berthelot]

1. We use the Brier score (MSE loss) to make the training more robust to incorrect guesses. This is detailed in the paper on arxiv.
2. You could use a big batch. Just be mindful that with training=True the batch size must always be the same for batch_norm to behave properly. In particular, we run the model in guess_labels (on u1, u2) and we run it in the main body again (on x', u1', u2'). That's why we ended up running each group (x', u1, u2, u1', u2') separately since combining them would result in two mismatched batches.

[CoinCheung]

In the settings of this code base, there are 5 forward computations per iteration: 2 for label guessing on u1 and u2, and 3 on the mixed input. All of these 5 forward computations would affect the running mean/var status of bn layers, though computation on u1 and u2 are not meant for training. Would it be better if we find a way to remove the impact on bn status brought by u1 and u2?

[david-berthelot]

Actually only one forward computation affects the batch norm running mean and var. We explicitly make sure of that in the code by shuffling the batches contents (layers.interleave) and by ignoring the global updates (see post_ops) other than for that one batch.

[CoinCheung]

Thanks, I will try it.

[HeimingX]

Just be mindful that with training=True the batch size must always be the same for batch_norm to behave properly.

Hi, thank you for your in-depth discussion. With regard to the statement above, I wonder is there any literature shows that it is best to ensure that the batch sizes are consistent for infering through the model during one batch iteration?

[david-berthelot]

I am not sure about literature on the topic. It's just that with different batches sizes, I would expect the variance to change. So basically I erred on the safe side in the code. But feel free to experiment may it can be simplified.

[HeimingX]

Thanks for reply.

[HeimingX]

Hi, I have one more question about the CIFAR100 experiment. In your paper, a 28-layer Wide Resnet model (which has 135 filters per layer, resulting in 26 million parameters) is used for CIFAR100. According to the tutorial in readme file, I just set the parameter filter equals to 135 and the code shows that total number of parameter is 25,985,290. I also notice that the filter number of the first conv layer(filter=16) is fixed in your ResNet class . so I wonder if this setup is the same as yours for getting the results in your paper. Thanks.

[david-berthelot]

Yes it is the same setup. In reality we had a little bit less parameters than the methods we compared to, so yes the number is correct it's almost 26M.

[HeimingX]

Thanks

[CoinCheung]

Hi, Thanks for these replies, I am not familiar with tensorflow, so I feel maybe I better ask a bit more about the details :)

Did you use a batch of 64 interleaved samples to train the model with the other 128 samples neglected in computing the gradients of all parameters (or only use 64 samples to update bn, and use all the 192 samples to update other parameters)? If only 64 samples are used on all parameters, since I want to try a big batch of 192 samples per forward computation, should I multiply the learning rate by 3? As for the number of epoches and iterations, you trained with batch size of 64 for 1024 epoches and with each epoch consists of 1024 iterations, so if I train with 192 samples per iteration, I should reduce the number of iterations per epoch to 1024/3, am I correct ?

[david-berthelot]

All batches (192 samples) are used in gradient computation.

[lichunlei111]

Hello, I am very inspired to read your thesis, but when I run the code, I find that the code is missing files. Can you send me the complete code? Appreciate it!

[david-berthelot]

I'm closing this issue since I answered the original question. For unrelated questions, please open new issues.