Closed bl0 closed 5 years ago
Also, I can't understand the role of post_ops
. I know that only the ops in post_ops
can be evaluated, such that BN is updated only for some special cases, in other cases BN will be frozen, but for what cases?
Thanks very much.
Thanks for your reply.
If the BN is updated for all batches, what will happen?
I am not sure, typically in fully supervised learning one only runs one batch and thus updates batch norm only once.
Thanks very much.
@bl0 did you try update BN for all batches, what do you observe ?
The results are bad. So maybe we should be careful about the BN.
Hi, I didn't know this trick. As @bl0, I also found that this interleaving avoids performance drop quite well. Are there any references for this?
I'm not sure about references. There are many ways to train with batch norm (for example, one could make a giant batch of everything), I simply chose that solution (interleaving) because I was considering doing multi-GPU and I wanted a homogeneous batch.
Hi authors, In mixmatch.py, you forward three batches(one labeled and two unlabeled) separately and use interleave to get a batch which can represent the whole data, why we cannot forward them together? I tried to forward them together but got a bad performance.
I don't know. It could be the way you made change introduced a bug, it could be that there it introduces a different behavior but I don't remember if it is the case: this research was done a year ago. Please update this thread if you find the reason so others can benefit.
Hi authors, thanks very much for the amazing work. I wonder what's the purpose of
interleave
function in the mixmatch.py. I tried to remove this, but the performance is very bad. I can't find any description about this in the github repo or papers.https://github.com/google-research/mixmatch/blob/9096b685ceae4bbdae8d342619ad56a8cb992d1c/mixmatch.py#L71-L78