lucasb-eyer
commented
7 years ago Hi @dnaq, thanks for your feedback. We haven't tried ms-celeb-1m or casia-webface yet, but I have gained some more experience applying batch-hard to other datasets since (DukeMTMC, CelebA mainly), and had similar experience to what you describe, as those are significantly harder. The effect you observe is what we describe in the appendix F, the initial "difficult" phase. When the network is not able to leave that difficult phase, it means the task is too hard for the network. In order to fix that and make the network converge, you need to make it easier in the beginning. You can achieve this in several ways:
1) make the network more powerful. For example, on one dataset where the network often got stuck, changing from ResNet to a ResNeXt architecture made the training very stable and smooth. 2) make the task easier, by starting with smaller batches. In a PK-batch of, for example, 2 instances of 4 persons, there will unlikely be difficult examples, and thus it becomes semi-hard instead. You can then slowly grow the batch-size over time. On a difficult dataset, I was able to train a network reliably by growing the batch-size linearly from 24 to 432 over time. 3) If the above two don't work or are not possible to do for you, another possibility to help the network overcome this initial phase of collapsing embeddings, is to add a regularization loss on the distances between embeddings, such as 1/d or 1/d², -d or anything that has the effect of countering collapse to a single point. You can then disable or turn down this loss later once the network has gotten past the difficult phase. Note that normalizing the embeddings won't help here, as they can (and will) still collapse somewhere on the hypersphere.
Finally, I want to mention that I believe I have found another simple to implement trick to make it work reliably on hard/unclean datasets, but don't want to reveal it publicly yet as I might want to write a paper about it. Let me know if you are potentially interested in collaboration, or, if you are not into publications, we can discuss privately.
But in the end, use whatever works for your use-case. If center-loss works great for you, use it :smile:
dnaq
ergysr
filipetrocadoferreira
spott
I've been doing some experiments with your batch hard triplet loss function and different architectures/datasets. On MARS I manage to reproduce the results from your paper (network seems to converge), but with many other datasets I get stuck at a loss of ~0.6931 which is softplus(0). Looking at the embedding it seems like the network starts to yield the same embeddings for all different classes.
Worth to know is that a center loss formulation works quite well for generating usable embeddings for these datasets, I've tried with me-celeb-1m (after cleaning it up), and with casia-webface.
My interpretation of these results is that the batch hard triplet loss function is really sensitive to mislabeled datasets, and it might get stuck in a local minima if the dataset contains mislabeled images. I've tried some hyperparameter tuning (e. g. changing lr and optimizer), but I haven't managed to avoid the local minimum.
Have you seen similar results in your work when experimenting with different datasets?