search

adambielski / siamese-triplet

Siamese and triplet networks with online pair/triplet mining in PyTorch
BSD 3-Clause "New" or "Revised" License
3.1k stars 633 forks source link

Whether embedding is normalized ||x||=1 ? #51

Closed BerenLuthien closed 4 years ago

BerenLuthien commented 4 years ago

Thanks for sharing great work

Whether embedding is normalized ||x||=1 ? It looks EmbeddingNetL2 adopted normalization of embedding, but it is not used. From ReadMe it is EmbeddingNet that is adopted but it does not normalize embeddings.

adambielski commented 4 years ago

As you can see it's implemented but not used in the notebooks, the visualizations look better without it. It should be beneficial to use it though (the margin value must be adjusted then)

BerenLuthien commented 3 years ago

Hello (sorry for late response), 1, any recommendation of margin for EmbeddingNetL2 (and why) ? I remember the Facenet paper used margin=0.2

2, replacing with EmbeddingNetL2 gives Error : RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [128, 128]], which is output 0 of AddmmBackward, is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True). Did you ever try it and any luck of success ? I got a fix, would you be interested in a PR ? Thanks

adambielski commented 3 years ago
  1. This should be ideally cross-validated, 0.2 sounds like a reasonable starting point, depending on your embedding dimensionality.
  2. Not sure why it doesn't work anymore, but simply try to replace the inplace operators in EmbeddingNetL2 e.g. output = output / ... instead of output /= ...