omoindrot / tensorflow-triplet-loss

Implementation of triplet loss in TensorFlow
https://omoindrot.github.io/triplet-loss
MIT License
1.12k stars 283 forks source link

pairwise_dist drops to 0,loss is near the margin and can't go down #38

Open xiaomingdaren123 opened 5 years ago

xiaomingdaren123 commented 5 years ago

hi,omoindrot I have encountered some problems,after training for a while,pairwise_dist drops to 0,loss is near the margin and can't go down,visualize the training set and discover that they are all together,I don't know what caused it. Learning_rate is 0.0001,The network structure is vgg16 and the output dimension is 128,Data augmentation is used because of the small amount of data(random crop and horizontal flip),This will not happen if I don't use data augmentation.I hope you can reply,thanks!

omoindrot commented 5 years ago

It looks like training is collapsing, so you may want to decrease your learning rate maybe?

Or maybe have bigger batches to stabilize training? You can also monitor the average distance between embeddings to see how the collapse happens (suddenly or gradually).

xiaomingdaren123 commented 5 years ago

I don't decrease the learning rate(learning rate is 0.001),if I decrease learning rate,training process become slow,batch_size is set to 96。Can triple loss be used directly for classification task? or the data set must be pre-trained by softmax loss?

omoindrot commented 5 years ago

Maybe pre-training with a softmax loss could help.

cyrusvahidi commented 5 years ago

Maybe pre-training with a softmax loss could help.

Hi Olivier,

I considered this approach to try learning some supervised representation from the data, then refining it with triplet learning. I have not been able to stabilise training an embedding solely using triplet loss.

Could you elaborate on utility of the pre-training approach that you suggest?

omoindrot commented 5 years ago

The pretraining approach is just to get a good embedding with a softmax loss, since this loss is very stable and you should be able to converge.

Once you have this good enough representation, the triplet loss may help to separate further the class clusters and get you better performance.

cyrusvahidi commented 5 years ago

The pretraining approach is just to get a good embedding with a softmax loss, since this loss is very stable and you should be able to converge.

Once you have this good enough representation, the triplet loss may help to separate further the class clusters and get you better performance.

Is it also important to change the activation of the penultimate label vector layer to linear?

omoindrot commented 5 years ago

Yes so you have two steps:

  1. Train with softmax loss. You have the network computing the embedding, then a linear layer with softmax activation
  2. Remove the linear layer. Train with triplet loss using the embedding only
cyrusvahidi commented 5 years ago

Ok thanks. I more wanted to ask if the embedding's activation should be linear instead of ReLu, which I have seen mentioned before

omoindrot commented 5 years ago

It’s a good point, I think without relu makes more sense since you want the embedding to possibly have negative values.