Validation on LFW

[YongtaoGe]

Hi, @Joker316701882. I use the CASIA-Webface as training set and the LFW dataset as validation set. However, during the training process, the accuracy on LFW dataset is always 50% and the selected threshold is always 0. To find the problem, I read the source code and find the euclidean distance is used to calculate the distance between the two embedded features. As I know, AM-Softmax is a angular-based face recognition algorithm. My problem is:

1. Is that ok just using the euclidean-based distance to evaluate?
2. Using the euclidean-based algorithm to evaluate is logically feasible, so what's problem for the fixed accuracy? Is there anything I forget to configure before training?

[GangqiangZhao]

def evaluate_with_no_cv(emb_array, actual_issame): thresholds = np.arange(-4, 4, 0.01) embeddings1 = emb_array[0::2] embeddings2 = emb_array[1::2]

nrof_thresholds = len(thresholds)
accuracys = np.zeros((nrof_thresholds))

diff = np.subtract(embeddings1, embeddings2)
dist = -np.sum(np.square(diff),1)

for threshold_idx, threshold in enumerate(thresholds):
    _, _, accuracys[threshold_idx] = facenet.calculate_accuracy(threshold, dist, actual_issame)

best_acc = np.max(accuracys)
best_thre = thresholds[np.argmax(accuracys)]
return best_acc,best_thre

[YongtaoGe]

Well, I have done a litte change to the code so that it can calulate the cos similarity between embeddings when evaluating.

My reference is https://github.com/wy1iu/sphereface/tree/master/test/code/evaluation.m.

[Joker316701882]

@geyongtao @GangqiangZhao Hey guys, sorry for late reply! I notice the problem when evaluating on LFW. It was caused by np.greater() function in function "calculate_accuracy" from facenet.py. Just modify it into np.less(). The reason I use np.greater() is that cosine distance was used to calculate similarity. But actually in the repo, euclidean distance was used, so np.less() is right. I already corrected it, it should works fine now.

@geyongtao

Is that ok just using the euclidean-based distance to evaluate?

Yes, of course. Euclidean distance and cosine distance are actually same when the length of vector are both 1.

Using the euclidean-based algorithm to evaluate is logically feasible, so what's problem for the fixed accuracy? Is there anything I forget to configure before training?

Sorry about this. It's my problem. The reason is on top of this reply, just simply change np.greater() into np.less() (which I already corrected it in this repo), because I used cos-distance to compare before, (now is euclidean distance).

Thanks for pointing this out : )

[weihehdc]

Hi, @Joker316701882. I use the CASIA-Webface as training set and the LFW dataset as validation set. However, during the training process, the accuracy on LFW dataset is always 100% and the selected threshold is ok. I read the source code print the 'acc' in facenet.calculate_accuracy find that the Acc gradually increases from 0 to 1. In train.evaluate_with_no_cv, you do those op： best_acc = np.max(accuracys) best_thre = thresholds[np.argmax(accuracys)] return best_acc,best_thre so the acc is always 100%. my problem is how to fix it and it is ok to get thres without cross-validation?

[Joker316701882]

@weihehdc

how to fix it?

There is no threshold can lead acc to 1 (theoretically upper-bound acc is 99.86%). There must be something wrong with your implementation (or mine).

without cross-validation?

It's also theoretically better to use cross-validation. But there is almost no performance difference on lfw (on other test set there might be, because lfw here is validation set).