Vote is much better than mediator

[panyiming]

In my experiments, i found that the effect of vote mode is much better than mediator. Are there some explanations for it or some tips for improving the effect of mediator.

[panyiming]

mediator mode

------------- Analysis -------------- num_images: 132772 num_class: 16984 num_per_class: 7.8 Discard ratio: 0.7345

------------- Evaluation -------------- Pair accuracy: 0.9999 (singular removed) prec / recall / fscore: 99.96, 44.87, 61.94 (singular kept) prec / recall / fscore: 99.96, 7.468, 13.9 Writing to:/ Runing time: knn: 0.2538 s, cdp: 36.84 s, total: 37.09 s

vote mode

------------- Analysis -------------- num_images: 493801 num_class: 26337 num_per_class: 19 Discard ratio: 0.01241

------------- Evaluation -------------- Pair accuracy: 0.9988 (singular removed) prec / recall / fscore: 96.81, 94.68, 95.73 (singular kept) prec / recall / fscore: 96.81, 92.27, 94.49 Writing to:/ Runing time: knn: 0.08636 s, cdp: 15.53 s, total: 15.61 s

@XiaohangZhan

[XiaohangZhan]

1. According to the results, in mediator mode, most of examples are discarded (0.7345) and the recall is very low. Pls decrease the "threshold" in "mediator" parameters to achieve higher recall. A balanced recall and precision will result in better fscore.
2. What's your training data? Is it still "labeled/emore_l200k"? If you use your own data for testing, you need to prepare a training set by yourself. The features of training set should be extracted in a same way as your testing set.

[panyiming]

Actually i have decreased the threshold and the train dataset was collected by ourselves, but have not got the expected results and the improvement was not obvious. There is a kind of possibility that the mediator model is overfitting.

[XiaohangZhan]

Pls check:

1. The similar metric of features should be Cosine, i.e., the features are extracted from a classification model trained with standard softmax loss or arcloss.
2. Try not use "structure" as input, i.e, set input: ['relationship', 'affinity']
3. There should be enough training data to avoid overfitting.

[panyiming]

I found that the ratio of positive pairs and negative pairs for training mediator was imbalanced, which was near to 4.25 and the train data included 3 million images. Have you accessed what effect can this imbalance will have on the mediator model.

[XiaohangZhan]

Yes it is imbalanced. It does not matter in my experiments.