search

clovaai / voxceleb_trainer

In defence of metric learning for speaker recognition
MIT License
1.03k stars 272 forks source link

my score file has ONLY NEGATIVE score.... #49

Closed zeek-han closed 4 years ago

zeek-han commented 4 years ago

Hello, I think this github page is very closely related to VOXSRC 2020.... In VoxSRC 2020 development toolkit, it requires score file, and on the development toolkit website, " score; must be a floating point number between the closed interval [0, 1], where 1 means the pair of segments correspond to the same speaker and 0 means the pair of segments correspond to different speakers."

but in the code, when I evaluate my model, the written score file has ALL NEGATIVE score, not between [0, 1]. my score file starts with -0.6494 id10270/x6uYqmx31kE/00001.wav id10270/8jEAjG6SegY/00008.wav -1.3724 id10270/x6uYqmx31kE/00001.wav id10300/ize_eiCFEg0/00003.wav -0.7400 id10270/x6uYqmx31kE/00001.wav id10270/GWXujl-xAVM/00017.wav -1.2865 id10270/x6uYqmx31kE/00001.wav id10273/0OCW1HUxZyg/00001.wav -0.6512 id10270/x6uYqmx31kE/00001.wav id10270/8jEAjG6SegY/00022.wav -1.1112 id10270/x6uYqmx31kE/00001.wav id10284/Uzxv7Axh3Z8/00001.wav -0.6810 id10270/x6uYqmx31kE/00001.wav id10270/GWXujl-xAVM/00033.wav -1.0992 id10270/x6uYqmx31kE/00001.wav id10284/7yx9A0yzLYk/00029.wav -0.7889 id10270/x6uYqmx31kE/00002.wav id10270/5r0dWxy17C8/00026.wav -1.1319 id10270/x6uYqmx31kE/00002.wav id10285/m-uILToQ9ss/00009.wav

Is there any problem??? Is everything OK? with this code to participate in VOXSRC 2020? Thank you.

009deep commented 4 years ago

Please check issue-15 . Also for submission, you can convert all the negative values to 0 if that is mandatory or causes any issue. Regarding all negative values, that shouldn't be case. For same user value should be high positives. I can confirm there is no issue with code in that regard.

zeek-han commented 4 years ago

Thank you for your answer... I think the issue 15 which you mentioned above is different with mine... actually, the score file I wrote is derived from the source code which was updated on 1st of August, while the issue 15 was in May.

I guess that the reason of negative values in the score file is the multiplication of -1 in SpeakerNet.py In SpeakerNet.py, there is the function named "evaluateFromListSave()", and in the function, there is calculating module for distance and score. At that time, score is multiplied by -1. Below is the part of the source code.

        dist = F.pairwise_distance(ref_feat.unsqueeze(-1).expand(-1,-1,num_eval), com_feat.unsqueeze(-1).expand(-1,-1,num_eval).transpose(0,2)).detach().cpu().numpy();
        score = -1 * numpy.mean(dist);
joonson commented 4 years ago

The scores do not need to be between 0 and 1 for the challenge. This will be corrected.

zeek-han commented 4 years ago

Thank you very much!!!!

009deep commented 4 years ago

@zeek-han fyi, that code change should not impact values being negative. In fact, you can replace pairwise_distance with cosine_similarity and remove score = -1 * numpy.mean(dist) and you'd still get negative values for unmatched users.