search

asvspoof-challenge / 2021

ASVspoof 2021 Baseline Systems
208 stars 75 forks source link

Evaluation #11

Open yunzqq opened 2 years ago

yunzqq commented 2 years ago

How to evaluate the ERR scores in the evaluation set?

TonyWangX commented 2 years ago

Hi,

Are you asking for the scripts to compute EER?

This repo has included scripts to compute the EER. For example, for LA, please check README in https://github.com/asvspoof-challenge/2021/tree/main/eval-package/LA/package-stage-1 Just follow the steps there to download the ground truth label, and compute EER using the scores your model produced.

Or are you asking how EER is computed? The function is here: https://github.com/asvspoof-challenge/2021/blob/80f8d4c8505a2b8ad94a76dbade72fc16217e689/eval-package/LA/package-stage-1/eval_metrics.py#L42 An general introduction to EER can be read from https://lands.let.ru.nl/literature/leeuwen.2007.12.pdf (before sec.2.3)

TonyWangX commented 2 years ago

Hi, new evaluation scripts with full set of meta-labels. Please check https://github.com/asvspoof-challenge/2021/tree/main/eval-package

Also available is a notebook showing the details: https://colab.research.google.com/github/asvspoof-challenge/2021/blob/main/eval-package/ASVspoof2021_eval_notebook.ipynb