This is a repo for reproducing the results presented at NoRefER paper.
Model checkpoints are available in the link below:
https://drive.google.com/file/d/1KgMiU_9asfDEKLTkc8sqagG1fIk84nV_/view?usp=sharing
test datasets are uploaded as csv files. These files are used for reported results in the paper.
The code for the results of NoRefER-Self and NoeRefER-Semi is provided in main_noref.py. To run the code, please especify the model and the filename in your command.
python main_noref.py --filename filename.csv --modelname model_checkpoint.ckpt
example:
python main_noref.py --filename en-common.csv --modelname self_super.ckpt
The code for the baseline results and calculating perplexity is also provided in perplexity_noref.py. You just need to specify the filename to run this code.
example:
python perplexity_noref.py --filename en-common.csv
More details are available in the following papers. Welcome to cite our work if you find it is helpful to your research.
[Paper]
@inproceedings{yuksel23_icassp,
author = {Kamer Ali Yuksel and Thiago Castro Ferreira and Ahmet Gunduz and Mohamed Al-Badrashiny and Golara Javadi},
title = {A Reference-Less Quality Metric for Automatic Speech Recognition via Contrastive-Learning of a Multi-Language Model with Self-Supervision},
booktitle = {IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2023, Rhodes Island, Greece, June 4-10, 2023},
pages = {1--5},
publisher = {IEEE},
year = {2023},
url = {https://doi.org/10.1109/ICASSPW59220.2023.10193003},
doi = {10.1109/ICASSPW59220.2023.10193003}
}[Paper]
@inproceedings{yuksel23_interspeech,
author={Kamer Ali Yuksel and Thiago Castro Ferreira and Golara Javadi and Mohamed Al-Badrashiny and Ahmet Gunduz},
title={NoRefER: a Referenceless Quality Metric for Automatic Speech Recognition via Semi-Supervised Language Model Fine-Tuning with Contrastive Learning},
year=2023,
booktitle={Proc. INTERSPEECH 2023},
pages={466--470},
doi={10.21437/Interspeech.2023-643}
}[Paper]
@inproceedings{javadi2024wordlevel,
title={Word-Level ASR Quality Estimation for Efficient Corpus Sampling and Post-Editing through Analyzing Attentions of a Reference-Free Metric},
author={Golara Javadi and Kamer Ali Yuksel and Yunsu Kim and Thiago Castro Ferreira and Mohamed Al-Badrashiny},
eprint={2401.11268},
archivePrefix={arXiv},
booktitle = {IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2024, Seoul, Korea, April 14-19, 2024},
publisher = {IEEE},
year = {2024},
url = {https://arxiv.org/abs/2401.11268},
doi = {10.48550/arXiv.2401.11268}
}