Development plan for VERSA

[ftshijt]

We are now actively developing VERSA for general-purpose speech/audio evaluation.

• Additional Metrics to add

  • [x] torch-squim (metrics done, pending adding to the interface) https://pytorch.org/audio/main/tutorials/squim_tutorial.html
  • [ ] torch-log-wmse-audio-quality https://github.com/crlandsc/torch-log-wmse-audio-quality
  • [x] WARP-Q https://github.com/wjassim/WARP-Q
  • [ ] WAWEnets https://github.com/NTIA/WEnets
  • [ ] Peceptual Audio https://github.com/pranaymanocha/PerceptualAudio
  • [ ] VQScore https://github.com/JasonSWFu/VQscore
  • [ ] LMScore https://github.com/soumimaiti/speechlmscore_tool
  • [ ] Audio metrics in https://github.com/haoheliu/audioldm_eval?tab=readme-ov-file#evaluation-metrics

• Additional integration

  • [x] WER (ASR evaluation from ESPnet)
  • [x] SingMOS https://github.com/South-Twilight/SingMOS

• Software development

  • [ ] Webpage for documentation
  • [ ] API level documentation
  • [ ] CI test
  • [ ] User interface for command line + local multiprocessing (currently only with slurm)

Please let us know if you have other metrics to be added to the toolkit as a general evaluation toolkit.

[Emrys365]

DNSMOS Pro (https://github.com/fcumlin/DNSMOSPro) might be a good option.