ASR evaluation

[keighrim]

• Goal: get numerical evaluation of two existing ASR apps in CLAMS
  • Kaldi: https://github.com/clamsproject/app-aapb-pua-kaldi-wrapper
  • Whisper: https://github.com/clamsproject/app-whisper-wrapper
• Some source data can be obtained from https://github.com/clamsproject/wgbh-collaboration/issues/21
• For evaluation metrics, we use WER (edit-distance) (https://torchmetrics.readthedocs.io/en/stable/text/word_error_rate.html, https://pypi.org/project/jiwer/ , etc.)

Specifically development of such evaluation code will involve,

• Set up two apps in a local computer
• Run N videos through two apps and collect MMIF output files
• Write a separate python program that takes a collection of MMIF files with ASR results and compute WER, individually and collectively (by averaging, maybe?)
• Optionally, it’d be great to have a visual evaluation such as (https://github.com/clamsproject/wgbh-collaboration/issues/22#issuecomment-1423098906)

[keighrim]

When the code is ready, please do this.

1. Create a local branch that starts with this issue number and a dash. Pick any name after that (e.g. 2-asr-ev).
2. Add commits to the branch.
3. Push the branch to GH via git push -u origin 2-somethingsomething
4. Create a pull request from 2- branch to the main branch, so we can start a code review before it's "released".

[keighrim]

@Brendayy , I'm still waiting for your push. Whenever it's ready.

[keighrim]

closed via #13