argmaxinc / WhisperKit

On-device Speech Recognition for Apple Silicon
http://argmaxinc.com/blog/whisperkit
MIT License
3.92k stars 331 forks source link

Implement memory and latency regression tests #61

Closed atiorh closed 6 months ago

atiorh commented 8 months ago

Implement tests to transcribe long audio files (at least several minutes worth) and measure the memory and latency over time. This is to guard against memory leaks or slowdowns potentially being introduced by new PRs (e.g. #40 fixed by #56 thanks to @finnvoor!)

Abhinay1997 commented 7 months ago

Any pointers on how to start with this. I'd like to take it up.

I assume the tests would be functionaltests and we'd be measuring the metrics but not sure on the details.

ZachNagengast commented 7 months ago

@Abhinay1997 Check inside FunctionalTests.swift for the measure calls. We'll want to save these results somewhere so they can be compared to in subsequent runs, but we'd need to makes sure it is hardware specific so we don't think there's a regression on a slower computer vs a faster one.

Abhinay1997 commented 7 months ago

PR: #99