search

argmaxinc / WhisperKit

On-device Speech Recognition for Apple Silicon
https://takeargmax.com/blog/whisperkit
MIT License
3.17k stars 268 forks source link

Indeterminate tests #100

Closed ZachNagengast closed 2 months ago

ZachNagengast commented 5 months ago

For some reason, occasionally visionOS unit tests are failing at the very end, see: https://github.com/argmaxinc/WhisperKit/actions/runs/8525582362/job/23352857267?pr=99

The only error visible is Error: The operation was canceled. Would be great to figure out why that's happening and resolve it so the tests succeed properly.

jkrukowski commented 5 months ago

Looks like here the reason is a timeout, see here https://github.com/argmaxinc/WhisperKit/actions/runs/8525582362?pr=99

Screenshot 2024-04-15 at 13 49 28

unit tests on CI seems to be configured that way https://github.com/argmaxinc/WhisperKit/blob/8564ce2bbec03649526678f51f4e249e9e052a6c/.github/workflows/unit-tests.yml#L17

Should we increase the timeout (to e.g. 30 minutes)? or we could parallelise tests for each platform

ZachNagengast commented 5 months ago

I see, makes sense. In this case, we should not only run our tests in parallel but also run each platform in parallel. If its taking longer than 20 minutes that's too long for unit tests imo.

jkrukowski commented 5 months ago

I see, makes sense. In this case, we should not only run our tests in parallel but also run each platform in parallel. If its taking longer than 20 minutes that's too long for unit tests imo.

ok, let me try to do it then

ZachNagengast commented 3 months ago

This issue appears to have started coming back, but for any test besides macOS

image
jkrukowski commented 3 months ago

@ZachNagengast could you give a link to a failing job (somehow I couldn't find a relevant one), I'll try to inspect it

ZachNagengast commented 3 months ago

@jkrukowski here is an example https://github.com/argmaxinc/WhisperKit/actions/runs/9543240677/job/26299468908?pr=166