I changed all model/architecture tests to use tiny random parameter checkpoints from hf-internal-testing HF org and in some cases I uploaded repos to bumblebee-testing HF org. These tests are no longer marked as slow, so we will run them on CI every time, I also marked all of them async since they no longer use that much memory. Serving tests are end-to-end and should produce human-readable output, so we still use actual checkpoints there (small whenever possible) and those are still tagged as slow.
Running mix test (excludes servings) for the first time took 47s for me (depends mostly on networking) and downloads only 315MB, next runs take 9s, and disabling network traffic with BUMBLEBEE_OFFLINE=true mix test drops it to 4s :D
Running all tests with populated cache BUMBLEBEE_OFFLINE=true mix test --include slow is down from 200s to 73s.
In the process I found a couple bugs, so there are a couple changes here and there.
I changed all model/architecture tests to use tiny random parameter checkpoints from
hf-internal-testingHF org and in some cases I uploaded repos tobumblebee-testingHF org. These tests are no longer marked as slow, so we will run them on CI every time, I also marked all of them async since they no longer use that much memory. Serving tests are end-to-end and should produce human-readable output, so we still use actual checkpoints there (small whenever possible) and those are still tagged as slow.Running
mix test(excludes servings) for the first time took 47s for me (depends mostly on networking) and downloads only 315MB, next runs take 9s, and disabling network traffic withBUMBLEBEE_OFFLINE=true mix testdrops it to 4s :DRunning all tests with populated cache
BUMBLEBEE_OFFLINE=true mix test --include slowis down from 200s to 73s.In the process I found a couple bugs, so there are a couple changes here and there.