We're not currently testing our benchmarks. This is a problem, because we tend to only run the benchmarks in phases when we're digging into performance. This means it's easy for our benchmarks to be silently broken when we make code changes. This PR adds a benchmark testing workflow, and makes generate_readme_data.py easy to run in a test mode.
We'll want to expand this workflow to test all benchmarks we care about. This means we'll probably need to move it to GPU instances, but we can iterate on that.
PR Status
The benchmark test passes, but what I needed to do to get it to pass has surprised me:
We're installing PyTorch, TorchAudio and TorchVision through conda, not pip. Installing TorchVision through pip encounters the error where it says that the video_reader backend is not available. Installing from source following the official instructions causes a segfault in the benchmark. I admit I did not investigate that much, and eventually tried using conda instead of pip because that's how I happened to have set up my own dev environment, and I don't believe I installed TorchVision from source.
I have had to specify ffmpeg=4 in order for TorchAudio to be able to find the ffmpeg libraries at runtime.
I suspect, but have not yet confirmed, that part of the problem is caused by Decord bundling the FFmpeg binaries in their library. I will test this by running the benchmark without Decord next.
We're not currently testing our benchmarks. This is a problem, because we tend to only run the benchmarks in phases when we're digging into performance. This means it's easy for our benchmarks to be silently broken when we make code changes. This PR adds a benchmark testing workflow, and makes
generate_readme_data.pyeasy to run in a test mode.We'll want to expand this workflow to test all benchmarks we care about. This means we'll probably need to move it to GPU instances, but we can iterate on that.
PR Status The benchmark test passes, but what I needed to do to get it to pass has surprised me:
video_readerbackend is not available. Installing from source following the official instructions causes a segfault in the benchmark. I admit I did not investigate that much, and eventually tried using conda instead of pip because that's how I happened to have set up my own dev environment, and I don't believe I installed TorchVision from source.ffmpeg=4in order for TorchAudio to be able to find the ffmpeg libraries at runtime.I suspect, but have not yet confirmed, that part of the problem is caused by Decord bundling the FFmpeg binaries in their library. I will test this by running the benchmark without Decord next.