raynelfss
commented
2 years ago Since we'll be dealing with actions when it comes to continuous integration, I want to recommend using act. This addition allows running our actions locally for testing purposes. This might be helpful for extending CI jobs. It works well on Ubuntu (wsl2).
The next step would be to figure out a way to detect new additions in our benchmark files for which I've been checking the tj-actions/changed-files repo in order to figure that out.
Right now the CI jobs configured on CI do not attempt to run benchmarks at all, we only run the lint job. This is primarily because running benchmarks can be slow so for CI we started just with the lint jobs. However, to aid in review and catching potential issues prior to merging we should add a new github actions job that detects when a new benchmark file is added or we're modifying an existing benchmark file and run pytest on that file. This should hopefully limit the runtime to be manageable (except for PRs that touch every benchmark I guess) so that we can run it in our CI time budget.