search

bencherdev / bencher

🐰 Bencher - Continuous Benchmarking
https://bencher.dev
Other
532 stars 24 forks source link

Bencher succeed without running any benchmarks with `rust_criterion` adapter on Windows. #516

Open TheVeryDarkness opened 1 week ago

TheVeryDarkness commented 1 week ago

Hello! I'm using bencher in Github Actions for one of my repos in Rust, and I'm using adapter rust_criterion, but I found bencher works well on macos-latest and linux-latest, but succeed without running any benchmarks on Windows.

You can find results from a workflow run in https://github.com/TheVeryDarkness/iof/actions/runs/11302133786.

epompeii commented 6 days ago

@TheVeryDarkness thank you for using Bencher, and I'm sorry you're running into trouble.

I do have a CI job that runs the example Criterion benchmarks on Windows. This is output from my most recent push: https://github.com/bencherdev/bencher/actions/runs/11301003886/job/31434759390#step:4:2883

Taking a look at your workflow file: https://github.com/TheVeryDarkness/iof/blob/main/.github%2Fworkflows%2Fbenchmark.yml#L32 I would recommend trying to wrap cargo bench in quotes: "cargo bench" And see if that helps. This is effectively what the example does: https://github.com/bencherdev/bencher/blob/8a1cc54db6deaffcae017824ff5d2bf3236484ac/tasks/test_api/src/task/test/examples.rs#L156-L158

TheVeryDarkness commented 5 days ago

Adding quotes there doesn't seem to be working :(

Workflow run: https://github.com/TheVeryDarkness/iof/actions/runs/11320056080

There was an error when running on linux-latest because I had some bugs in my benchmarks.

I suspect it has something to do with the spaces in my benchmark names.

TheVeryDarkness commented 4 days ago

Well, it seems it's not caused by the spaces.

Workflow run: https://github.com/TheVeryDarkness/iof/actions/runs/11323643624/job/31486739251

TheVeryDarkness commented 4 days ago

And cargo bench -- --list can list all benchmarks.

Workflow run: https://github.com/TheVeryDarkness/iof/actions/runs/11323828042/job/31487307640.

epompeii commented 3 days ago

@TheVeryDarkness thank you for trying out using quotes. I'm not very knowledgeable on Windows development, so I figured it was worth a shot.

If you run your benchmarks on a Windows box without Bencher, what does the output look like?