search

python-trio / flake8-async

Highly opinionated linter for Trio code
https://flake8-async.readthedocs.io
MIT License
17 stars 2 forks source link

run fuzz in CI #165

Closed jakkdl closed 1 year ago

jakkdl commented 1 year ago

let's see how long they take to run in CI, I usually don't run them in my workflow locally since they're slow - but it's probably nice if they're run regularly somewhere

jakkdl commented 1 year ago

The check is once again fixed in #163, I kind of expect the fuzz tests to fail currently - so make sure to rerun after merging the others ... if the runtime looks the least bit reasonable.

Zac-HD commented 1 year ago

~20 minutes to run the fuzztests seems fine to me, we could make that a non-required CI check. pytest-xdist would probably make that ~2x faster too.

If we're getting up above 20-30 minutes, I'd probably make this a scheduled job and look at the results occasionally. Fancier schemes can actually share a cache between them and use that to feed discovered failures back into the main job, but that's still pretty early for Hypothesis+HypoFuzz so I don't think I'd use it here yet.

jakkdl commented 1 year ago

Added a flag so slow_tests only runs the slow tests, added xdist, and rebased on top of main. with "non-required" do you simple mean not listing it in needs for the release test?

Amusingly enough it actually runs faster in CI than on my personal computer now, due to me having way more installed packages ... _iter_python_files() gives 3900 files :sweat_smile: (inside tox environment it's a more reasonable 1561 ... which itself takes 11 mins on my laptop :pensive: )

Zac-HD commented 1 year ago

with "non-required" do you simple mean not listing it in needs for the release test?

I mean "oops, let me go set up some branch protection rules so you can't merge a PR until the tests pass" 😆