(🎁) Show time diff in output

hauntsaninja / mypy_primer

Run mypy and pyright over millions of lines of code

MIT License

55 stars 29 forks source link

(🎁) Show time diff in output #79

Closed KotlinIsland closed 1 year ago

KotlinIsland commented 1 year ago

Looks a little like this https://github.com/KotlinIsland/basedmypy/pull/357#issuecomment-1501731245

JelleZijlstra commented 1 year ago

I don't think showing the raw times is a good idea; timing is too finicky and it may vary randomly based on whatever GH Actions is up to.

To do this well, you'd need some real benchmarking setup (like what pyperf does), but that requires many runs and would make mypy-primer slower overall.

JukkaL commented 1 year ago

Could we highlight major regression at least? E.g. if the runtime gets 5x slower, it's probably a real regression. This might still be too flaky to be useful, but it seems worth a try.

hauntsaninja commented 1 year ago

Yeah, I've been meaning to do this, that's why I added the runtime measurements. I think I fixed a bug and the timing differences aren't too bad, I'll make a version of this that only shows perf differences >10%

hauntsaninja commented 1 year ago

Okay, I pushed a version of this in https://github.com/hauntsaninja/mypy_primer/commit/bedddb75e10fe0f2cbd01e5d0b4f3b6098d89e0c

Thanks for bumping this!