search

airspeed-velocity / asv

Airspeed Velocity: A simple Python benchmarking tool with web-based reporting
https://asv.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
875 stars 180 forks source link

More feedback if benchmark segfaults? #900

Open takluyver opened 4 years ago

takluyver commented 4 years ago

I don't really know what's the best approach to this, but it's something that confused me recently. I had a benchmark segfault, and the only visible feedback was asv saying 'failed', even when I used -e to show stderr. It was only by running the benchmark manually outside asv that I was even able to see it was a segfault. In this case it was a consistent failure, but if you have a stochastic failure that only occurs 1 time in 20, it would be even more frustrating.

It would be good to see some more information if a benchmark process crashes - even just 'failed (SIGSEGV)' would be a useful hint. Maybe it would be worth enabling faulthandler by default in the benchmark processes as well? If it's too much information for the main output, it could be logged to a file somewhere. Maybe more detail is already logged to a file somewhere, but I didn't know about it?

jrbourbeau commented 4 years ago

Perhaps the new --strict option (xref https://github.com/airspeed-velocity/asv/pull/865) would be of use in the case that a benchmark segfaults

pv commented 4 years ago

The current asv master version should print the exit status when running with -e (#817).

takluyver commented 4 years ago

Showing the exit status is probably sufficient, so feel free to close this if you want.

I might favour some way of recording or reporting some kind of additional information about failures when not using -e, because if you run without it normally, debugging a rare failure could be annoying. But that's orthogonal to what prompted me to open this issue, and I haven't spent enough time with asv to have a good idea how it fits into the overall design.