Normalize stacktrace diffs

JelleZijlstra commented 3 years ago

Reference: https://github.com/python/mypy/pull/10686#issuecomment-905110424

It would be nice if mypy-primer didn't tell us that the stacktrace for a crash now uses "new_mypy" instead of "old_mypy" in its path, or that the version of mypy changed due to a new commit hash.

JukkaL commented 3 years ago

Also, we could filter out line numbers from tracebacks.

Somewhat unrelated: What about always reporting crashes, even if they happen with old mypy? These are usually something we'd want to fix quickly.

hauntsaninja commented 3 years ago

Makes sense, although the current lack of stacktrace special casing effectively amounts to "always report crashes". The arviz crash is something I added to mypy_primer this week: we've had multiple crashes in the past reported by people using the xarray library so I went looking for projects that would give us better coverage of that.

hauntsaninja commented 2 years ago

I pushed b4e8559 to normalise filenames. So for typeshed's CI should no longer see mypy crashes (that were not caused by the relevant typeshed change).

Given this, I'll probably add an option to always report a crash, since for mypy's CI, we're still very interested in seeing all crashes. However this is low pri:

In practice it seems like people are good about paying attention to new crashes in PRs, which is the chief avenue I'd expect to see them. So I don't think we need to worry too much about merging a change that causes a crash and then needing reminders about it.
I believe there are no crashes with mypy master.
That leaves the case where one of the projects we check makes a change that causes a crash. I don't think this has happened. Most projects with crashes are from me trying to find projects where mypy breaks.

hauntsaninja / mypy_primer

Normalize stacktrace diffs #18