SpectralSequences / sseq

The root repository for the SpectralSequences project.
Apache License 2.0
25 stars 10 forks source link

Flaky chart test #86

Closed dalcde closed 2 years ago

dalcde commented 2 years ago

e.g. https://github.com/SpectralSequences/sseq/runs/5711814604 and https://github.com/SpectralSequences/sseq/runs/5703909304 .

hoodmane commented 2 years ago

Thanks, will look at it. Does seem to have the same falsifying example both times:

state = StateMachinePythonToJavascript()
v1 = state.add_class(k=(0, 0))
v2 = state.add_extension(c1=v1, c2=v1)
state.set_visible(obj_and_range=(v2, None), val=False)
state.update_1()
state.teardown()

Ideally it is probably possible to store the hypothesis cache between runs which makes it remember failing tests and consistently rerun them. This still allows it to find the counterexample for the first time on an unrelated commit but it turns into a unidirectional state change passing ==> failing. Though of course that could still be annoying because it looks like a regression on the commit but it's actually unrelated. In this case since most commits are unrelated to the charts it shouldn't be a big deal.

dalcde commented 2 years ago

If you don't have time to look at this, is there a way to temporarily disable the test?

hoodmane commented 2 years ago

Resolved by 4fee0d11c712834d25d721b7306271685c1e859c.