Randomly appearing failure in schemes/elliptic_curves/padics.py

jdemeyer commented 6 years ago

This failure has been reported after applying two unrelated patches (#23719 and #24735):

File "src/sage/schemes/elliptic_curves/padics.py", line 315, in sage.schemes.elliptic_curves.padics.padic_regulator
Failed example:
    E.padic_regulator(int(5))
Exception raised:
    Traceback (most recent call last):
      File "/home/jdemeyer/sage-git/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 573, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/home/jdemeyer/sage-git/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 983, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.schemes.elliptic_curves.padics.padic_regulator[14]>", line 1, in <module>
        E.padic_regulator(int(Integer(5)))
      File "/home/jdemeyer/sage-git/local/lib/python2.7/site-packages/sage/schemes/elliptic_curves/padics.py", line 339, in padic_regulator
        height = self.padic_height(p, prec, check_hypotheses=False)
      File "/home/jdemeyer/sage-git/local/lib/python2.7/site-packages/sage/schemes/elliptic_curves/padics.py", line 756, in padic_height
        sigma = self.padic_sigma(p, adjusted_prec, check_hypotheses=False)
      File "/home/jdemeyer/sage-git/local/lib/python2.7/site-packages/sage/schemes/elliptic_curves/padics.py", line 1095, in padic_sigma
        E2 = self.padic_E2(p, N-2, check_hypotheses=False)
      File "/home/jdemeyer/sage-git/local/lib/python2.7/site-packages/sage/schemes/elliptic_curves/padics.py", line 1503, in padic_E2
        frob_p = self.matrix_of_frobenius(p, prec, check, check_hypotheses, algorithm).change_ring(Integers(p**prec))
      File "/home/jdemeyer/sage-git/local/lib/python2.7/site-packages/sage/schemes/elliptic_curves/padics.py", line 1653, in matrix_of_frobenius
        Q, p, adjusted_prec, trace)
      File "/home/jdemeyer/sage-git/local/lib/python2.7/site-packages/sage/schemes/hyperelliptic_curves/monsky_washnitzer.py", line 1658, in matrix_of_frobenius
        F1_reduced = reduce_all(Q, p, F1_coeffs, offset)
      File "/home/jdemeyer/sage-git/local/lib/python2.7/site-packages/sage/schemes/hyperelliptic_curves/monsky_washnitzer.py", line 1024, in reduce_all
        exact_form = reduce_negative(Q, p, coeffs, offset, exact_form)
      File "/home/jdemeyer/sage-git/local/lib/python2.7/site-packages/sage/schemes/hyperelliptic_curves/monsky_washnitzer.py", line 800, in reduce_negative
        a[1] = base_ring(lift(a[1]) / (j+1))
      File "sage/structure/parent.pyx", line 920, in sage.structure.parent.Parent.__call__ (build/cythonized/sage/structure/parent.c:9727)
        return mor._call_(x)
      File "sage/structure/coerce_maps.pyx", line 145, in sage.structure.coerce_maps.DefaultConvertMap_unique._call_ (build/cythonized/sage/structure/coerce_maps.c:4554)
        raise
      File "sage/structure/coerce_maps.pyx", line 140, in sage.structure.coerce_maps.DefaultConvertMap_unique._call_ (build/cythonized/sage/structure/coerce_maps.c:4422)
        return C._element_constructor(x)
      File "/home/jdemeyer/sage-git/local/lib/python2.7/site-packages/sage/rings/finite_rings/integer_mod_ring.py", line 1167, in _element_constructor_
        return integer_mod.IntegerMod(self, x)
      File "sage/rings/finite_rings/integer_mod.pyx", line 197, in sage.rings.finite_rings.integer_mod.IntegerMod (build/cythonized/sage/rings/finite_rings/integer_mod.c:4555)
        return t(parent, value)
      File "sage/rings/finite_rings/integer_mod.pyx", line 361, in sage.rings.finite_rings.integer_mod.IntegerMod_abstract.__init__ (build/cythonized/sage/rings/finite_rings/integer_mod.c:5736)
        z = value % self.__modulus.sageInteger
      File "sage/rings/rational.pyx", line 2869, in sage.rings.rational.Rational.__mod__ (build/cythonized/sage/rings/rational.c:25807)
        d = d.inverse_mod(other)
      File "sage/rings/integer.pyx", line 6574, in sage.rings.integer.Integer.inverse_mod (build/cythonized/sage/rings/integer.c:41101)
        raise ZeroDivisionError("Inverse does not exist.")
    ZeroDivisionError: Inverse does not exist.

Component: misc

Keywords: random_fail

Issue created by migration from https://trac.sagemath.org/ticket/25969

timokau commented 6 years ago

comment:1

Either I'm incredibly "lucky" or #24735 really triggered this for me. Observations until now:

only happened with the nix package, not sage-the-distribution. But since I only incremented the singular version and added the #24735 patch, it should not be a packaging issue
doesn't happen if the test is run individually
doesn't happen if I unroll the loop (manually copying the assert 29 times)

timokau commented 6 years ago

Description changed:

--- 
+++ 
@@ -1,4 +1,4 @@
-This failure has been reported after applying two unrelated patches (#23719 and ##24735):
+This failure has been reported after applying two unrelated patches (#23719 and #24735):

File "src/sage/schemes/elliptic_curves/padics.py", line 315, in sage.schemes.elliptic_curves.padics.padic_regulator

timokau commented 6 years ago

Changed keywords from none to random_fail

jdemeyer commented 6 years ago

comment:3

I'm not entirely sure to what extent the failure is really random. It might be deterministic but occur only under a very precise set of conditions.

jdemeyer commented 6 years ago

comment:4

A possible cannon-to-shoot-a-mosquito solution might be #25971.

timokau commented 6 years ago

comment:5

I cannot reproduce on my desktop PC, even though that is executing the exact same build script in a sandbox with the exact same dependencies. It looks like I can reproduce deterministically on my laptop. Unfortunately that takes forever to run the whole testsuite...

timokau commented 6 years ago

comment:6

It looks like this was actually triggered by the rc2 -> rc3 upgrade for me, not the singular update. I didn't test the rc upgrade individually and assumed the singular update would be at fault because the rc upgrade only contained a couple of bug fixes.

timokau commented 6 years ago

comment:7

Although looking at the diff, I can't see anything that might cause this. All the changes in build can be ignored. So the only things remaining are

marking a couple of tests as known bugs
update the version metadata
add a stopgap to DuadicCodeEvenPair and DuadicCodeOddPair

jdemeyer commented 6 years ago

comment:8

Replying to @timokau:

Although looking at the diff, I can't see anything that might cause this. All the changes in build can be ignored. So the only things remaining are

marking a couple of tests as known bugs

update the version metadata

add a stopgap to DuadicCodeEvenPair and DuadicCodeOddPair

I don't think that it's any of those changes in particular. This is a kind of non-local bug that I've seen in the past. Most likely it has something to do with precise memory (de)allocations: an unrelated change in Sage triggers some garbage collection which subtly changes something which eventually leads to this bug.

timokau commented 6 years ago

comment:9

Then what can we do about it? All I can say is that it pretty reliably (2 times in a row) happens after the upgrade and pretty reliable doesn't happen (3 times in a row) before.

vbraun commented 6 years ago

comment:10

Looks like only integer arithmetic is involved; Might be a corrupted mpir limb in the pool due to a preceeding interrupt test. Where exactly the interrupt hits depends on may external factors...

timokau commented 6 years ago

comment:11

Probably unrelated, but when trying to reproduce this on sage-the-dist (I could not), I got another error in the same file:

sage -t --long src/sage/schemes/elliptic_curves/padics.py
**********************************************************************
File "src/sage/schemes/elliptic_curves/padics.py", line 1409, in sage.schemes.elliptic_curves.padics.padic_E2
Failed example:
    EllipticCurve([-1, 1/4]).padic_E2(5, 100)
Expected:
    2 + 4*5 + 2*5^3 + 5^4 + 3*5^5 + 2*5^6 + 5^8 + 3*5^9 + 4*5^10 + 2*5^11 + 2*5^12 + 2*5^14 + 3*5^15 + 3*5^16 + 3*5^17 + 4*5^18 + 2*5^19 + 4*5^20 + 5^21 + 4*5^22 + 2*5^23 + 3*5^24 + 3*5^26 + 2*5^27 + 3*5^28 + 2*5^30 + 5^31 + 4*5^33 + 3*5^34 + 4*5^35 + 5^36 + 4*5^37 + 4*5^38 + 3*5^39 + 4*5^41 + 2*5^42 + 3*5^43 + 2*5^44 + 2*5^48 + 3*5^49 + 4*5^50 + 2*5^51 + 5^52 + 4*5^53 + 4*5^54 + 3*5^55 + 2*5^56 + 3*5^57 + 4*5^58 + 4*5^59 + 5^60 + 3*5^61 + 5^62 + 4*5^63 + 5^65 + 3*5^66 + 2*5^67 + 5^69 + 2*5^70 + 3*5^71 + 3*5^72 + 5^74 + 5^75 + 5^76 + 3*5^77 + 4*5^78 + 4*5^79 + 2*5^80 + 3*5^81 + 5^82 + 5^83 + 4*5^84 + 3*5^85 + 2*5^86 + 3*5^87 + 5^88 + 2*5^89 + 4*5^90 + 4*5^92 + 3*5^93 + 4*5^94 + 3*5^95 + 2*5^96 + 4*5^97 + 4*5^98 + 2*5^99 + O(5^100)
Got:
    2 + 4*5 + 2*5^3 + 5^4 + 3*5^5 + 2*5^6 + 5^8 + 3*5^9 + 4*5^10 + 2*5^11 + 2*5^12 + 2*5^14 + 3*5^15 + 3*5^16 + 3*5^17 + 4*5^18 + 2*5^19 + 4*5^20 + 5^21 + 4*5^22 + 2*5^23 + 3*5^24 + 3*5^26 + 2*5^27 + 3*5^28 + 2*5^30 + 5^31 + 4*5^33 + 3*5^34 + 4*5^35 + 4*5^36 + 2*5^37 + 2*5^38 + 5^39 + 3*5^41 + 3*5^42 + 3*5^43 + 3*5^44 + 4*5^46 + 5^48 + 3*5^49 + 4*5^51 + 2*5^52 + 3*5^53 + 2*5^54 + 2*5^57 + 3*5^59 + 2*5^60 + 5^61 + 3*5^62 + 2*5^64 + 2*5^65 + 2*5^66 + 2*5^67 + 3*5^68 + 2*5^69 + 2*5^70 + 4*5^71 + 5^72 + 4*5^73 + 4*5^76 + 3*5^77 + 4*5^78 + 2*5^79 + 5^80 + 3*5^81 + 2*5^82 + 4*5^83 + 4*5^84 + 3*5^85 + 3*5^87 + 4*5^88 + 3*5^90 + 2*5^92 + 4*5^94 + 4*5^96 + 4*5^98 + O(5^100)
**********************************************************************
1 item had failures:
   1 of  30 in sage.schemes.elliptic_curves.padics.padic_E2
    [199 tests, 1 failure, 33.56 s]

jdemeyer commented 6 years ago

comment:12

Replying to @vbraun:

Looks like only integer arithmetic is involved; Might be a corrupted mpir limb in the pool due to a preceeding interrupt test.

As far as I can see, there is no such test in that file. The doctester tests every file in a separate process, so there can be no interference from interrupt tests in other files.

sagemath / sage

Randomly appearing failure in schemes/elliptic_curves/padics.py #25969