doctest failures with 9.6

[strogdon]

I have the following failures

sage -t --long --warn-long 91.7 --random-seed=225993847208842150880405223281052662665 /usr/lib/python3.10/site-packages/sage/misc/sagedoc.py  # 1 doctest failed
sage -t --long --warn-long 91.7 --random-seed=225993847208842150880405223281052662665 /usr/lib/python3.10/site-packages/sage/interfaces/expect.py  # 2 doctests failed
sage -t --long --warn-long 91.7 --random-seed=225993847208842150880405223281052662665 /usr/lib/python3.10/site-packages/sage/modules/fp_graded/morphism.py  # 1 doctest failed
sage -t --long --warn-long 91.7 --random-seed=225993847208842150880405223281052662665 /usr/lib/python3.10/site-packages/sage/misc/lazy_import.pyx  # 1 doctest failed
sage -t --long --warn-long 91.7 --random-seed=225993847208842150880405223281052662665 /usr/lib/python3.10/site-packages/sage/features/pkg_systems.py  # 1 doctest failed

Most are pip related which I do not have installed, but the following may be of concern

sage -t --long --warn-long 91.7 --random-seed=225993847208842150880405223281052662665 /usr/lib/python3.10/site-packages/sage/misc/sagedoc.py
**********************************************************************
File "/usr/lib/python3.10/site-packages/sage/misc/sagedoc.py", line 1404, in sage.misc.sagedoc._sage_doc.__call__
Failed example:
    browse_sage_doc(identity_matrix, 'html', False)             # optional - sphinx sagemath_doc_html
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 695, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 1093, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.misc.sagedoc._sage_doc.__call__[2]>", line 1, in <module>
        browse_sage_doc(identity_matrix, 'html', False)             # optional - sphinx sagemath_doc_html
      File "sage/misc/lazy_import.pyx", line 391, in sage.misc.lazy_import.LazyImport.__call__ (build/cythonized/sage/misc/lazy_import.c:4256)
        return self.get_object()(*args, **kwds)
      File "/usr/lib/python3.10/site-packages/sage/misc/sagedoc.py", line 1450, in __call__
        html = sphinxify(s)
      File "/usr/lib/python3.10/site-packages/sage/misc/sphinxify.py", line 122, in sphinxify
        sphinx_app = Sphinx(srcdir, confdir, outdir, doctreedir, format,
      File "/usr/lib/python3.10/site-packages/sphinx/application.py", line 263, in __init__
        self._init_builder()
      File "/usr/lib/python3.10/site-packages/sphinx/application.py", line 321, in _init_builder
        self.builder.init()
      File "/usr/lib/python3.10/site-packages/sphinx/builders/html/__init__.py", line 220, in init
        self.init_templates()
      File "/usr/lib/python3.10/site-packages/sphinx/builders/html/__init__.py", line 268, in init_templates
        self.theme = theme_factory.create(themename)
      File "/usr/lib/python3.10/site-packages/sphinx/theming.py", line 249, in create
        raise ThemeError(__('no theme named %r found (missing theme.conf?)') % name)
    sphinx.errors.ThemeError: no theme named 'sage-classic' found (missing theme.conf?)
**********************************************************************

and

sage -t --long --warn-long 91.7 --random-seed=225993847208842150880405223281052662665 /usr/lib/python3.10/site-packages/sage/modules/fp_graded/morphism.py
**********************************************************************
File "/usr/lib/python3.10/site-packages/sage/modules/fp_graded/morphism.py", line 1571, in sage.modules.fp_graded.morphism.FPModuleMorphism.image
Failed example:
    K.is_injective()  # long time
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 695, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 1093, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.modules.fp_graded.morphism.FPModuleMorphism.image[11]>", line 1, in <module>
        K.is_injective()  # long time
      File "/usr/lib/python3.10/site-packages/sage/modules/fp_graded/steenrod/morphism.py", line 122, in is_injective
        finite_algebra = SteenrodAlgebra_generic(algebra.prime(), profile=self.profile())
      File "/usr/lib/python3.10/site-packages/sage/modules/fp_graded/steenrod/morphism.py", line 90, in profile
        return enveloping_profile_elements(elements,
      File "/usr/lib/python3.10/site-packages/sage/modules/fp_graded/steenrod/profile.py", line 132, in enveloping_profile_elements
        profiles = [profile_elt(x) for x in alist if x != 0]
      File "/usr/lib/python3.10/site-packages/sage/modules/fp_graded/steenrod/profile.py", line 132, in <listcomp>
        profiles = [profile_elt(x) for x in alist if x != 0]
      File "/usr/lib/python3.10/site-packages/sage/modules/fp_graded/steenrod/profile.py", line 82, in profile_elt
        minprofile = [max(0, n.exact_log(char) + 1) for n in elt]
      File "/usr/lib/python3.10/site-packages/sage/modules/fp_graded/steenrod/profile.py", line 82, in <listcomp>
        minprofile = [max(0, n.exact_log(char) + 1) for n in elt]
    AttributeError: 'int' object has no attribute 'exact_log'
**********************************************************************

Relative to sagedoc.py the missing theme.conf does exist as theme.conf.bz2 and I believe I've seen the int failure but within a different context.

[strogdon]

morphism.py does pass when tested individually.

[strogdon]

sage -t --long --warn-long 91.7 --random-seed=225993847208842150880405223281052662665 /usr/lib/python3.10/site-packages/sage/interfaces/expect.py
**********************************************************************
File "/usr/lib/python3.10/site-packages/sage/interfaces/expect.py", line 920, in sage.interfaces.expect.Expect._eval_line
Failed example:
    singular.interrupt()
Expected:
    True
Got:
    False
**********************************************************************
File "/usr/lib/python3.10/site-packages/sage/interfaces/expect.py", line 926, in sage.interfaces.expect.Expect._eval_line
Failed example:
    singular('2+3')
Expected:
    Singular crashed -- automatically restarting.
    5
Got:
    5 
**********************************************************************

also passes when tested individually. So the question is with sagedoc.py.

[kiwifb]

The singular one is new and may reflect some issues with the pexpect interface. The morphism one is a know issue in parallel testing. Probably a threading issue. The sagedoc one may be a recent change I made because I started to use pkgcheck scan from package_core (?). I followed the lead of the science overlay there, for a bit at least https://github.com/gentoo/sci/blob/master/CONTRIBUTING.md. And it recommended a change that may have triggered compression in the doc subfolder. I should inspect shortly.

https://github.com/cschwan/sage-on-gentoo/commit/64267372429a6636aaec8684e8bd4ec71dddb43e

I'd say that's it, I replaced to a doins by a dodoc and it probably automatically compressed it. I'll need to add protection against compression there.

[kiwifb]

Hurgh, my last builds have double "prefix" for the installation directory of common.

$ ll /usr/share/doc/sage-doc-9999/
total 100K
drwxr-xr-x    5 root root 4.0K May 15 14:53 .
drwxr-xr-x 1302 root root  68K May 16 16:18 ..
drwxr-xr-x    5 root root 4.0K May  5 09:14 html
-rw-r--r--    1 root root 8.8K May 15 14:53 index.html
drwxr-xr-x    4 root root 4.0K May  5 09:14 pdf
drwxr-xr-x    3 root root 4.0K May  5 14:33 usr
$ ll /usr/share/doc/sage-doc-9999/usr/share/doc/sage-doc-9999/
total 12K
drwxr-xr-x 3 root root 4.0K May  5 14:33 .
drwxr-xr-x 3 root root 4.0K May  5 14:33 ..
drwxr-xr-x 3 root root 4.0K May  5 14:33 common

Fortunately I am preparing a superior install solution now. One last docompress to make sure of.

[kiwifb]

OK, sagedoc should be fixed in master. There are two other failures to inspect.

[kiwifb]

And they all pass here atm.

[strogdon]

I was so concerned with theme.conf that I didn't notice the double "prefix"

find /usr/share/doc/ | grep theme.conf
/usr/share/doc/sage-doc-9999/usr/share/doc/sage-doc-9999/common/themes/sage-classic/theme.conf.bz2

With the updates sagedoc.py passes

sage -t --long --warn-long 96.7 --random-seed=79921747654407628927535307993053749255 /usr/lib/python3.10/site-packages/sage/misc/sagedoc.py
    [120 tests, 70.88 s]
----------------------------------------------------------------------
All tests passed!
----------------------------------------------------------------------

[strogdon]

On Prefix I have the extra failures:

sage -t --long --random-seed=225934686585708412109830030936156222984 usr/lib/python3.10/site-packages/sage/matrix/matrix_double_dense.pyx
**********************************************************************
File "usr/lib/python3.10/site-packages/sage/matrix/matrix_double_dense.pyx", line 1624, in sage.matrix.matrix_double_dense.Matrix_double_dense.right_eigenvectors
Failed example:
    spectrum[3]  # tol 1e-13
Expected:
    (-1.0, [(1.0, -0.5, 2.0, 0.5)], 1)
Got:
    (-1.0000000000000828,
     [(1.0, -0.4999999999999238, 1.9999999999997218, 0.4999999999999161)],
     1)
Tolerance exceeded in 3 of 6:
    -0.5 vs -0.4999999999999238, tolerance 2e-13 > 1e-13
    2.0 vs 1.9999999999997218, tolerance 2e-13 > 1e-13
    0.5 vs 0.4999999999999161, tolerance 2e-13 > 1e-13

and

sage -t --long --random-seed=225934686585708412109830030936156222984 usr/lib/python3.10/site-packages/sage/schemes/affine/affine_homset.py
**********************************************************************
File "usr/lib/python3.10/site-packages/sage/schemes/affine/affine_homset.py", line 379, in sage.schemes.affine.affine_homset.SchemeHomset_points_affine.numerical_points
Failed example:
    len(E(A.base_ring()).numerical_points(F=CDF, zero_tolerance=1e-9))
Expected:
    100
Got:
    99

Perhaps an out-of-date component? No expect.py failure when doctesting.

[strogdon]

Curiously on the machine where I had the s-o-g failures ( https://github.com/cschwan/sage-on-gentoo/issues/695#issue-1236649054) I have only the vanilla failure

sage -t --long --warn-long 93.1 --random-seed=326768892435172328506243023113108037760 src/sage/schemes/cyclic_covers/cycliccover_finite_field.py
**********************************************************************
File "src/sage/schemes/cyclic_covers/cycliccover_finite_field.py", line 1139, in sage.schemes.cyclic_covers.cycliccover_finite_field.CyclicCover_finite_field.frobenius_polynomial
Failed example:
    CyclicCover(11, PolynomialRing(GF(1129), 'x')([-1] + [0]*(5-1) + [1])).frobenius_polynomial() # long time
Exception raised:
    Traceback (most recent call last):
      File "sage/misc/cachefunc.pyx", line 1943, in sage.misc.cachefunc.CachedMethodCaller.__call__ (build/cythonized/sage/misc/cachefunc.c:10414)
        return cache[k]
    KeyError: ((11,), ())

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/local/sage-git/sage/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages/sage/doctest/forker.py", line 695, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/local/sage-git/sage/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages/sage/doctest/forker.py", line 1093, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.schemes.cyclic_covers.cycliccover_finite_field.CyclicCover_finite_field.frobenius_polynomial[23]>", line 1, in <module>
        CyclicCover(Integer(11), PolynomialRing(GF(Integer(1129)), 'x')([-Integer(1)] + [Integer(0)]*(Integer(5)-Integer(1)) + [Integer(1)])).frobenius_polynomial() # long time
      File "sage/misc/cachefunc.pyx", line 2310, in sage.misc.cachefunc.CachedMethodCallerNoArgs.__call__ (build/cythonized/sage/misc/cachefunc.c:13024)
        self.cache = f(self._instance)
      File "/local/sage-git/sage/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages/sage/schemes/cyclic_covers/cycliccover_finite_field.py", line 1234, in frobenius_polynomial
        F = self.frobenius_matrix(self._N0)
      File "sage/misc/cachefunc.pyx", line 1948, in sage.misc.cachefunc.CachedMethodCaller.__call__ (build/cythonized/sage/misc/cachefunc.c:10550)
        w = self._instance_call(*args, **kwds)
      File "sage/misc/cachefunc.pyx", line 1824, in sage.misc.cachefunc.CachedMethodCaller._instance_call (build/cythonized/sage/misc/cachefunc.c:10016)
        return self.f(self._instance, *args, **kwds)
      File "/local/sage-git/sage/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages/sage/schemes/cyclic_covers/cycliccover_finite_field.py", line 1072, in frobenius_matrix
        FrobP = _frobenius_matrix_p(self._N0)
      File "/local/sage-git/sage/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages/sage/schemes/cyclic_covers/cycliccover_finite_field.py", line 1068, in _frobenius_matrix_p
        ] = self._frob(i, j + self._epsilon * self._r, N0)
      File "/local/sage-git/sage/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages/sage/schemes/cyclic_covers/cycliccover_finite_field.py", line 994, in _frob
        self._initialize_fat_horizontal(
      File "/local/sage-git/sage/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages/sage/schemes/cyclic_covers/cycliccover_finite_field.py", line 738, in _initialize_fat_horizontal
        DH = [elt[0, 0] for elt in interval_products(D0, D1, targets)]
      File "sage/schemes/hyperelliptic_curves/hypellfrob.pyx", line 140, in sage.schemes.hyperelliptic_curves.hypellfrob.interval_products (build/cythonized/sage/schemes/hyperelliptic_curves/hypellfrob.cpp:4008)
        sig_on()
    SystemError: calling remove_from_pari_stack() inside sig_on()

which is https://trac.sagemath.org/ticket/33304

[kiwifb]

Currently running doctesting on sage 9.6 proper before pushing it to master and the "general public". I do get the failure from https://trac.sagemath.org/ticket/33304 I am suspecting a recent upgrade to cython may have been the trigger.

I have no ideas about the other two. I was thinking matrix_double_dense could be linked to a library upgrade, but if so, not one used directly because it links to virtually nothing

$ readelf -d /usr/lib/python3.10/site-packages/sage/matrix/matrix_double_dense.cpython-310-x86_64-linux-gnu.so 

Dynamic section at offset 0x49cf0 contains 24 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000c (INIT)               0x6000

still needs libpython but that's OK.

[strogdon]

I suppose the Prefix failures could be architecture-related

uname -a
Linux vaio-home 3.13.0-170-generic #220-Ubuntu SMP Thu May 9 12:40:49 UTC 2019 x86_64 Intel(R) Core(TM) i7-3632QM CPU @ 2.20GHz GenuineIntel GNU/Linux

If I change

tol 1e-13 -> tol 1e-12

and

zero_tolerance=1e-9 -> zero_tolerance=1e-8

the doctests pass.

[kiwifb]

Which version of ubuntu is this? 14.04? I mean kernel 3.13.0-170 that's fairly old. I could see issue with how old the host glibc is, even if you use the prefix glibc, there may be stuff about how old the host userland is.

[strogdon]

It's Ubuntu 14.04.6 LTS. I hesitate in upgrading.

[strogdon]

The issue:

This is a new Prefix install - my old one got corrupted. The default s-o-g install installs blas/lapack reference libs. I installed openblas and eselected it for blas/lapack and the above failing tests pass. Nice.

[kiwifb]

Some sage doctests are way too precise when it comes to numerical precision. Those were probably within error boundaries for such operations.

[kiwifb]

Closing this one as "sorted".