search

cschwan / sage-on-gentoo

(Unofficial) Gentoo Overlay for Sage- and Sage-related ebuilds
80 stars 26 forks source link

doctest failures in Prefix (9.1.rc4) #589

Closed strogdon closed 3 years ago

strogdon commented 4 years ago

Some failures intersect with those at https://github.com/cschwan/sage-on-gentoo/issues/560.

sage -t --long usr/lib/python3.7/site-packages/sage/rings/polynomial/pbori.pyx  # Bad exit: 1
##  sage -t --long usr/lib/python3.7/site-packages/sage/doctest/test.py  # 12 doctests failed
sage -t --long usr/lib/python3.7/site-packages/sage/matrix/matrix2.pyx  # Bad exit: 1
sage -t --long usr/lib/python3.7/site-packages/sage/rings/function_field/function_field.py  # Timed out
#  sage -t --long usr/lib/python3.7/site-packages/sage/graphs/generic_graph.py  # 2 doctests failed
sage -t --long usr/lib/python3.7/site-packages/sage/manifolds/differentiable/vectorfield.py  # Bad exit: 1
#  sage -t --long usr/lib/python3.7/site-packages/sage/rings/number_field/number_field.py  # 1 doctest failed
##  sage -t --long usr/lib/python3.7/site-packages/sage/rings/integer.pyx  # AlarmInterrupt in doctesting framework
#  sage -t --long usr/lib/python3.7/site-packages/sage/cpython/dict_del_by_value.pyx  # 2 doctests failed
#  sage -t --long usr/lib/python3.7/site-packages/sage/interfaces/tests.py  # 1 doctest failed
sage -t --long usr/lib/python3.7/site-packages/sage/interfaces/singular.py  # Killed due to segmentation fault
#  sage -t --long usr/lib/python3.7/site-packages/sage/repl/ipython_extension.py  # 5 doctests failed
sage -t --long usr/lib/python3.7/site-packages/sage/interfaces/sagespawn.pyx  # 2 doctests failed
#  sage -t --long usr/lib/python3.7/site-packages/sage/interfaces/giac.py  # 1 doctest failed
#   -  failures also on Gentoo with same number of failures ( failures are probably inconsequential )
##  -  failures also on Gentoo but number/type of failures is different.

sage/rings/function_field/function_field.py passed when tested individually. I need to examine the failures with Bad exit or seg fault. The sage/interfaces/sagespawn.pyx failure is rather odd

sage -t --long usr/lib/python3.7/site-packages/sage/interfaces/sagespawn.pyx                                                                                                                         
**********************************************************************                                                                                                                               
File "usr/lib/python3.7/site-packages/sage/interfaces/sagespawn.pyx", line 148,                                                                                                                      in sage.interfaces.sagespawn.SageSpawn.expect_peek                                                                                                                                                   
Failed example:                                                                                                                             
    E.read().decode('ascii')                                                                                                                                                                    
Expected:
    u'hello world\r\n'
Got:
    'orld\r\n'
**********************************************************************
File "usr/lib/python3.7/site-packages/sage/interfaces/sagespawn.pyx", line 166, in sage.interfaces.sagespawn.SageSpawn.expect_upto
Failed example:
    E.read().decode('ascii')
Expected:
    u'world\r\n'
Got:
    'orld\r\n'
**********************************************************************
2 items had failures:
   1 of   5 in sage.interfaces.sagespawn.SageSpawn.expect_peek
   1 of   5 in sage.interfaces.sagespawn.SageSpawn.expect_upto
    [35 tests, 2 failures, 5.29 s]
kiwifb commented 4 years ago

OK, I got some of those. sagespawn.pyx and ipython_extension.py are especially weird. For doctest/test.py you probably need a newer pyzmq. That's what we figured out on sage-packaging. I am guessing you moved to matplotlib-3.2 and had to upgrade a number of packages and pyzmq has escaped because the runtime dependency is not stated in the terminado ebuild has it should be.

kiwifb commented 4 years ago

pyzmq should take care of a few tests besides test.py.

strogdon commented 4 years ago

Still on matplotlib-3.1.2 with pyzmq-19.0 on Gentoo and Prefix.

kiwifb commented 4 years ago

Hum, show me sage -t --long usr/lib/python3.7/site-packages/sage/doctest/test.py please.

strogdon commented 4 years ago

The failures in sage/doctest/test.py in Prefix are the same as in https://github.com/cschwan/sage-on-gentoo/issues/560. On Gentoo the test fails with

sage -t --long --warn-long 139.5 /usr/lib/python3.7/site-packages/sage/doctest/test.py
**********************************************************************
File "/usr/lib/python3.7/site-packages/sage/doctest/test.py", line 521, in sage.doctest.test
Failed example:
    if system() == "Linux":
        P = subprocess.Popen(["sage", "-t", "--warn-long", "0", "--memlimit=2000", "memlimit.rst"], stdout=subprocess.PIPE, **kwds)
        out, err = P.communicate()
        ok = ("MemoryError: failed to allocate" in bytes_to_str(out))
Expected nothing
Got:
    Process DocTestWorker-1:
    Traceback (most recent call last):
      File "/usr/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
        self.run()
      File "/usr/lib/python3.7/site-packages/sage/doctest/forker.py", line 2172, in run
        task(self.options, self.outtmpfile, msgpipe, self.result_queue)
      File "/usr/lib/python3.7/site-packages/sage/doctest/forker.py", line 2525, in __call__
        result_queue.put(result, False)
      File "/usr/lib/python3.7/multiprocessing/queues.py", line 87, in put
        self._start_thread()
      File "/usr/lib/python3.7/multiprocessing/queues.py", line 170, in _start_thread
        self._thread.start()
      File "/usr/lib/python3.7/threading.py", line 852, in start
        _start_new_thread(self._bootstrap, ())
    RuntimeError: can't start new thread

because the memlimit is not large enough. This also appears in Prefix.

kiwifb commented 4 years ago

Annoying, what about something new like pbori.pyx, what's happening there?

strogdon commented 4 years ago

It must be the doctesting framework that contaminated the pbori.pyx doctest

sage -t --long usr/lib/python3.7/site-packages/sage/rings/polynomial/pbori.pxd
    [0 tests, 0.00 s]
Process DocTestWorker-4:
Traceback (most recent call last):
  File "/storage/strogdon/gentoo-rap/usr/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/storage/strogdon/gentoo-rap/usr/lib/python3.7/site-packages/sage/doctest/forker.py", line 2172, in run
    task(self.options, self.outtmpfile, msgpipe, self.result_queue)
  File "/storage/strogdon/gentoo-rap/usr/lib/python3.7/site-packages/sage/doctest/forker.py", line 2525, in __call__
    result_queue.put(result, False)
  File "/storage/strogdon/gentoo-rap/usr/lib/python3.7/multiprocessing/queues.py", line 87, in put
    self._start_thread()
  File "/storage/strogdon/gentoo-rap/usr/lib/python3.7/multiprocessing/queues.py", line 170, in _start_thread
    self._thread.start()
  File "/storage/strogdon/gentoo-rap/usr/lib/python3.7/threading.py", line 852, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread
sage -t --long usr/lib/python3.7/site-packages/sage/rings/polynomial/pbori.pyx
    Bad exit: 1

since when doctested by itself it passed

sage -t --long usr/lib/python3.7/site-packages/sage/rings/polynomial/pbori.pyx
    [1551 tests, 19.08 s]
----------------------------------------------------------------------
All tests passed!
----------------------------------------------------------------------
Total time for all tests: 20.5 seconds
    cpu time: 15.8 seconds
    cumulative wall time: 19.1 seconds
kiwifb commented 4 years ago

OK so it looks like there is a lot of stuff about managing the load on the machines from memory to thread management. And of course some of these may not be apparent if your machine is big enough. Lastly, there is a segfault in sage/interfaces/singular.py, is it the same kind of things, or is there something in singular to look at?

strogdon commented 4 years ago

Here is the first part of the Cython backtrace for sage/interfaces/singular.py

Cython backtrace
----------------
#0  0x00007f054d5657f0 in __GI___wait4 () at /storage/strogdon/gentoo-rap/var/tmp/portage/sys-libs/glibc-2.31-r2/work/glibc-2.31/posix/../sysdeps/unix/sysv/linux/wait4.c:27
#1  0x00007f0545a8099f in print_enhanced_backtrace () at /storage/strogdon/gentoo-rap/var/tmp/portage/dev-python/cysignals-1.10.2/work/cysignals-1.10.2/build/src/cysignals/implementation.c:563
#2  0x00007f0545a80b3b in sigdie () at /storage/strogdon/gentoo-rap/var/tmp/portage/dev-python/cysignals-1.10.2/work/cysignals-1.10.2/build/src/cysignals/implementation.c:589
#3  0x00007f0545a7fece in sigdie_for_sig () at /storage/strogdon/gentoo-rap/var/tmp/portage/dev-python/cysignals-1.10.2/work/cysignals-1.10.2/build/src/cysignals/implementation.c:164
#4  0x00007f0545a80066 in cysigs_signal_handler () at /storage/strogdon/gentoo-rap/var/tmp/portage/dev-python/cysignals-1.10.2/work/cysignals-1.10.2/build/src/cysignals/implementation.c:262
#5  0x00007f054de3e550 in __restore_rt ()
#6  0x00007f0545a85c48 in __pyx_f_9cysignals_7signals_verify_exc_value () at /storage/strogdon/gentoo-rap/var/tmp/portage/dev-python/cysignals-1.10.2/work/cysignals-1.10.2/build/src/cysignals/signals.c:3682
#7  0x00007f0534b0069b in sig_occurred () at /storage/strogdon/gentoo-rap/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_7/sage/graphs/matchpoly.pyx:0
     1    # cython: binding=True
     2    """
     3    Matching Polynomial
     4
#8  0x00007f0534afebab in fast_tp_dealloc () at /storage/strogdon/gentoo-rap/var/tmp/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_7/sage/rings/integer.pyx:7478
  7473        cdef mpz_ptr o_mpz = <mpz_ptr>((<Integer>o).value)
  7474
  7475        # If we are recovering from an interrupt, throw the mpz_t away
  7476        # without recycling or freeing it because it might be in an
  7477        # inconsistent state (see Trac #24986).
> 7478        if sig_occurred() is NULL:
  7479            if integer_pool_count < integer_pool_size:
  7480                # Here we free any extra memory used by the mpz_t by
  7481                # setting it to a single limb.
  7482                if o_mpz._mp_alloc > 10:
#9  0x00007f054d9edae3 in _PyEval_EvalFrameDefault () at /storage/strogdon/gentoo-rap/var/tmp/portage/dev-lang/python-3.7.7-r2/work/Python-3.7.7/Python/ceval.c:1208
#10 0x00007f054d9eda6d in <Cython.Debugger.libpython.PyUnicodeObjectPtr object at 0x7f21228a8690> ()
#11 0x00007f054d9fe578 in _PyEval_EvalCodeWithName () at /storage/strogdon/gentoo-rap/var/tmp/portage/dev-lang/python-3.7.7-r2/work/Python-3.7.7/Python/ceval.c:3930
#12 0x00007f054d9ff965 in PyEval_EvalCodeEx () at /storage/strogdon/gentoo-rap/var/tmp/portage/dev-lang/python-3.7.7-r2/work/Python-3.7.7/Python/ceval.c:3959
#13 0x00007f054d9eda01 in PyEval_EvalCode () at /storage/strogdon/gentoo-rap/var/tmp/portage/dev-lang/python-3.7.7-r2/work/Python-3.7.7/Python/ceval.c:524
#14 0x00007f054d9e7e17 in builtin_exec_impl () at /storage/strogdon/gentoo-rap/var/tmp/portage/dev-lang/python-3.7.7-r2/work/Python-3.7.7/Python/bltinmodule.c:1079

The start of the failure

sage: a = singular(1) ## line 506 ##
sage: _ = singular._expect.sendline('1+')  # unfinished input ## line 507 ##
sage: try:
    alarm(0.5)
    singular._expect_expr('>')  # interrupt this
except KeyboardInterrupt:
    pass ## line 508 ##
Control-C pressed. Interrupting Singular. Please wait a few seconds...
sage: 2*a ## line 517 ##
------------------------------------------------------------------------
/storage/strogdon/gentoo-rap/usr/lib/python3.7/site-packages/cysignals/signals.cpython-37m-x86_64-linux-gnu.so(+0x5933)[0x7f0545a80933]
/storage/strogdon/gentoo-rap/usr/lib/python3.7/site-packages/cysignals/signals.cpython-37m-x86_64-linux-gnu.so(+0x5b65)[0x7f0545a80b65]
/storage/strogdon/gentoo-rap/usr/lib/python3.7/site-packages/cysignals/signals.cpython-37m-x86_64-linux-gnu.so(+0x4f62)[0x7f0545a7ff62]
/storage/strogdon/gentoo-rap/usr/lib/python3.7/site-packages/cysignals/signals.cpython-37m-x86_64-linux-gnu.so(+0x50e8)[0x7f0545a800e8]
/storage/strogdon/gentoo-rap/lib64/libc.so.6(+0x37940)[0x7f054d4d6940]
/storage/strogdon/gentoo-rap/usr/lib/python3.7/site-packages/cysignals/signals.cpython-37m-x86_64-linux-gnu.so(+0xb968)[0x7f0545a86968]
/storage/strogdon/gentoo-rap/usr/lib/python3.7/site-packages/sage/rings/integer.cpython-37m-x86_64-linux-gnu.so(+0x916d5)[0x7f0534b006d5]
/storage/strogdon/gentoo-rap/usr/lib/python3.7/site-packages/sage/rings/integer.cpython-37m-x86_64-linux-gnu.so(+0x8fbc8)[0x7f0534afebc8]
/storage/strogdon/gentoo-rap/usr/lib64/libpython3.7m.so.1.0(_PyEval_EvalFrameDefault+0x197b)[0x7f054d9ef45e]
/storage/strogdon/gentoo-rap/usr/lib64/libpython3.7m.so.1.0(PyEval_EvalFrameEx+0x60)[0x7f054d9edacd]
kiwifb commented 4 years ago

The cython backtrace seems to imply memory management again. What are the machine specs?

strogdon commented 4 years ago

sage/interfaces/singular.py seg faults even when tested individually. The specs

$ uname -a
Linux blitzen 3.2.0-4-amd64 #1 SMP Debian 3.2.88-1 x86_64 Intel(R) Xeon(R) CPU E5-1650 0 @ 3.20GHz GenuineIntel GNU/Linux
$ cat /proc/meminfo | grep MemTotal
MemTotal:       32945884 kB

It has 12 cores.

 $ cat /proc/cpuinfo | tail -n 27

processor       : 11
vendor_id       : GenuineIntel
cpu family      : 6
model           : 45
model name      : Intel(R) Xeon(R) CPU E5-1650 0 @ 3.20GHz
stepping        : 7
microcode       : 0x70e
cpu MHz         : 3201.000
cache size      : 12288 KB
physical id     : 0
siblings        : 12
core id         : 5
cpu cores       : 6
apicid          : 11
initial apicid  : 11
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips        : 6384.44
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:

Anything else?

strogdon commented 4 years ago

The memory failure in doctest/test.py is slightly different between Gentoo and Prefix

Gentoo:

sage -t --long --warn-long 139.5 /usr/lib/python3.7/site-packages/sage/doctest/test.py
**********************************************************************
File "/usr/lib/python3.7/site-packages/sage/doctest/test.py", line 521, in sage.doctest.test
Failed example:
    if system() == "Linux":
        P = subprocess.Popen(["sage", "-t", "--warn-long", "0", "--memlimit=2000", "memlimit.rst"], stdout=subprocess.PIPE, **kwds)
        out, err = P.communicate()
        ok = ("MemoryError: failed to allocate" in bytes_to_str(out))
Expected nothing
Got:
    Process DocTestWorker-1:
    Traceback (most recent call last):
      File "/usr/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
        self.run()
      File "/usr/lib/python3.7/site-packages/sage/doctest/forker.py", line 2172, in run
        task(self.options, self.outtmpfile, msgpipe, self.result_queue)
      File "/usr/lib/python3.7/site-packages/sage/doctest/forker.py", line 2525, in __call__
        result_queue.put(result, False)
      File "/usr/lib/python3.7/multiprocessing/queues.py", line 87, in put
        self._start_thread()
      File "/usr/lib/python3.7/multiprocessing/queues.py", line 170, in _start_thread
        self._thread.start()
      File "/usr/lib/python3.7/threading.py", line 852, in start
        _start_new_thread(self._bootstrap, ())
    RuntimeError: can't start new thread

Prefix:

File "usr/lib/python3.7/site-packages/sage/doctest/test.py", line 521, in sage.doctest.test
Failed example:
    if system() == "Linux":
        P = subprocess.Popen(["sage", "-t", "--warn-long", "0", "--memlimit=2000", "memlimit.rst"], stdout=subprocess.PIPE, **kwds)
        out, err = P.communicate()
        ok = ("MemoryError: failed to allocate" in bytes_to_str(out))
Expected nothing
Got:
    Traceback (most recent call last):
      File "/storage/strogdon/gentoo-rap/usr/lib/python-exec/python3.7/sage-runtests", line 178, in <module>
        err = DC.run()
      File "/storage/strogdon/gentoo-rap/usr/lib/python3.7/site-packages/sage/doctest/control.py", line 1216, in run
        self.run_doctests()
      File "/storage/strogdon/gentoo-rap/usr/lib/python3.7/site-packages/sage/doctest/control.py", line 912, in run_doctests
        self.dispatcher = DocTestDispatcher(self)
      File "/storage/strogdon/gentoo-rap/usr/lib/python3.7/site-packages/sage/doctest/forker.py", line 1648, in __init__
        init_sage()
      File "/storage/strogdon/gentoo-rap/usr/lib/python3.7/site-packages/sage/doctest/forker.py", line 172, in init_sage
        import sage.repl.ipython_kernel.all_jupyter
      File "/storage/strogdon/gentoo-rap/usr/lib/python3.7/site-packages/sage/repl/ipython_kernel/all_jupyter.py", line 7, in <module>
        from .widgets_sagenb import (input_box, text_control, slider,
      File "/storage/strogdon/gentoo-rap/usr/lib/python3.7/site-packages/sage/repl/ipython_kernel/widgets_sagenb.py", line 30, in <module>
        from ipywidgets.widgets import (IntSlider, IntRangeSlider, FloatSlider,
      File "/storage/strogdon/gentoo-rap/usr/lib/python3.7/site-packages/ipywidgets/__init__.py", line 25, in <module>
        from .widgets import *
      File "/storage/strogdon/gentoo-rap/usr/lib/python3.7/site-packages/ipywidgets/widgets/__init__.py", line 4, in <module>
        from .widget import Widget, CallbackDispatcher, register, widget_serialization
      File "/storage/strogdon/gentoo-rap/usr/lib/python3.7/site-packages/ipywidgets/widgets/widget.py", line 15, in <module>
        from ipykernel.comm import Comm
      File "/storage/strogdon/gentoo-rap/usr/lib/python3.7/site-packages/ipykernel/__init__.py", line 2, in <module>
        from .connect import *
      File "/storage/strogdon/gentoo-rap/usr/lib/python3.7/site-packages/ipykernel/connect.py", line 18, in <module>
        import jupyter_client
      File "/storage/strogdon/gentoo-rap/usr/lib/python3.7/site-packages/jupyter_client/__init__.py", line 4, in <module>
        from .connect import *
      File "/storage/strogdon/gentoo-rap/usr/lib/python3.7/site-packages/jupyter_client/connect.py", line 24, in <module>
        import zmq
      File "/storage/strogdon/gentoo-rap/usr/lib/python3.7/site-packages/zmq/__init__.py", line 47, in <module>
        from zmq import backend
      File "/storage/strogdon/gentoo-rap/usr/lib/python3.7/site-packages/zmq/backend/__init__.py", line 40, in <module>
        reraise(*exc_info)
      File "/storage/strogdon/gentoo-rap/usr/lib/python3.7/site-packages/zmq/utils/sixcerpt.py", line 34, in reraise
        raise value
      File "/storage/strogdon/gentoo-rap/usr/lib/python3.7/site-packages/zmq/backend/__init__.py", line 27, in <module>
        _ns = select_backend(first)
      File "/storage/strogdon/gentoo-rap/usr/lib/python3.7/site-packages/zmq/backend/select.py", line 28, in select_backend
        mod = __import__(name, fromlist=public_api)
      File "/storage/strogdon/gentoo-rap/usr/lib/python3.7/site-packages/zmq/backend/cython/__init__.py", line 6, in <module>
        from . import (constants, error, message, context,
    ImportError: /storage/strogdon/gentoo-rap/usr/lib/python3.7/site-packages/zmq/backend/cython/_version.cpython-37m-x86_64-linux-gnu.so: failed to map segment from shared object
kiwifb commented 4 years ago

The prefix one is interesting and I have never seen it before. But that's definitely some memory management kind of issue. Does the debian host run selinux? Does the gentoo host run hardened?

strogdon commented 4 years ago

No selinux on the Prefix host and the Gentoo machine is not hardened.

strogdon commented 4 years ago

This fails

P = subprocess.Popen(["sage", "-t", "--warn-long", "0", "--memlimit=2000", "memlimit.rst"], stdout=subprocess.PIPE, **kwds)

but this doesn't

P = subprocess.Popen(["sage", "-t", "--warn-long", "0", "--memlimit=3000", "memlimit.rst"], stdout=subprocess.PIPE, **kwds)

The **kwds is defined at the top of doctest/test.py.

strogdon commented 4 years ago

Perhaps similar to https://trac.sagemath.org/ticket/28106

kiwifb commented 4 years ago

Yes, this is the same kind of boat. For some reason the limit is probably too small for your setup. Working out what should be the correct limit and how to properly manage the memory is hard.

strogdon commented 4 years ago

Downgrading from pexpect-4.8.0 to pexpect-4.6.0 fixed

sage -t --long usr/lib/python3.7/site-packages/sage/interfaces/sagespawn.pyx  # 2 doctests failed
kiwifb commented 4 years ago

Oh, yes! I should have remembered that.

strogdon commented 4 years ago

One of the failures in ipython_extension.py

File "usr/lib/python3.7/site-packages/sage/repl/ipython_extension.py", line 385,                                                                                                                      in sage.repl.ipython_extension.SageMagics.fortran                                                                                                                                                   
Failed example:                                                                                                                             
    shell.run_cell('''                                                                                                                                                  
    %%fortran                                                                                                                       
    C FILE: FIB1.F                                                                                                                                      
          SUBROUTINE FIB(A,N)                                                                                                                                                                       
    C                                                                                               
    C     CALCULATE FIRST N FIBONACCI NUMBERS                                                                                                                                                        
    C                                                                                               
          INTEGER N                                                                                                                                         
          REAL*8 A(N)                                                                                                                                               
          DO I=1,N                                                                                                                                      
             IF (I.EQ.1) THEN                                                                                                                                                                       
                A(I) = 0.0D0                                                                                                                                                                    
             ELSEIF (I.EQ.2) THEN                                                                                                                                                                    
                A(I) = 1.0D0                                                                                                                                                                    
             ELSE                                                                                                                                   
                A(I) = A(I-1) + A(I-2)                                                                                                                                                               
             ENDIF                                                                                                                                      
          ENDDO                                                                                                                             
          END                                                                                                                       
    C END FILE FIB1.F                                                                                                                                               
    ''')                                                                                                        
Expected nothing                                                                                                                                
Got:                                                                                            
      File "<ipython-input-1-ab346d8f879f>", line 3                                                                                                                                                  
        C FILE: FIB1.F                                                                                                                                                  
             ^                                                                                                                          
    SyntaxError: invalid syntax                                                                                                                                                                      
    <BLANKLINE>

which is odd.

kiwifb commented 4 years ago

Yes, I know. Some "magic" is not working, specifically "cell_magic ", "line_magic" are fine if I remember correctly. I asked Antonio Arojas on the ticket to upgrade to ipython-7 and he had no clue whatsoever.

strogdon commented 4 years ago

OK, I had ipython-7.5.0 installed. Installed ipython-7.10.0 and 

sage -t --long usr/lib/python3.7/site-packages/sage/repl/ipython_extension.py
    [80 tests, 6.46 s]
----------------------------------------------------------------------
All tests passed!
kiwifb commented 4 years ago

Good to know that it is a ipython problem. But now all the doctests involving dictionary will be failing.

strogdon commented 4 years ago

:disappointed:

kiwifb commented 4 years ago

For the record, I just moved to ipython-7.14.0

sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/combinat/similarity_class_type.py  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/plot/plot3d/shapes2.py  # 6 doctests failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/rings/asymptotic/asymptotic_expansion_generators.py  # 2 doctests failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/graphs/generic_graph.py  # 17 doctests failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/matroids/catalog.py  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/plot/plot3d/base.pyx  # 3 doctests failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/rings/number_field/number_field.py  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/crypto/mq/sr.py  # 2 doctests failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/geometry/polyhedron/base.py  # 3 doctests failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/plot/graphics.py  # 6 doctests failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/coding/linear_code.py  # 2 doctests failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/plot/plot3d/shapes.pyx  # 2 doctests failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/combinat/partition_kleshchev.py  # 8 doctests failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/combinat/crystals/kirillov_reshetikhin.py  # 3 doctests failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/rings/asymptotic/asymptotics_multivariate_generating_functions.py  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/schemes/riemann_surfaces/riemann_surface.py  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/combinat/k_tableau.py  # 4 doctests failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/plot/plot3d/parametric_surface.pyx  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/graphs/graph.py  # 3 doctests failed
sage -t --long --warn-long 100.1 /usr/share/doc/sage-9999/html/en/thematic_tutorials/_sources/sandpile.rst  # 6 doctests failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/combinat/partition.py  # 2 doctests failed
sage -t --long --warn-long 100.1 /usr/share/doc/sage-9999/html/en/prep/_sources/Advanced-2DPlotting.rst  # 3 doctests failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/sandpiles/sandpile.py  # 11 doctests failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/combinat/crystals/mv_polytopes.py  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/plot/plot3d/index_face_set.pyx  # 4 doctests failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/libs/ppl.pyx  # 4 doctests failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/doctest/forker.py  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/combinat/cluster_algebra_quiver/cluster_seed.py  # 2 doctests failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/geometry/polyhedron/ppl_lattice_polytope.py  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/groups/perm_gps/cubegroup.py  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/algebras/cluster_algebra.py  # 2 doctests failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/combinat/multiset_partition_into_sets_ordered.py  # 4 doctests failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/combinat/partition_tuple.py  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/structure/element.pyx  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/plot/plot3d/implicit_surface.pyx  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/categories/pushout.py  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/modular/arithgroup/arithgroup_perm.py  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/rings/tate_algebra_element.pyx  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/rings/number_field/S_unit_solver.py  # 2 doctests failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/databases/sql_db.py  # 8 doctests failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/rings/polynomial/multi_polynomial_ideal.py  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/rings/invariants/invariant_theory.py  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/graphs/digraph.py  # 4 doctests failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/categories/finite_dimensional_algebras_with_basis.py  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/combinat/posets/posets.py  # 3 doctests failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/rings/function_field/divisor.py  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/schemes/toric/morphism.py  # 5 doctests failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/categories/category.py  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/schemes/elliptic_curves/cm.py  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/rings/polynomial/multi_polynomial.pyx  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/knots/link.py  # 6 doctests failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/combinat/finite_state_machine.py  # 12 doctests failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/geometry/triangulation/element.py  # 2 doctests failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/rings/polynomial/polynomial_ring.py  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/combinat/parallelogram_polyomino.py  # 2 doctests failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/categories/primer.py  # 2 doctests failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/modular/arithgroup/congroup_gammaH.py  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/cpython/dict_del_by_value.pyx  # 2 doctests failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/rings/polynomial/groebner_fan.py  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/categories/map.pyx  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/plot/histogram.py  # 2 doctests failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/ext/fast_callable.pyx  # 5 doctests failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/combinat/words/finite_word.py  # 4 doctests failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/combinat/growth.py  # 6 doctests failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/numerical/backends/glpk_backend.pyx  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/interfaces/tests.py  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/rings/padics/lattice_precision.py  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/combinat/root_system/weyl_characters.py  # 7 doctests failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/rings/ring_extension.pyx  # 2 doctests failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/rings/polynomial/multi_polynomial_libsingular.pyx  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/matroids/matroid.pyx  # 4 doctests failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/graphs/graph_coloring.pyx  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/libs/gap/element.pyx  # 4 doctests failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/algebras/quantum_groups/representations.py  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/combinat/ncsf_qsym/tutorial.py  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/modular/dirichlet.py  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/libs/gap/libgap.pyx  # 2 doctests failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/matroids/graphic_matroid.py  # 4 doctests failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/matroids/matroids_plot_helpers.py  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/schemes/toric/weierstrass.py  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/combinat/dyck_word.py  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/graphs/tutte_polynomial.py  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/geometry/polyhedron/combinatorial_polyhedron/base.pyx  # 3 doctests failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/algebras/lie_algebras/lie_algebra_element.pyx  # 3 doctests failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/combinat/matrices/latin.py  # 2 doctests failed
sage -t --long --warn-long 100.1 /usr/share/doc/sage-9999/html/en/thematic_tutorials/_sources/lie/weyl_character_ring.rst  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/rings/polynomial/plural.pyx  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/libs/glpk/error.pyx  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/combinat/set_partition.py  # 2 doctests failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/schemes/projective/projective_space.py  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/graphs/base/c_graph.pyx  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/categories/crystals.py  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/graphs/isgci.py  # 2 doctests failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/matroids/linear_matroid.pyx  # 8 doctests failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/combinat/words/suffix_trees.py  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/rings/multi_power_series_ring_element.py  # 3 doctests failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/rings/polynomial/laurent_polynomial.pyx  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/rings/polynomial/polynomial_element_generic.py  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/structure/unique_representation.py  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/categories/complex_reflection_or_generalized_coxeter_groups.py  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/share/doc/sage-9999/html/en/thematic_tutorials/_sources/tutorial-implementing-algebraic-structures.rst  # 2 doctests failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/rings/finite_rings/hom_finite_field.pyx  # 2 doctests failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/combinat/symmetric_group_representations.py  # 2 doctests failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/graphs/distances_all_pairs.pyx  # 2 doctests failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/graphs/centrality.pyx  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/graphs/base/boost_graph.pyx  # 2 doctests failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/graphs/schnyder.py  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/categories/morphism.pyx  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/algebras/weyl_algebra.py  # 2 doctests failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/combinat/words/words.py  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/rings/polynomial/polydict.pyx  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/structure/indexed_generators.py  # 3 doctests failed
sage -t --long --warn-long 100.1 /usr/share/doc/sage-9999/html/en/thematic_tutorials/_sources/tutorial-programming-python.rst  # 4 doctests failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/matroids/basis_exchange_matroid.pyx  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/schemes/hyperelliptic_curves/invariants.py  # 3 doctests failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/matroids/circuit_closures_matroid.pyx  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/matroids/set_system.pyx  # 2 doctests failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/structure/set_factories.py  # 9 doctests failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/graphs/graph_database.py  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/sandpiles/examples.py  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/sets/finite_set_map_cy.pyx  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/categories/regular_supercrystals.py  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/combinat/integer_lists/base.pyx  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/structure/global_options.py  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/share/doc/sage-9999/html/en/thematic_tutorials/_sources/tutorial-objects-and-classes.rst  # 2 doctests failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/graphs/partial_cube.py  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/doctest/util.py  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/libs/lrcalc/lrcalc.pyx  # 7 doctests failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/misc/abstract_method.py  # 1 doctest failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/parallel/parallelism.py  # 2 doctests failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/structure/proof/all.py  # 2 doctests failed
sage -t --long --warn-long 100.1 /usr/lib/python3.7/site-packages/sage/repl/display/pretty_print.py  # 1 doctest failed

I am going to push thing in https://trac.sagemath.org/ticket/29042 so we can get rid of this distraction and noise.

strogdon commented 4 years ago

doctesting vanilla 9.1 gives

sage -t --long --warn-long 133.2 src/sage/crypto/mq/sr.py  # Killed due to abort
sage -t --long --warn-long 133.2 src/sage/rings/integer.pyx  # 2 doctests failed
sage -t --long --warn-long 133.2 src/sage/rings/polynomial/multi_polynomial_sequence.py  # Killed due to abort
sage -t --long --warn-long 133.2 src/sage/rings/polynomial/pbori.pyx  # Killed due to abort
sage -t --long --warn-long 133.2 src/sage/rings/polynomial/polynomial_ring_constructor.py  # Killed due to abort
sage -t --long --warn-long 133.2 src/sage/sat/solvers/dimacs.py  # Killed due to abort
sage -t --long --warn-long 133.2 src/doc/en/reference/sat/index.rst  # Killed due to abort
sage -t --long --warn-long 133.2 src/sage/sat/boolean_polynomials.py  # Killed due to abort

I suspect (hoping) that these are due to some system component.

kiwifb commented 4 years ago

Almost certainly, but that is extremely annoying to say the least. I am concerned by the number of "killed due to abort". Those don't pass when run individually do they? What is integer.pyx one?

strogdon commented 4 years ago

The integer.pyx failure is a combination of MemoryError and qsieve else

sage -t --long --warn-long 133.2 src/sage/rings/integer.pyx
**********************************************************************
File "src/sage/rings/integer.pyx", line 3118, in sage.rings.integer.Integer.divisors
Failed example:
    for i in range(20):  # long time
        try:
            alarm(RDF.random_element(1e-3, 0.5))
            _ = n.divisors()
            cancel_alarm()  # we never get here
        except AlarmInterrupt:
            pass
Exception raised:
    Traceback (most recent call last):
      File "/local/sage-git/sage/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 681, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/local/sage-git/sage/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 1123, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.rings.integer.Integer.divisors[20]>", line 4, in <module>
        _ = n.divisors()
      File "sage/rings/integer.pyx", line 3188, in sage.rings.integer.Integer.divisors (build/cythonized/sage/rings/integer.c:20545)
        ptr = <unsigned long*>check_allocarray(divisor_count, 3 * sizeof(unsigned long))
      File "memory.pxd", line 87, in cysignals.memory.check_allocarray (build/cythonized/sage/rings/integer.c:46770)
    MemoryError: failed to allocate 33554432 * 24 bytes
**********************************************************************
File "src/sage/rings/integer.pyx", line 3966, in sage.rings.integer.Integer.?
Failed example:
    n.factor(algorithm='qsieve')
Exception raised:
    Traceback (most recent call last):
      File "/local/sage-git/sage/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 681, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/local/sage-git/sage/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 1123, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.rings.integer.Integer.?[14]>", line 1, in <module>
        n.factor(algorithm='qsieve')
      File "sage/rings/integer.pyx", line 4056, in sage.rings.integer.Integer.factor (build/cythonized/sage/rings/integer.c:26295)
        res = [(p, 1) for p in qsieve(n)[0]]
      File "/local/sage-git/sage/local/lib/python3.7/site-packages/sage/interfac
        return qsieve_block(n, time, verbose)
      File "/local/sage-git/sage/local/lib/python3.7/site-packages/sage/interfac
        out, err = p.communicate(str(n))
      File "/local/sage-git/sage/local/lib/python3.7/subprocess.py", line 939, i
        stdout, stderr = self._communicate(input, endtime, timeout)
      File "/local/sage-git/sage/local/lib/python3.7/subprocess.py", line 1681, 
        ready = selector.select(timeout)
      File "/local/sage-git/sage/local/lib/python3.7/selectors.py", line 415, in
        fd_event_list = self._selector.poll(timeout)
      File "src/cysignals/signals.pyx", line 320, in cysignals.signals.python_ch
    cysignals.signals.AlarmInterrupt

I do not have qsieve installed. Wonder why vanilla didn't pull it in if a doctest depends on it? The "Killed due to abort" failures persist when doctested individually.

kiwifb commented 4 years ago

Do the killed doctest have a common denominator?

strogdon commented 4 years ago

Every "Killed due to abort" seems to be related to polybori. For example in

sage -t --long --warn-long 133.2 src/doc/en/reference/sat/index.rst

I have

sage: sr = mq.SR(1,1,1,4,gf2=True,polybori=True) ## line 126 ##
sage: F,s = sr.polynomial_system() ## line 127 ##
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
------------------------------------------------------------------------
/local/sage-git/sage/local/lib/python3.7/site-packages/cysignals/signals.cpython-37m-x86_64-linux-gnu.so(+0x847b)[0x7fa1e508a47b]
/local/sage-git/sage/local/lib/python3.7/site-packages/cysignals/signals.cpython-37m-x86_64-linux-gnu.so(+0x8529)[0x7fa1e508a529]
/local/sage-git/sage/local/lib/python3.7/site-packages/cysignals/signals.cpython-37m-x86_64-linux-gnu.so(+0xbf4d)[0x7fa1e508df4d]
/lib64/libc.so.6(+0x37b40)[0x7fa1ecacab40]
/lib64/libc.so.6(gsignal+0x141)[0x7fa1ecacaac1]
/lib64/libc.so.6(abort+0x127)[0x7fa1ecab485b]
/usr/lib/gcc/x86_64-pc-linux-gnu/9.3.0/libstdc++.so.6(+0xa0aa5)[0x7fa1d359caa5]
/usr/lib/gcc/x86_64-pc-linux-gnu/9.3.0/libstdc++.so.6(+0xd09a6)[0x7fa1d35cc9a6]
/usr/lib/gcc/x86_64-pc-linux-gnu/9.3.0/libstdc++.so.6(+0xd0a11)[0x7fa1d35cca11]
/usr/lib/gcc/x86_64-pc-linux-gnu/9.3.0/libstdc++.so.6(+0xd0c63)[0x7fa1d35ccc63]
/usr/lib/gcc/x86_64-pc-linux-gnu/9.3.0/libstdc++.so.6(+0xa070c)[0x7fa1d359c70c]
/local/sage-git/sage/local/lib/python3.7/site-packages/sage/rings/polynomial/pbori.cpython-37m-x86_64-linux-gnu.so(_ZNSt5dequeIN8polybori14CCuddNavigatorESaIS1_EE17_M_reallocate_mapEmb+0xba)[0x7fa17d43de8a]
/local/sage-git/sage/local/lib/python3.7/site-packages/sage/rings/polynomial/pbori.cpython-37m-x86_64-linux-gnu.so(_ZNSt5dequeIN8polybori14CCuddNavigatorESaIS1_EE23_M_new_elements_at_backEm+0xc5)[0x7fa17d43e015]
/local/sage-git/sage/local/lib/python3.7/site-packages/sage/rings/polynomial/pbori.cpython-37m-x86_64-linux-gnu.so(_ZNSt5dequeIN8polybori14CCuddNavigatorESaIS1_EE13_M_insert_auxISt15_Deque_iteratorIS1_RKS1_PS6_EEEvS5_IS1_RS1_PS1_ET_SD_m+0xb83)[0x7fa17d43ed03]
/local/sage-git/sage/local/lib/python3.7/site-packages/sage/rings/polynomial/pbori.cpython-37m-x86_64-linux-gnu.so(_ZNSt5dequeIN8polybori14CCuddNavigatorESaIS1_EE19_M_range_insert_auxISt15_Deque_iteratorIS1_RKS1_PS6_EEEvS5_IS1_RS1_PS1_ET_SD_St20forward_iterator_tag+0xef)[0x7fa17d43f12f]
/local/sage-git/sage/local/lib/libbrial.so.3(_ZN8polybori13CDegTermStackINS_14CCuddNavigatorENS_9valid_tagENS_11invalid_tagENS_18CAbstractStackBaseIS1_EEE8findTermEm+0x5be)[0x7fa17d12f1ee]
/local/sage-git/sage/local/lib/libbrial.so.3(_ZN8polybori13CDegTermStackINS_14CCuddNavigatorENS_9valid_tagENS_11invalid_tagENS_18CAbstractStackBaseIS1_EEE9incrementEv+0x26a)[0x7fa17d12f7ca]
/local/sage-git/sage/local/lib/libbrial.so.3(_ZN8polybori12COrderedIterINS_14CCuddNavigatorENS_13BooleExponentEE9incrementEv+0x8b)[0x7fa17d11652b]
/local/sage-git/sage/local/lib/libbrial.so.3(_ZN8polybori14dd_print_termsINS_12COrderedIterINS_14CCuddNavigatorENS_13BooleExponentEEENS_13variable_nameINS_13BoolePolyRingEiPKcEENS_14CStringLiteralILj3EEENSA_ILj4EEENS_17integral_constantIjLj1EjEESoEEvT_SF_RKT0_RKT1_RKT2_RKT3_RT4_+0x319)[0x7fa17d116b99]
/local/sage-git/sage/local/lib/libbrial.so.3(_ZNK8polybori15BoolePolynomial5printERSo+0xcd)[0x7fa17d10f08d]

Perhaps vanilla brial needs to be rebuilt? Relative to qsieve, there is no vanilla qsieve package.

strogdon commented 4 years ago

std::bad_alloc is still a memory problem.

strogdon commented 4 years ago

The failures either terminated because of std::bad_alloc or std::length_error.

kiwifb commented 4 years ago

qsieve has left vanilla some time ago. I just haven't gone around to remove it from the overlay for some reason.

strogdon commented 4 years ago

Actually from within the doctesting framework integer.pyx has the second failure as

File "src/sage/rings/integer.pyx", line 3790, in sage.rings.integer.Integer.trial_division
Failed example:
    n = 2 * next_prime(10^40); n.trial_division()
Exception raised:
    Traceback (most recent call last):
      File "/local/sage-git/sage/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 681, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/local/sage-git/sage/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 1123, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.rings.integer.Integer.trial_division[10]>", line 1, in <module>
        n = Integer(2) * next_prime(Integer(10)**Integer(40)); n.trial_division()
      File "/local/sage-git/sage/local/lib/python3.7/site-packages/sage/arith/misc.py", line 1193, in next_prime
        return ZZ(n).next_prime(proof)
      File "sage/rings/integer.pyx", line 5774, in sage.rings.integer.Integer.next_prime (build/cythonized/sage/rings/integer.c:35866)
        while not p._pseudoprime_is_prime(proof):
      File "sage/rings/integer.pyx", line 5309, in sage.rings.integer.Integer._pseudoprime_is_prime (build/cythonized/sage/rings/integer.c:33384)
        return self.__pari__().isprime()
      File "cypari2/gen.pyx", line 2112, in cypari2.gen.Gen.isprime
    cysignals.signals.AlarmInterrupt

but as indicated in https://github.com/cschwan/sage-on-gentoo/issues/589#issuecomment-632955740 when doctested individually.

strogdon commented 4 years ago

Rebuilt brial (sage -f brial) with whatever came along with the rebuild and I now have no "Killed due to abort" failures. Somewhat strange.

kiwifb commented 4 years ago

I notice, with brial that you were already using the vanilla sage's version. Not that it should matter.

/local/sage-git/sage/local/lib/libbrial.so.3(_ZNK8polybori15BoolePolynomial5printERSo+0xcd)[0x7fa17d10f08d]
kiwifb commented 4 years ago

Or may be you configured with the system version but at runtime, it found an old version in the sage/local/ tree which was used instead. In which case cleaning the one from vanilla should have helped.

strogdon commented 4 years ago

Vanilla does not support checking for system brial. I find it curious that rebuilding the vanilla brial was needed. gcc-9.3.0 was installed on 4/23. Perhaps brial was built with gcc-9.2.0-r2? I would hope that that's not the issue.

kiwifb commented 4 years ago

Hum. May be an underlying system library was updated? m4ri would be a prime suspect. I don't think you can use system boost just yet but that would be a big one.

strogdon commented 4 years ago

Vanilla is using system boost. It was upgraded over 2 months ago. System m4rie is also being used. It was upgraded over 3 months ago. Curious. I did have to rebuild a bunch of vanilla packages because system readline had been upgraded. I believe this is unrelated since it was obvious where sage packages had undefined readline symbols.

strogdon commented 4 years ago

pillow-7.1.2 does not seem to play nicely with matplotlib-3.1.2. pillow-7.0.0 seems to work fine. The unstable pillow package was pulled in by Prefix:

sage -t --long usr/lib/python3.7/site-packages/sage/plot/plot.py
**********************************************************************
File "usr/lib/python3.7/site-packages/sage/plot/plot.py", line 94, in sage.plot.plot
Failed example:
    plot(x^2, (x,0,5))
Expected:
    Graphics object consisting of 1 graphics primitive
Got:
    doctest:warning
...
        import matplotlib.pyplot as plt
      File "<frozen importlib._bootstrap>", line 983, in _find_and_load
      File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
      File "<frozen importlib._bootstrap_external>", line 728, in exec_module
      File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
      File "/storage/strogdon/gentoo-rap/usr/lib/python3.7/site-packages/matplotlib/pyplot.py", line 33, in <module>
        import matplotlib.image
      File "<frozen importlib._bootstrap>", line 983, in _find_and_load
      File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
      File "<frozen importlib._bootstrap_external>", line 728, in exec_module
      File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
      File "/storage/strogdon/gentoo-rap/usr/lib/python3.7/site-packages/matplotlib/image.py", line 17, in <module>
        from matplotlib.backend_bases import FigureCanvasBase
      File "<frozen importlib._bootstrap>", line 983, in _find_and_load
      File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
      File "<frozen importlib._bootstrap_external>", line 728, in exec_module
      File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
      File "/storage/strogdon/gentoo-rap/usr/lib/python3.7/site-packages/matplotlib/backend_bases.py", line 57, in <module>
        from PIL import PILLOW_VERSION
      File "<frozen importlib._bootstrap>", line 1032, in _handle_fromlist
      File "/storage/strogdon/gentoo-rap/usr/lib/python3.7/site-packages/PIL/__init__.py", line 40, in __getattr__
        _raise_version_warning()
      File "/storage/strogdon/gentoo-rap/usr/lib/python3.7/site-packages/PIL/__init__.py", line 32, in _raise_version_warning
        stacklevel=3,
      File "/storage/strogdon/gentoo-rap/usr/lib/python3.7/warnings.py", line 110, in _showwarnmsg
        msg.file, msg.line)
    :
    DeprecationWarning: PILLOW_VERSION is deprecated and will be removed in a future release. Use __version__ instead.
    Graphics object consisting of 1 graphics primitive
strogdon commented 4 years ago

Actually with pillow-7.0.0

>>> import PIL
>>> PIL.__version__
'7.0.0'
>>> from PIL import PILLOW_VERSION
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name 'PILLOW_VERSION' from 'PIL' (/storage/strogdon/gentoo-rap/usr/lib/python3.7/site-packages/PIL/__init__.py)

So I'm not quite sure why 7.0.0 works.

kiwifb commented 4 years ago

Neither do I!

strogdon commented 4 years ago

I think I see. Starting with line 57 in /usr/lib/python3.7/site-packages/matplotlib/backend_bases.py there is

try:
    from PIL import PILLOW_VERSION
    from distutils.version import LooseVersion
    if LooseVersion(PILLOW_VERSION) >= "3.4":
        _has_pil = True
    else:
        _has_pil = False
    del PILLOW_VERSION
except ImportError:
    _has_pil = False

The from PIL import PILLOW_VERSION throws an ImportError with 7.0.0 while with 7.1.2 the import throws a DeprecationWarning. The same code exists in matplotlib-3.2.1.

strogdon commented 4 years ago

So, maybe not even pillow-7.0.0 should be used, since the above code returns 

_has_pil = False

even when an appropriate pillow is installed; unless I'm missing something.

kiwifb commented 4 years ago

Possibly. In that case this is a ticket for bugzilla that the dependency on pillow should be changed. Off course it would be awesome if matplotlib itself had a requirement list somewhere.

strogdon commented 4 years ago

It appears that matplotlib only needs pillow with USE=doc. I have USE=-doc. So technically it should not be needed.

kiwifb commented 4 years ago

It appears to me, reading backend_bases.py, that it is a bogus claim. pillow is a backend to handle jpeg and tiff files by default with agg.

strogdon commented 4 years ago

OK matplotlib bug: https://github.com/matplotlib/matplotlib/issues/16083. I guess either patch or use <=pillow-6.2.2 if jpeg or tiff needed. The gentoo ebuild should have dependency on pillow in some other way.

strogdon commented 4 years ago

FYI, see https://bugs.gentoo.org/727032