search

sagemath / sage

Main repository of SageMath
https://www.sagemath.org
Other
1.33k stars 453 forks source link

Update doctesting framework #12415

Closed robertwb closed 11 years ago

robertwb commented 12 years ago

There are several improvements that would be good to make, including but not limited to:

  1. Forking rather than starting up a new Sage each time.
  2. Customizable comparison rather than mutating doctests into new doctests (e.g. followup to #10952).
  3. Eliminate need to write temporary files for every doctested file.
  4. Constant headaches caused by the difference in doctesting library vs. non-library files.
  5. The ability to drop into a debugger if a doctest raises an exception.

Robert's code is at

http://code.google.com/p/sagemath-timer/

Apply attachment: 12415_framework.patch, attachment: 12415_doctest_fixes.patch, attachment: 12415_doc.patch, attachment: 12415_review.patch, attachment: 12415_test.patch, attachment: 12415_review_review.patch, attachment: 12415_review3.patch, attachment: 12415_manifest.patch, attachment: 12415_rebase_58.patch

Apply attachment: 12415_script.patch and attachment: 12415_script_review.patch to the scripts repo

Apply attachment: 12415_spkg_bin_sage.patch to the root repo

For follow-up or other doctest-related tickets see #11337.

Depends on #13147 Depends on #13146 Depends on #13145 Depends on #12723 Depends on #12392 Depends on #12393 Depends on #12395 Depends on #12396 Depends on #12397 Depends on #12381 Depends on #12382 Depends on #12383 Depends on #12384 Depends on #11871 Depends on #13195 Depends on #13121 Depends on #13748 Depends on #13899 Depends on #12719 Depends on #5155 Depends on #14070 Depends on #14079 Depends on #14150 Depends on #14158 Depends on #14182 Depends on #14184 Depends on #14054 Depends on #14063 Depends on #13605 Depends on #14111 Depends on #14254 Depends on #14242 Depends on #14253

CC: @kini @ohanar @jhpalmieri

Component: doctest framework

Author: David Roe, Robert Bradshaw, Jeroen Demeyer

Reviewer: Jeroen Demeyer, David Roe

Merged: sage-5.9.beta0

Issue created by migration from https://trac.sagemath.org/ticket/12415

jdemeyer commented 12 years ago
comment:53

Replying to @roed314:

Design question: when running tests in parallel, failed examples are printed as they arise, meaning that examples from different files are intermingled. Is this a problem that needs to be fixed?

Even better might be an option to show only the first failure in each file.

roed314 commented 12 years ago
comment:54

I've updated the code to group results for each file, and added an option (sage -t --initial) so that only the first failure is reported.

jdemeyer commented 12 years ago
comment:55

The following is cut from a make ptestlong run:

sage -t devel/sage/sage/symbolic/expression.pyx
    [1960 tests, 38.2 s]
sage -t devel/sage/sage/pl**********************************************************************
File "sage.schemes.elliptic_curves.ell_rational_field", line 1468, in sage.schemes.elliptic_curves.ell_rational_field.EllipticCurve_ration
al_field.simon_two_descent
Failed example:
    E.simon_two_descent()  # long time (9s on sage.math, 2011)
Expected:
    (5, 5, [(5 : 8 : 1), (4 : 9 : 1), (3 : 11 : 1), (-1 : 20 : 1), (-6 : -25 : 1)])
Got:
    Saturation index bound = 265
    WARNING: saturation at primes p > 97 will not be done;
    points may be unsaturated at primes between 97 and index bound
    Failed to saturate MW basis at primes [ ]
    Saturation index bound = 265
    WARNING: saturation at primes p > 199 will not be done;
    points may be unsaturated at primes between 199 and index bound
    Failed to saturate MW basis at primes [ ]
    (5, 5, [(5 : 8 : 1), (4 : 9 : 1), (3 : 11 : 1), (-1 : 20 : 1), (-6 : -25 : 1)])
**********************************************************************
1 items had failures:
   1 of  38 in sage.schemes.elliptic_curves.ell_rational_field.EllipticCurve_rational_field.simon_two_descent
***Test Failed*** 1 failures.
ot/plot3d/implicit_plot3d.py
    [57 tests, 26.9 s]
sage -t devel/sage/sage/modular/modsym/ambient.py
    [397 tests, 26.4 s]

There are 3 issues:

  1. The doctest failure itself.

  2. The fact that sage -t foo is written instead of sage -t --long foo.

  3. Output flushing: note the

sage -t devel/sage/sage/pl

and then later

ot/plot3d/implicit_plot3d.py
    [57 tests, 26.9 s]

By all means please keep working on this, it already looks very cool!

jdemeyer commented 12 years ago
comment:56

Some more doctest failures from make ptest:

sage -t devel/sage/doc/common/__init__.py
    [0 tests, 0.0 s]
sage -t devel/sage/doc/common/build_optiUsage: sage -docbuild [OPTIONS] DOCUMENT (FORMAT | COMMAND)

sage-runtests: error: no such option: -p
ons.py
    [0 tests, 0.0 s]
sage -t devel/sage/doc/common/builder.py
    Bad exit: 2
Tests run before process failed:

sage -t devel/sage/doc/common/conf.py
    [0 tests, 0.0 s]
sage -t devel/sage/doc/de/tutorial/interactive_shell.rst
    Error: TAB character found at line 656
    [8 tests, 0.2 s]
sage -t devel/sage/doc/en/tutorial/interactive_shell.rst
    [8 tests, 0.2 s]
sage -t devel/sage/doc/en/bordeaux_2008/birds_other.rst
    [26 tests, 5.2**********************************************************************
File "interfaces", line 331, in interfaces
Failed example:
    maxima.plot3d("[cos(x)*(3 + y*cos(x/2)), sin(x)*(3 + y*cos(x/2)),\
      y*sin(x/2)]", "[x, -4, 4]", "[y, -4, 4]",\
      '[plot_format, openmath]') # not tested
Exception raised:
    Traceback (most recent call last):
      File "/release/merger/sage-5.1.beta5-12415/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 590, in _run
        compileflags, 1)
      File "<doctest interfaces[0]>", line 2
        maxima.plot3d("[cos(x)*(3 + y*cos(x/2)), sin(x)*(3 + y*cos(x/2)),\
      y*sin(x/2)]", "[x, -4, 4]", "[y, -4, 4]",\

                                                 ^
    SyntaxError: unexpected character after line continuation character
**********************************************************************
1 items had failures:
   1 of  57 in interfaces
***Test Failed*** 1 failures.
**********************************************************************
File "interfaces", line 318, in interfaces
Failed example:
    maxima.plot3d("[cos(x)*(3 + y*cos(x/2)), sin(x)*(3 + y*cos(x/2)),\
      y*sin(x/2)]", "[x, -4, 4]", "[y, -4, 4]",\
      '[plot_format, openmath]') # not tested
Exception raised:
    Traceback (most recent call last):
      File "/release/merger/sage-5.1.beta5-12415/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 590, in _run
        compileflags, 1)
      File "<doctest interfaces[0]>", line 2
        maxima.plot3d("[cos(x)*(3 + y*cos(x/2)), sin(x)*(3 + y*cos(x/2)),\
      y*sin(x/2)]", "[x, -4, 4]", "[y, -4, 4]",\

                                                 ^
    SyntaxError: unexpected character after line continuation character
**********************************************************************
1 items had failures:
   1 of  57 in interfaces
***Test Failed*** 1 failures.
 s]
sage -t devel/sage/doc/fr/tutorial/interactive_shell.rst
    [8 tests, 0.2 s]

(why is the filename simply "interfaces")

sage -t devel/sage/doc/en/reference/probability.rst
    [0 tests, 0.0 s]
sage -t de**********************************************************************
File "programming", line 620, in programming
Failed example:
    for i in range(5):
Exception raised:
    Traceback (most recent call last):
      File "/release/merger/sage-5.1.beta5-12415/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 590, in _run
        compileflags, 1)
      File "<doctest programming[0]>", line 1
        for i in range(5):
                         ^
    SyntaxError: unexpected EOF while parsing
**********************************************************************
1 items had failures:
   1 of 113 in programming
***Test Failed*** 1 failures.
sage -t devel/sage/doc/en/reference/rings_standard.rst
    [0 tests, 0.0 s]
sage -t devel/sage/doc/en/reference/combinat/root_syste**********************************************************************
File "programming", line 578, in programming
Failed example:
    for i in range(5):
Exception raised:
    Traceback (most recent call last):
      File "/release/merger/sage-5.1.beta5-12415/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 590, in _run
        compileflags, 1)
      File "<doctest programming[0]>", line 1
        for i in range(5):
                         ^
    SyntaxError: unexpected EOF while parsing
**********************************************************************
1 items had failures:
   1 of 113 in programming
***Test Failed*** 1 failures.
ms.rst
    [0 tests, 0.0 s]
sage -t devel/sage/sage/databases/odlyzko.py
    [0 tests, 0.0 s]
sag**********************************************************************
File "sage.doctest.reporting", line 93, in sage.doctest.reporting.DocTestReporter.report
Failed example:
    DTR.report(FDS, True, 0, None, "Output so far...")
Expected:
    sage -t .../devel/sage/sage/doctest/reporting.py
        Timed out!
    Output so far...
Got:
    sage -t /release/merger/sage-5.1.beta5-12415/devel/sage/sage/doctest/reporting.py
        Timed out!
    Tests run before process froze:
    Output so far...
**********************************************************************
File "sage.doctest.reporting", line 102, in sage.doctest.reporting.DocTestReporter.report
Failed example:
    DTR.report(FDS, False, 3, None, "Output before bad exit")
Expected:
    sage -t .../devel/sage/sage/doctest/reporting.py
        Bad exit: 3
    Output before bad exit
Got:
    sage -t /release/merger/sage-5.1.beta5-12415/devel/sage/sage/doctest/reporting.py
        Bad exit: 3
    Tests run before process failed:
    Output before bad exit
**********************************************************************
File "sage.doctest.reporting", line 215, in sage.doctest.reporting.DocTestReporter.finalize
Failed example:
    DTR.report(FDS, True, 0, None, "Output so far...")
Expected:
    sage -t .../devel/sage/sage/doctest/reporting.py
        Timed out!
    Output so far...
Got:
    sage -t /release/merger/sage-5.1.beta5-12415/devel/sage/sage/doctest/reporting.py
        Timed out!
    Tests run before process froze:
    Output so far...
**********************************************************************
File "sage.doctest.reporting", line 219, in sage.doctest.reporting.DocTestReporter.finalize
Failed example:
    DTR.report(FDS, False, 3, None, "Output before bad exit")
Expected:
    sage -t .../devel/sage/sage/doctest/reporting.py
        Bad exit: 3
    Output before bad exit
Got:
    sage -t /release/merger/sage-5.1.beta5-12415/devel/sage/sage/doctest/reporting.py
        Bad exit: 3
    Tests run before process failed:
    Output before bad exit
**********************************************************************
2 items had failures:
   2 of  26 in sage.doctest.reporting.DocTestReporter.finalize
   2 of  23 in sage.doctest.reporting.DocTestReporter.report
***Test Failed*** 4 failures.
e -t devel/sage/sage/databases/sloane.py
    [1 tests, 0.1 s]
sage -t devel/sage/sage/ext/cdefs.pxi
    [0 tests, 0.0 s]
sage -t devel/sage/sage/docte**********************************************************************
File "sage.doctest.control", line 252, in sage.doctest.control.DocTestController.add_files
Failed example:
    DC = DocTestController(DD, [])
Exception raised:
    Traceback (most recent call last):
      File "/release/merger/sage-5.1.beta5-12415/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 591, in _run
        self.execute(example, compiled, test.globs)
      File "/release/merger/sage-5.1.beta5-12415/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 873, in execute
        exec compiled in globs
      File "<doctest sage.doctest.control.DocTestController.add_files[2]>", line 1, in <module>
        DC = DocTestController(DD, [])
      File "/release/merger/sage-5.1.beta5-12415/local/lib/python2.7/site-packages/sage/doctest/control.py", line 132, in __init__
        print "Unable to open logfile at %s\nProceeding without logging."%(self.logfile)
    AttributeError: 'DocTestController' object has no attribute 'logfile'
**********************************************************************
sage -t devel/sage/sage/misc/profiler.py
    [0 tests, 0.0 s]
sage -t devel/sage/**********************************************************************
File "sage.matrix.benchmark", line 11, in sage.matrix.benchmark
Failed example:
    print "starting"; b.report([b.det_ZZ], 'Test', systems=['sage'])
Expected:
    starting...
    ======================================================================
              Test
    ======================================================================
    ...
    ======================================================================
Got:
    Linux sage.math.washington.edu 2.6.24-28-server #1 SMP Fri Feb 11 18:08:32 UTC 2011 x86_64 GNU/Linux
    starting
    ======================================================================
              Test
    ======================================================================
    <BLANKLINE>
    <BLANKLINE>
    ----------------------------------------------------------------------
    Dense integer determinant over ZZ.
        Given an n x n matrix A over ZZ with random entries
        between min and max, inclusive, compute det(A).
    <BLANKLINE>
        INPUT:
    <BLANKLINE>
        - ``n`` - matrix dimension (default: ``200``)
        - ``min`` - minimal value for entries of matrix (default: ``1``)
        - ``max`` - maximal value for entries of matrix (default: ``100``)
        - ``system`` - either 'sage' or 'magma' (default: 'sage')
    <BLANKLINE>
        EXAMPLES::
    <BLANKLINE>
            sage: import sage.matrix.benchmark as b
            sage: ts = b.det_ZZ(200)
            sage: tm = b.det_ZZ(200, system='magma')  # optional - magma
               sage
              1.370
    ======================================================================
**********************************************************************
jdemeyer commented 12 years ago
comment:57

Another annoyance, compare in the output:

sage -t devel/sage/sage/rings/polynomial/padics/polynomial_padic_capped_relative_dense.py
sage -t /release/merger/sage-5.1.beta5-12415/devel/sagenb-main/sagenb/interfaces/expect.py

Why a relative path in the first case but a full path in the second case? Also with -a, only absolute paths are shown. I would prefer all paths to be relative to the current directory.

jdemeyer commented 12 years ago
comment:58
jdemeyer@sage:/release/merger/sage-5.1.beta5-12415$ ./sage -tp 1000000000 devel/sage/sage/rings/padics/
Running doctests with ID 2012-06-26-11-58-02-3accd40f.
Doctesting 32 files using 1000000000 threads

The previous doctester would reduce the number of threads to be at most the number of files to test.

jdemeyer commented 12 years ago
comment:59

A very useful feature from the previous doctester which is missing here is that --verbose actually shows the output as the test is running. So when it gets stuck somewhere, you actually see that it gets stuck. Currently, the whole test result is simply dumped at the end.

Also: previously, line numbers were shown with --verbose (albeit in a very bad way).

One more feature-request: with --verbose, show the time taken for each single test, i.e. I would like

Trying (line 242):
    EllipticCurve('900d1').integral_points()
Expecting:
    [(-11 : 27 : 1), (-4 : 34 : 1), (4 : 18 : 1), (16 : 54 : 1)]
ok [0.10s]
Trying (line 243):
    E=EllipticCurve([-879984,319138704])
Expecting nothing
ok [1.92s]
Trying (line 255):
    P1=E.point((540,1188)); P2=E.point((576,1836))
Expecting nothing
ok [4.59s]

It's totally okay for me to disallow --verbose in parallel tests.

jdemeyer commented 12 years ago
comment:60

What does the --serial option do and how is it different from -p 1?

jdemeyer commented 12 years ago
comment:61

I would expect sage -t -f not to need any filename arguments (similarly to -a), just run all the failed tests.

However:

jdemeyer@sage:/release/merger/sage-5.1.beta5-12415$ ./sage -t -f
Usage: sage -t [options] filenames
jdemeyer commented 12 years ago
comment:62

Perhaps add a -T option as a synonym for --timeout?

jdemeyer commented 12 years ago
comment:63

I'm collecting some "special cases" for doctests in http://boxen.math.washington.edu/home/jdemeyer/doctest/ (maybe these should eventually become "meta-doctests").

The doctesting framework seems to hang on

jdemeyer@sage:/release/merger/sage-5.1.beta5-12415$ ./sage -t /home/jdemeyer/doctest/sigdie.py
Running doctests with ID 2012-06-26-14-12-55-fbefc7c4.
Doctesting 1 file.

While the following gives an error inside the doctesting framework (this also used to fail before):

jdemeyer@sage:/release/merger/sage-5.1.beta5-12415$ ./sage -t /home/jdemeyer/doctest/keyboardinterrupt.py
Running doctests with ID 2012-06-26-14-12-38-b70e54c4.
Doctesting 1 file.
sage -t /home/jdemeyer/doctest/keyboardinterrupt.py
Traceback (most recent call last):
  File "/release/merger/sage-5.1.beta5-12415/local/lib/python2.7/site-packages/sage/doctest/reporting.py", line 146, in report
    ntests, runner = results
TypeError: 'NoneType' object is not iterable

------------------------------------------------------------------------
Doctests interrupted: 0/1 files tested
------------------------------------------------------------------------
Total time for all tests: 1.0 seconds
    cpu time: 0.0 seconds
    cumulative wall time: 0.0 seconds

The following should report an error but doesn't:

jdemeyer@sage:/release/merger/sage-5.1.beta5-12415$ ./sage -t /home/jdemeyer/doctest/sig_on.py
Running doctests with ID 2012-06-26-14-14-08-06a01dbf.
Doctesting 1 file.
sage -t /home/jdemeyer/doctest/sig_on.py
    [1 tests, 1.1 s]
------------------------------------------------------------------------
All tests passed!
------------------------------------------------------------------------
Total time for all tests: 2.1 seconds
    cpu time: 0.1 seconds
    cumulative wall time: 1.1 seconds
roed314 commented 12 years ago
comment:64

I'm working on fixing these. Thanks for the feedback!

roed314 commented 12 years ago
comment:65

Replying to @jdemeyer:

What does the --serial option do and how is it different from -p 1?

The -p 1 has one worker: there's another thread which collates and prints answers. -p 1 is the default.

roed314 commented 12 years ago
comment:66

Replying to @jdemeyer:

Perhaps add a -T option as a synonym for --timeout?

Done. Are there other options that should have a short form? Maybe -L for --long? I'd like to avoid too much mixing of upper and lower case....

roed314 commented 12 years ago
comment:67

There doesn't seem to be an objection to diverting stderr, so I'm going to not worry about keeping changes resulting from that decision separate (I don't want to update that patch since it requires rebuilding a lot of files).

roed314 commented 12 years ago
comment:68

Replying to @jdemeyer:

The following should report an error but doesn't:

jdemeyer@sage:/release/merger/sage-5.1.beta5-12415$ ./sage -t /home/jdemeyer/doctest/sig_on.py
Running doctests with ID 2012-06-26-14-14-08-06a01dbf.
Doctesting 1 file.
sage -t /home/jdemeyer/doctest/sig_on.py
    [1 tests, 1.1 s]
------------------------------------------------------------------------
All tests passed!
------------------------------------------------------------------------
Total time for all tests: 2.1 seconds
    cpu time: 0.1 seconds
    cumulative wall time: 1.1 seconds

If I do cython('sig_on()') at the command line nothing happens. Is it supposed to crash Sage?

roed314 commented 12 years ago
comment:69

Replying to @roed314:

Replying to @jdemeyer:

The following should report an error but doesn't:

jdemeyer@sage:/release/merger/sage-5.1.beta5-12415$ ./sage -t /home/jdemeyer/doctest/sig_on.py
Running doctests with ID 2012-06-26-14-14-08-06a01dbf.
Doctesting 1 file.
sage -t /home/jdemeyer/doctest/sig_on.py
    [1 tests, 1.1 s]
------------------------------------------------------------------------
All tests passed!
------------------------------------------------------------------------
Total time for all tests: 2.1 seconds
    cpu time: 0.1 seconds
    cumulative wall time: 1.1 seconds

If I do cython('sig_on()') at the command line nothing happens. Is it supposed to crash Sage?

I can even run sage commands afterward, like factor(2^199-1). But if I hit Ctl-C, I get a segfault (and it screws up my terminal).

roed314 commented 12 years ago
comment:70

New version posted. sage -t --long -a passes all tests except the one related to #13145.

/home/jdemeyer/doctest/keyboardinterrupt.py passes if you remove the trailing colon. I've updated the framework to recover a bit more gracefully if a doctest raises an unexpected KeyboardInterrupt, but it behaves a bit strangely since it's difficult to distinguish between an actual interrupt (in which case the testing should terminate) and one raised by a doctest (in which case tests of other files could continue). I think the new behavior is actually fairly reasonable: try it on your file without removing the colon.

roed314 commented 12 years ago
comment:71

Replying to @jdemeyer:

The doctesting framework seems to hang on

jdemeyer@sage:/release/merger/sage-5.1.beta5-12415$ ./sage -t /home/jdemeyer/doctest/sigdie.py
Running doctests with ID 2012-06-26-14-12-55-fbefc7c4.
Doctesting 1 file.

Is this behavior acceptable? Eventually the test will time out (you can try running sage -t -T 5 /home/jdemeyer/doctest/sigdie.py) and the ending summary will be printed with a time out.

jdemeyer commented 12 years ago
comment:72

Replying to @roed314:

it's difficult to distinguish between an actual interrupt (in which case the testing should terminate) and one raised by a doctest (in which case tests of other files could continue).

A "real" interrupt would interrupt also the master process, a fake doctest-interrupt only the child process. That's an easy way to distinguish.

jdemeyer commented 12 years ago
comment:73

Replying to @roed314:

Replying to @jdemeyer:

The doctesting framework seems to hang on

jdemeyer@sage:/release/merger/sage-5.1.beta5-12415$ ./sage -t /home/jdemeyer/doctest/sigdie.py
Running doctests with ID 2012-06-26-14-12-55-fbefc7c4.
Doctesting 1 file.

Is this behavior acceptable?

Not really. One should get a message that Sage crashed (compare with the old doctesting framework).

jdemeyer commented 12 years ago
comment:74

Replying to @roed314:

If I do cython('sig_on()') at the command line nothing happens. Is it supposed to crash Sage?

Let me elaborate: #10030 added a check to the doctesting code which checks

sage: sig_on_count()
0

after each doctest (i.e. it appends this to every doctest), to ensure there are no unbalanced sig_on()s. Could you add this also to your doctesting framework?

roed314 commented 12 years ago
comment:75

Replying to @jdemeyer:

Replying to @roed314:

it's difficult to distinguish between an actual interrupt (in which case the testing should terminate) and one raised by a doctest (in which case tests of other files could continue).

A "real" interrupt would interrupt also the master process, a fake doctest-interrupt only the child process. That's an easy way to distinguish.

Sure, and for reporting what happened to the user this is fine. But from the child process' perspective, an unexpected KeyboardInterrupt has been raised and it should stop testing that file. The parent process can continue the doctests though....

roed314 commented 12 years ago
comment:76

Replying to @jdemeyer:

Replying to @roed314:

Replying to @jdemeyer:

The doctesting framework seems to hang on

jdemeyer@sage:/release/merger/sage-5.1.beta5-12415$ ./sage -t /home/jdemeyer/doctest/sigdie.py
Running doctests with ID 2012-06-26-14-12-55-fbefc7c4.
Doctesting 1 file.

Is this behavior acceptable?

Not really. One should get a message that Sage crashed (compare with the old doctesting framework).

Okay, I will look into it. I assume that the underlying process has terminated and I should be able to get at the exit code somehow.

roed314 commented 12 years ago
comment:77

Replying to @jdemeyer:

Replying to @roed314:

If I do cython('sig_on()') at the command line nothing happens. Is it supposed to crash Sage?

Let me elaborate: #10030 added a check to the doctesting code which checks

sage: sig_on_count()
0

after each doctest (i.e. it appends this to every doctest), to ensure there are no unbalanced sig_on()s. Could you add this also to your doctesting framework?

This shouldn't be too hard.

roed314 commented 12 years ago
comment:78

Replying to @roed314:

Alright, I'm going to start running tests. There are still some remaining issues, but the patches are in a state where someone else might want to look at them. :-)

Remaining problems:

  • 13145 needs work, generating some doctest failures related to deallocating multivariate polynomials

  • hg_sage seems to be broken, causing the --new option to fail
  • valgrinding sage -t doesn't work; I'm not sure whether it did before
  • The new code doesn't support xml output

I'll see if I find more when running sage -t --all.

I'm going to sleep; let me know which of these need to be fixed in order for this ticket to get a positive review. I will also keep working on the tests in /home/jdemeyer/doctest/. And of course if you find more problems or have other feature requests, keep those coming too!

jdemeyer commented 12 years ago
comment:79

Replying to @roed314:

Sure, and for reporting what happened to the user this is fine. But from the child process' perspective, an unexpected KeyboardInterrupt has been raised and it should stop testing that file.

Okay, I see what you mean. I would propose not to special-case KeyboardInterrupt at all in the child process. When the parent process gets a KeyboardInterrupt, kill all child processes (without showing their output) and exit.

(note: since the old doctesting framework didn't do this either, I don't consider this a prerequisite for positive review, but a nice feature to have.)

jdemeyer commented 12 years ago
comment:80

Replying to @roed314:

Replying to @jdemeyer:

What does the --serial option do and how is it different from -p 1?

The -p 1 has one worker: there's another thread which collates and prints answers. -p 1 is the default.

I suppose you mean "process", not thread? Still, I don't see the point of --serial.

roed314 commented 12 years ago
comment:81

Replying to @jdemeyer:

Replying to @roed314:

Replying to @jdemeyer:

What does the --serial option do and how is it different from -p 1?

The -p 1 has one worker: there's another thread which collates and prints answers. -p 1 is the default.

I suppose you mean "process", not thread? Still, I don't see the point of --serial.

The main point of --serial is that it allows you to debug problems in the doctesting framework without worrying about multiprocessing. :-)

I'm fine making it less prominent (having it not show up in the help display for sage -t for example, and not using it for --verbose mode). But I think it should stick around since it currently exists, and it makes some kinds of bugs much easier to track down.

roed314 commented 12 years ago
comment:82

Replying to @roed314:

Replying to @jdemeyer:

Replying to @roed314:

Replying to @jdemeyer:

What does the --serial option do and how is it different from -p 1?

The -p 1 has one worker: there's another thread which collates and prints answers. -p 1 is the default.

I suppose you mean "process", not thread? Still, I don't see the point of --serial.

The main point of --serial is that it allows you to debug problems in the doctesting framework without worrying about multiprocessing. :-)

I'm fine making it less prominent (having it not show up in the help display for sage -t for example, and not using it for --verbose mode). But I think it should stick around since it currently exists, and it makes some kinds of bugs much easier to track down.

I'm also using it for --gdb since it makes the tracebacks simpler, though I'm not sure that's necessary.

Do you ever use sage -t with valgrind? I'm not sure exactly what's wrong with sage -t --valgrind <FILENAME>.

roed314 commented 12 years ago
comment:83

Replying to @jdemeyer:

Replying to @roed314:

Sure, and for reporting what happened to the user this is fine. But from the child process' perspective, an unexpected KeyboardInterrupt has been raised and it should stop testing that file.

Okay, I see what you mean. I would propose not to special-case KeyboardInterrupt at all in the child process. When the parent process gets a KeyboardInterrupt, kill all child processes (without showing their output) and exit.

(note: since the old doctesting framework didn't do this either, I don't consider this a prerequisite for positive review, but a nice feature to have.)

I just tried changing the interrupt handling so that KeyboardInterrupt is handled in the parent process and it doesn't work as well: there's extraneous printing happening when the doctest framework thinks it's just printing an unexpected exception, and I don't see a good way to get rid of it.

I think the current handling of KeyboardInterrupts is fine.

roed314 commented 12 years ago
comment:84

I'm going to be offline until Tuesday. More feedback while I'm gone would be great. :-)

jhpalmieri commented 12 years ago
comment:85

As far as I understand it, this new framework should allow for the same commands (e.g., sage -tp 6 --long devel/sage/sage/matrix/). In addition, though, there are new command-line options to sage-runtest, which should available through sage -t ..., right? Which of these new options should be documented in the help messages usage and/or usage_advanced at the top of spkg/bin/sage? Which should be documented in the developer's guide?

roed314 commented 12 years ago
comment:86

I'm back, and I've been working on a patch to the developer's guide. I'll post something soon, and more comments on which options will be shown in usage.

jdemeyer commented 12 years ago
comment:87

I'm going to build and test Sage again with these patches. Stay tuned...

jdemeyer commented 12 years ago

Description changed:

--- 
+++ 
@@ -12,7 +12,7 @@

 and does most of the above (plus more which has been moved to followup tickets #12720 and #12722)

-**Apply** [attachment: 12415_stderr_vs_51b5.patch,](https://github.com/sagemath/sage/files/ticket12415/c7a4af89fe800f7cdb7c2cc42809d45b.gz) [attachment: 12415_doctest_fixes.patch,](https://github.com/sagemath/sage/files/ticket12415/5e0b23d995017a7b6b3d2800c98454e1.gz) [attachment: 12415_framework.patch](https://github.com/sagemath/sage-prod/files/10654698/12415_framework.patch.gz)
+**Apply** [attachment: 12415_stderr_vs_51b5.patch](https://github.com/sagemath/sage/files/ticket12415/12415_stderr_vs_51b5.patch.gz), [attachment: 12415_doctest_fixes.patch](https://github.com/sagemath/sage-prod/files/10654706/12415_doctest_fixes.patch.gz), [attachment: 12415_framework.patch](https://github.com/sagemath/sage-prod/files/10654698/12415_framework.patch.gz), [attachment: 12415_doc.patch](https://github.com/sagemath/sage-prod/files/10654702/12415_doc.patch.gz)

 **Apply** [attachment: 12415_script.patch](https://github.com/sagemath/sage-prod/files/10654697/12415_script.patch.gz) to the scripts repo
jdemeyer commented 12 years ago

Changed dependencies from #13147,#13146, #13145, #12723, #12392, #12393, #12395, #12396, #12397, #12381, #12382, #12383, #12384 to #13147,#13146, #13145, #12723, #12392, #12393, #12395, #12396, #12397, #12381, #12382, #12383, #12384, #11871

jdemeyer commented 12 years ago
comment:89

Needs to be rebased to #11871.

roed314 commented 12 years ago
comment:90

Replying to @jdemeyer:

Needs to be rebased to #11871.

Done. I will make sure the suggestions in comment 56 are addressed.

roed314 commented 12 years ago
comment:91

The change to 12415_script.patch fixes a problem with interrupting doctests on sage.math.

jdemeyer commented 12 years ago
comment:92

When doing make ptest and make ptestlong, all doctests are run but there is no summary at the end. The processes end with exit code 132 (which usually means an Illegal Instruction but probably not here).

The following are the last lines output by make ptest and make ptestlong:

sage -t devel/sage/doc/en/thematic_tutorials/lie/weyl_groups.rst
    [44 tests, 4.6 s]
sage -t devel/sage/doc/fr/tutorial/tour_plotting.rst
    [35 tests, 9.2 s]
sage -t devel/sage/doc/de/tutorial/tour_plotting.rst
    [51 tests, 10.7 s]
sage -t devel/sage/doc/en/tutorial/tour_plotting.rst
    [51 tests, 10.9 s]
sage -t devel/sage/doc/ru/tutorial/tour_plotting.rst
    [46 tests, 10.9 s]

and

sage -t --long devel/sage/sage/combinat/crystals/__init__.py
    [0 tests, 0.0 s]
sage -t --long devel/sage/sage/gsl/callback.pyx
    [0 tests, 0.0 s]
sage -t --long devel/sage/sage/ext/python_module.pxi
    [0 tests, 0.0 s]
sage -t --long devel/sage/sage/groups/perm_gps/all.py
    [0 tests, 0.0 s]
sage -t --long devel/sage/doc/en/bordeaux_2008/modular_forms_and_hecke_operators.rst
    [0 tests, 0.0 s]
jdemeyer commented 12 years ago
comment:93

Then I get the following make ptestlong failures:

Running doctests with ID 2012-07-08-23-57-28-0d45df42.
Doctesting the Sage notebook.
Sorting sources by runtime so that slower doctests are run first....
Doctesting 2067 files using 6 threads.
Usage: sage -docbuild [OPTIONS] DOCUMENT (FORMAT | COMMAND)

sage-runtests: error: no such option: -p
sage -t --long devel/sage/doc/common/builder.py
    Bad exit: 2
********************************************************************************
Tests run before process failed:

********************************************************************************
sage -t --long devel/sage/sage/tests/interrupt.pyx
    [80 tests, 15.0 s]
sage -t --long devel/sage/sage/matrix/matrix2.pyx
    [1980 tests, 24.9 s]
**********************************************************************
File "devel/sage/sage/tests/cmdline.py", line 272, in sage.tests.cmdline.test_executable
Failed example:
    out.find("All tests passed!") >= 0
Expected:
    True
Got:
    False
**********************************************************************
File "devel/sage/sage/tests/cmdline.py", line 277, in sage.tests.cmdline.test_executable
Failed example:
    out.find("All tests passed!") >= 0
Expected:
    True
Got:
    False
**********************************************************************
1 items had failures:
   2 of 185 in sage.tests.cmdline.test_executable
***Test Failed*** 2 failures.
sage -t --long devel/sage/sage/tests/cmdline.py
    [184 tests, 23.0 s]
**********************************************************************
File "devel/sage/sage/doctest/control.py", line 263, in sage.doctest.control.DocTestController.add_files
Failed example:
    DC = DocTestController(DD, [])
Exception raised:
    Traceback (most recent call last):
      File "/release/merger/sage-5.1.rc0-12415/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 590, in _run
        self.execute(example, compiled, test.globs)
      File "/release/merger/sage-5.1.rc0-12415/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 881, in execute
        exec compiled in globs
      File "<doctest sage.doctest.control.DocTestController.add_files[2]>", line 1, in <module>
        DC = DocTestController(DD, [])
      File "/release/merger/sage-5.1.rc0-12415/local/lib/python2.7/site-packages/sage/doctest/control.py", line 142, in __init__
        print "Unable to open logfile at %s\nProceeding without logging."%(self.logfile)
    AttributeError: 'DocTestController' object has no attribute 'logfile'
**********************************************************************
roed314 commented 12 years ago
comment:94

I thought I'd solved the sporadic problem in sage.tests.interrupt by increasing the delays in test_sig_block_outside_sig_on: after the change I ran sage -tp --sagenb --long devel/sage/doc/common devel/sage/doc/[a-z][a-z] devel/sage/sage 10 times with no failures (where before there had been 5 failures in interrupt.pyx). But I just got another one. Maybe it has to do with the load on sage.math? There are more processes running now than when I ran the test earlier.

Jeroen, would you be willing to take a look at it since you wrote the tests in that file?

All the other tests pass for me, with the exception of startup.py.

jdemeyer commented 12 years ago

Changed dependencies from #13147,#13146, #13145, #12723, #12392, #12393, #12395, #12396, #12397, #12381, #12382, #12383, #12384, #11871 to #13147,#13146, #13145, #12723, #12392, #12393, #12395, #12396, #12397, #12381, #12382, #12383, #12384, #11871, #13195

jdemeyer commented 12 years ago
comment:96

Rebased the scripts patch to sage-5.1.

jdemeyer commented 12 years ago
comment:97

The sagenb patch doesn't apply to the new notebook (#11080).

jdemeyer commented 12 years ago
comment:98

Replying to @roed314:

I thought I'd solved the sporadic problem in sage.tests.interrupt by increasing the delays in test_sig_block_outside_sig_on

The tests should be designed in such a way that they work regardless of the delay.

roed314 commented 12 years ago
comment:99

Replying to @jdemeyer:

Replying to @roed314:

I thought I'd solved the sporadic problem in sage.tests.interrupt by increasing the delays in test_sig_block_outside_sig_on

The tests should be designed in such a way that they work regardless of the delay.

Agreed, but I'm at a loss for what's causing the problem, and it's difficult to debug since it's sporadic and a segfault.

jdemeyer commented 12 years ago
comment:100

I have added some changes to local/bin/.hgignore to the scripts patch. I discovered there was a problem when I put the file sage-runtests in local/bin without adding it using hg. Surprisingly, hg status didn't show sage-runtests.

roed314 commented 12 years ago

Description changed:

--- 
+++ 
@@ -16,6 +16,6 @@

 **Apply** [attachment: 12415_script.patch](https://github.com/sagemath/sage-prod/files/10654697/12415_script.patch.gz) to the scripts repo

-**Apply** [attachment: 12415_sagenb_fixes.patch](https://github.com/sagemath/sage/files/ticket12415/12415_sagenb_fixes.patch.gz) to the sagenb repo (fixes doctest errors)
+**Apply** [attachment: 12415_sagenb_fixes_vs_11080.patch](https://github.com/sagemath/sage/files/ticket12415/12415_sagenb_fixes_vs_11080.patch.gz) to the sagenb repo (fixes doctest errors)

 **Apply** [attachment: 12415_spkg_bin_sage.patch](https://github.com/sagemath/sage-prod/files/10654696/12415_spkg_bin_sage.patch.gz) to the root repo