Update doctesting framework

robertwb commented 12 years ago

There are several improvements that would be good to make, including but not limited to:

Forking rather than starting up a new Sage each time.
Customizable comparison rather than mutating doctests into new doctests (e.g. followup to #10952).
Eliminate need to write temporary files for every doctested file.
Constant headaches caused by the difference in doctesting library vs. non-library files.
The ability to drop into a debugger if a doctest raises an exception.

Robert's code is at

http://code.google.com/p/sagemath-timer/

Apply attachment: 12415_framework.patch, attachment: 12415_doctest_fixes.patch, attachment: 12415_doc.patch, attachment: 12415_review.patch, attachment: 12415_test.patch, attachment: 12415_review_review.patch, attachment: 12415_review3.patch, attachment: 12415_manifest.patch, attachment: 12415_rebase_58.patch

Apply attachment: 12415_script.patch and attachment: 12415_script_review.patch to the scripts repo

Apply attachment: 12415_spkg_bin_sage.patch to the root repo

For follow-up or other doctest-related tickets see #11337.

Depends on #13147 Depends on #13146 Depends on #13145 Depends on #12723 Depends on #12392 Depends on #12393 Depends on #12395 Depends on #12396 Depends on #12397 Depends on #12381 Depends on #12382 Depends on #12383 Depends on #12384 Depends on #11871 Depends on #13195 Depends on #13121 Depends on #13748 Depends on #13899 Depends on #12719 Depends on #5155 Depends on #14070 Depends on #14079 Depends on #14150 Depends on #14158 Depends on #14182 Depends on #14184 Depends on #14054 Depends on #14063 Depends on #13605 Depends on #14111 Depends on #14254 Depends on #14242 Depends on #14253

CC: @kini @ohanar @jhpalmieri

Component: doctest framework

Author: David Roe, Robert Bradshaw, Jeroen Demeyer

Reviewer: Jeroen Demeyer, David Roe

Merged: sage-5.9.beta0

Issue created by migration from https://trac.sagemath.org/ticket/12415

robertwb commented 12 years ago

Description changed:

--- 
+++ 
@@ -1,8 +1,8 @@
 There are several improvements that would be good to make, including but not limited to:

 1. Forking rather than starting up a new Sage each time.
-2. Customizeable comparison rather than mutating doctests into new doctests. 
-3. Avoiding the use of writing out temporary files.
+2. Customizeable comparison rather than mutating doctests into new doctests (e.g. followup to #10952). 
+3. Eliminate need to write temporary files for every doctested file.
 4. Provide mechanisms for recording and comparing timings. 
 5. Constant headaches caused by the difference in doctesting library vs. non-library files.

robertwb commented 12 years ago

Description changed:

--- 
+++ 
@@ -1,9 +1,18 @@
 There are several improvements that would be good to make, including but not limited to:

 1. Forking rather than starting up a new Sage each time.
+
 2. Customizeable comparison rather than mutating doctests into new doctests (e.g. followup to #10952). 
+
 3. Eliminate need to write temporary files for every doctested file.
+
 4. Provide mechanisms for recording and comparing timings. 
+
 5. Constant headaches caused by the difference in doctesting library vs. non-library files. 

+
+Code at 
+
 http://code.google.com/p/sagemath-timer/
+
+does most of the above, should be added to the sage library.

roed314 commented 12 years ago

Description changed:

--- 
+++ 
@@ -10,6 +10,7 @@

 5. Constant headaches caused by the difference in doctesting library vs. non-library files. 

+6. The ability to drop into a debugger if a doctest raises an exception.

 Code at

roed314 commented 12 years ago

comment:3

I'm working on this. E-mail me if you want to collaborate.

roed314 commented 12 years ago

Description changed:

--- 
+++ 
@@ -12,6 +12,8 @@

 6. The ability to drop into a debugger if a doctest raises an exception.

+7. The ability to run different doctests in the same file in parallel.
+
 Code at 

 http://code.google.com/p/sagemath-timer/

roed314 commented 12 years ago

Description changed:

--- 
+++ 
@@ -1,21 +1,13 @@
 There are several improvements that would be good to make, including but not limited to:

 1. Forking rather than starting up a new Sage each time.
+2. Customizable comparison rather than mutating doctests into new doctests (e.g. followup to #10952). 
+3. Eliminate need to write temporary files for every doctested file.
+4. Constant headaches caused by the difference in doctesting library vs. non-library files. 
+5. The ability to drop into a debugger if a doctest raises an exception.

-2. Customizeable comparison rather than mutating doctests into new doctests (e.g. followup to #10952). 
-
-3. Eliminate need to write temporary files for every doctested file.
-
-4. Provide mechanisms for recording and comparing timings. 
-
-5. Constant headaches caused by the difference in doctesting library vs. non-library files. 
-
-6. The ability to drop into a debugger if a doctest raises an exception.
-
-7. The ability to run different doctests in the same file in parallel.
-
-Code at 
+Robert's code is at

 http://code.google.com/p/sagemath-timer/

-does most of the above, should be added to the sage library.
+and does most of the above (plus more which has been moved to followup tickets #12720 and #12722)

roed314 commented 12 years ago

Dependencies: #12392, #12393, #12395, #12396, #12397, #12381, #12382, #12383, #12384

roed314 commented 12 years ago

Changed dependencies from #12392, #12393, #12395, #12396, #12397, #12381, #12382, #12383, #12384 to #12723, #12392, #12393, #12395, #12396, #12397, #12381, #12382, #12383, #12384

robertwb commented 12 years ago

comment:9

12415_2.patch seems to contain a lot of tempfile changes, could you split out the doctest changes into a separate patch (or are these affecting doctests?)

ohanar commented 12 years ago

comment:11

apparently even I forget my trac username is not what it should be

roed314 commented 12 years ago

Changed dependencies from #12723, #12392, #12393, #12395, #12396, #12397, #12381, #12382, #12383, #12384 to #13145, #12723, #12392, #12393, #12395, #12396, #12397, #12381, #12382, #12383, #12384

roed314 commented 12 years ago

Changed dependencies from #13145, #12723, #12392, #12393, #12395, #12396, #12397, #12381, #12382, #12383, #12384 to #13146, #13145, #12723, #12392, #12393, #12395, #12396, #12397, #12381, #12382, #12383, #12384

roed314 commented 12 years ago

Changed dependencies from #13146, #13145, #12723, #12392, #12393, #12395, #12396, #12397, #12381, #12382, #12383, #12384 to #13147,#13146, #13145, #12723, #12392, #12393, #12395, #12396, #12397, #12381, #12382, #12383, #12384

roed314 commented 12 years ago

comment:15

I've moved the patches dealing with temporary files to #13147.

roed314 commented 12 years ago

Description changed:

--- 
+++ 
@@ -11,3 +11,5 @@
 http://code.google.com/p/sagemath-timer/

 and does most of the above (plus more which has been moved to followup tickets #12720 and #12722)
+
+Apply [attachment: 12415_stderr.patch,](https://github.com/sagemath/sage/files/ticket12415/3df95252d095e0ff2177ca171ef9634d.gz) [attachment: 12415_doctest_fixes.patch,](https://github.com/sagemath/sage/files/ticket12415/5e0b23d995017a7b6b3d2800c98454e1.gz) [attachment: 12415_framework.patch](https://github.com/sagemath/sage-prod/files/10654698/12415_framework.patch.gz)

roed314 commented 12 years ago

Description changed:

--- 
+++ 
@@ -12,4 +12,4 @@

 and does most of the above (plus more which has been moved to followup tickets #12720 and #12722)

-Apply [attachment: 12415_stderr.patch,](https://github.com/sagemath/sage/files/ticket12415/3df95252d095e0ff2177ca171ef9634d.gz) [attachment: 12415_doctest_fixes.patch,](https://github.com/sagemath/sage/files/ticket12415/5e0b23d995017a7b6b3d2800c98454e1.gz) [attachment: 12415_framework.patch](https://github.com/sagemath/sage-prod/files/10654698/12415_framework.patch.gz)
+**Apply** [attachment: 12415_stderr.patch,](https://github.com/sagemath/sage/files/ticket12415/3df95252d095e0ff2177ca171ef9634d.gz) [attachment: 12415_doctest_fixes.patch,](https://github.com/sagemath/sage/files/ticket12415/5e0b23d995017a7b6b3d2800c98454e1.gz) [attachment: 12415_framework.patch](https://github.com/sagemath/sage-prod/files/10654698/12415_framework.patch.gz)

roed314 commented 12 years ago

comment:19

Ready for review!

There are still a few known issues:

valgrinding under sage -t doesn't work. I haven't used valgrind much and don't know how to resolve this.
The fix at #13147 probably needs work: I'm getting a failure in sage.rings.polynomial.multi_polynomial_libsingular to do with garbage collection. I'm working on it.
I'm in the process of updating the developer guide.

But I wanted to get some other people involved in reviewing.

jhpalmieri commented 12 years ago

comment:20

This may be a silly question, but how do I use the new framework? Don't we need changes to the scripts repo, too? (If I just run sage -t FILES, I don't see the various messages, like Doctesting ..., that should be printed by run_doctests are not printed, so sage -t is not yet using the new framework.)

One of the patches also didn't apply cleanly to 5.1.beta5:

hg qimport -P ~/Downloads/12415_stderr.patch 
adding 12415_stderr.patch to series file
applying 12415_stderr.patch
patching file sage/lfunctions/lcalc.py
Hunk #1 FAILED at 226
1 out of 2 hunks FAILED -- saving rejects to file sage/lfunctions/lcalc.py.rej
patch failed, unable to continue (try -v)
patch failed, rejects left in working dir
errors during apply, please fix and refresh 12415_stderr.patch

roed314 commented 12 years ago

comment:21

Did you apply the new dependencies (#13145, #13146, #13147)? There are some more patches to other repositories: I'll update the ticket description.

roed314 commented 12 years ago

Description changed:

--- 
+++ 
@@ -12,4 +12,10 @@

 and does most of the above (plus more which has been moved to followup tickets #12720 and #12722)

-**Apply** [attachment: 12415_stderr.patch,](https://github.com/sagemath/sage/files/ticket12415/3df95252d095e0ff2177ca171ef9634d.gz) [attachment: 12415_doctest_fixes.patch,](https://github.com/sagemath/sage/files/ticket12415/5e0b23d995017a7b6b3d2800c98454e1.gz) [attachment: 12415_framework.patch](https://github.com/sagemath/sage-prod/files/10654698/12415_framework.patch.gz)
+**Apply** [attachment: 12415_stderr_vs_51b5.patch,](https://github.com/sagemath/sage/files/ticket12415/c7a4af89fe800f7cdb7c2cc42809d45b.gz) [attachment: 12415_doctest_fixes.patch,](https://github.com/sagemath/sage/files/ticket12415/5e0b23d995017a7b6b3d2800c98454e1.gz) [attachment: 12415_framework.patch](https://github.com/sagemath/sage-prod/files/10654698/12415_framework.patch.gz)
+
+**Apply** [attachment: 12415_script.patch](https://github.com/sagemath/sage-prod/files/10654697/12415_script.patch.gz) to the scripts repo
+
+**Apply** [attachment: 12415_sagenb_fixes.patch](https://github.com/sagemath/sage/files/ticket12415/12415_sagenb_fixes.patch.gz) to the sagenb repo (fixes doctest errors)
+
+**Apply** [attachment: 12415_spkg_bin_sage.patch](https://github.com/sagemath/sage-prod/files/10654696/12415_spkg_bin_sage.patch.gz) to `$SAGE_ROOT/spkg/bin/sage` (there's no repo here so you need to use patch)

roed314 commented 12 years ago

comment:23

The description is updated: let me know if you have any more problems. I think you need to apply 12415_stderr_vs_51b5.patch instead of 12415_stderr.patch.

jhpalmieri commented 12 years ago

comment:24

Regarding attachment: 12415_spkg_bin_sage.patch, the file in question should be part of the root repo.

jhpalmieri commented 12 years ago

comment:25

One regression: with the old set-up, running sage -tp FILES (with no argument for tp) would choose the number of threads automatically. With the new set-up, this command gives an error, because FILES is not an integer. Therefore I think that, for example, make ptestlong will fail. (The file sage-runtests says that the default value for nthreads is 1, but I don't think optparse is dealing with the missing argument properly here. Also, passing the argument 0 causes doctests to never happen: maybe it's using 0 threads instead of minimum(8, cpu_count()).

jhpalmieri commented 12 years ago

comment:26

I would suggest the following two changes: for the scripts repo:

diff --git a/sage-runtests b/sage-runtests
--- a/sage-runtests
+++ b/sage-runtests
@@ -5,7 +5,7 @@ import optparse, os, sys
 if __name__ == "__main__":
     parser = optparse.OptionParser()

-    parser.add_option("-p", "--nthreads", type=int, default=1, metavar="N", help="tests in parallel using N threads with 0 interpreted as minimum(8, cpu_count())")
+    parser.add_option("-p", "--nthreads", type="string", default=1, metavar="N", help="tests in parallel using N threads with 0 interpreted as minimum(8, cpu_count())")
     parser.add_option("--serial", action="store_true", default=False, help="run tests in a single process in series")
     parser.add_option("--timeout", type=int, default=-1, help="timeout (in seconds) for doctesting one file")
     parser.add_option("-a", "--all", action="store_true", default=False, help="test all files in the Sage library")
@@ -49,7 +49,20 @@ if __name__ == "__main__":
     parser.add_option("--stats_path", "--stats-path", default=os.path.join(os.path.expanduser("~/.sage/timings2.json")), \
                           help="path to a json dictionary for the latest run storing a timing for each file")

+    options, args = parser.parse_args()
+    try:
+        options.nthreads = int(options.nthreads)
+    except ValueError:
+        args.insert(0, options.nthreads.strip())
+        options.nthreads = 0
+
+    if options.nthreads == 0:
+        try:
+            options.nthreads = int(os.environ['SAGE_NUM_THREADS_PARALLEL'])
+        except KeyError:
+            options.nthreads = 1
+
     from sage.doctest.control import DocTestController
-    DC = DocTestController(*parser.parse_args())
+    DC = DocTestController(options, args)
     err = DC.run()
     sys.exit(err)

and for the Sage library:

diff --git a/sage/doctest/control.py b/sage/doctest/control.py
--- a/sage/doctest/control.py
+++ b/sage/doctest/control.py
@@ -433,8 +433,9 @@ class DocTestController(SageObject):
             else:
                 nother += 1
         if nfiles + ndbsources + nother:
-            self.log("Doctesting %s."%(", ".join((["%s file%s"%(nfiles, "s" if nfiles > 1 else "")] if nfiles else []) +
-                                                 (["%s other sources"%nother] if nother else []))))
+            self.log("Doctesting %s"%(", ".join((["%s file%s"%(nfiles, "s" if nfiles > 1 else "")] if nfiles else []) +
+                                                 (["%s other sources"%nother] if nother else [])))
+                + " using %s threads."%self.options.nthreads if self.options.nthreads > 1 else ".")
             self.reporter = DocTestReporter(self)
             self.dispatcher = DocTestDispatcher(self)
             try:

Or something like that. I guess you should also print the number of threads when you print "Doctesting entire Sage library.", etc.

roed314 commented 12 years ago

comment:27

I've updated the option parsing to allow -p to take no options (though I used a callback so that the -p doesn't have to be last). I changed 12415_spkg_bin_sage.patch to be an actual patch against the root repository. I also made a small change to the debug functionality (printing the doctest that failed before dropping into the debugger).

Since I'm modifying four repositories, I'll just update the patches rather than creating lots of referee patches (unless the referee objects!).

roed314 commented 12 years ago

Description changed:

--- 
+++ 
@@ -18,4 +18,4 @@

 **Apply** [attachment: 12415_sagenb_fixes.patch](https://github.com/sagemath/sage/files/ticket12415/12415_sagenb_fixes.patch.gz) to the sagenb repo (fixes doctest errors)

-**Apply** [attachment: 12415_spkg_bin_sage.patch](https://github.com/sagemath/sage-prod/files/10654696/12415_spkg_bin_sage.patch.gz) to `$SAGE_ROOT/spkg/bin/sage` (there's no repo here so you need to use patch)
+**Apply** [attachment: 12415_spkg_bin_sage.patch](https://github.com/sagemath/sage-prod/files/10654696/12415_spkg_bin_sage.patch.gz) to the root repo

jdemeyer commented 12 years ago

comment:29

sage/parallel/decorate.py: Use

Unhandled SIG...

(on systems, this would give a SIGBUS: Bus Error instead)

jdemeyer commented 12 years ago

comment:30

Please explain why you're adding nodoctest to some of the files in doc/?

jdemeyer commented 12 years ago

comment:31

Are all the options in sage-runtests actually implemented? (it seems some of them are meant for #12720, these should not be added here)

jdemeyer commented 12 years ago

comment:32

I guess you're no longer using the scripts sage-test, sage-ptest, sage-doctest, sage-maketest, perhaps sage-test-new and sage-test-import. Remove these.

jdemeyer commented 12 years ago

comment:33

Why does the doctesting framework need to be imported from sage.all?

ohanar commented 12 years ago

comment:34

Replying to @jdemeyer:

Why does the doctesting framework need to be imported from sage.all?

I don't think it needs to be, but it makes it easy to test (and debug) doctests from within Sage:

sage: run_doctests(sage.rings.integer)
...

jdemeyer commented 12 years ago

comment:35

You have two instances of

except:

in the new code.

The one for os.kill() should just catch OSError, for the other one replace

except KeyboardInterrupt:
    pass
except:

by

except Exception:

jdemeyer commented 12 years ago

comment:36

Replace

import multiprocessing 
options.nthreads = os.getenv('SAGE_NUM_THREADS_PARALLEL',min(8, multiprocessing.cpu_count()))

by

options.nthreads = int(os.getenv('SAGE_NUM_THREADS_PARALLEL', 1))

(don't reinvent the sage-num-threads.py wheel)

jdemeyer commented 12 years ago

Reviewer: Jeroen Demeyer

jdemeyer commented 12 years ago

comment:38

Planning to make a reviewer patch, so hang on...

jdemeyer commented 12 years ago

Author: David Roe, Robert Bradshaw

jdemeyer commented 12 years ago

comment:40

This actually fails on the very first thing I tried (after applying all patches and doing sage -b):

jdemeyer@sage:/release/sage-5.1.beta5-doctest$ ./sage -t devel/sage/sage/rings/padics/
argv ['/release/sage-5.1.beta5-doctest/local/bin/sage-runtests', 'devel/sage/sage/rings/padics/']
Traceback (most recent call last):
  File "/release/sage-5.1.beta5-doctest/local/bin/sage-runtests", line 68, in <module>
    DC = DocTestController(*parser.parse_args())
  File "/release/sage-5.1.beta5-doctest/local/lib/python2.7/site-packages/sage/doctest/control.py", line 116, in __init__
    if options.nthreads == 0:
AttributeError: Values instance has no attribute 'nthreads'

Am I doing something wrong?

jdemeyer commented 12 years ago

comment:41

Building from scratch and doing make ptest gives:

argv ['/release/merger/sage-5.1.beta5-12415/local/bin/sage-runtests', '-p', '--sagenb', 'devel/sage/doc/common', 'devel/sage/doc/de', 'devel/sage/doc/en', 'devel/sage/doc/fr', 'devel/sage/doc/ru', 'devel/sage/doc/tr', 'devel/sage/sage']
next_arg: --sagenb
Running doctests with ID 2012-06-25-12-57-22-d71deecd.
Doctesting the Sage notebook.
Doctesting 2067 files using 6 threads
Usage: sage -docbuild [OPTIONS] DOCUMENT (FORMAT | COMMAND)

sage-runtests: error: no such option: -p
**********************************************************************
File "programming", line 620, in programming
Failed example:
    for i in range(5):
Exception raised:
    Traceback (most recent call last):
      File "/release/merger/sage-5.1.beta5-12415/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 572, in _run
        compileflags, 1)
      File "<doctest programming[0]>", line 1
        for i in range(5):
                         ^
    SyntaxError: unexpected EOF while parsing
**********************************************************************
1 items had failures:
   1 of 113 in programming
***Test Failed*** 1 failures.
**********************************************************************
File "programming", line 578, in programming
Failed example:
    for i in range(5):
Exception raised:
    Traceback (most recent call last):
      File "/release/merger/sage-5.1.beta5-12415/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 572, in _run
        compileflags, 1)
      File "<doctest programming[0]>", line 1
        for i in range(5):
                         ^
    SyntaxError: unexpected EOF while parsing
**********************************************************************
1 items had failures:
   1 of 113 in programming
***Test Failed*** 1 failures.
**********************************************************************
File "sage.misc.sh", line 20, in sage.misc.sh.Sh.eval
Failed example:
    sh.eval('''echo "Hello there"\nif [ $? -eq 0 ]; then\necho "good"\nfi''')
Expected:
    /...
    ''
Got:
    Hello there
    good
    /release/merger/sage-5.1.beta5-12415
    ''
**********************************************************************
File "interfaces", line 331, in interfaces
Failed example:
    maxima.plot3d("[cos(x)*(3 + y*cos(x/2)), sin(x)*(3 + y*cos(x/2)),\
      y*sin(x/2)]", "[x, -4, 4]", "[y, -4, 4]",\ 
      '[plot_format, openmath]') # not tested
Exception raised:
    Traceback (most recent call last):
      File "/release/merger/sage-5.1.beta5-12415/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 572, in _run
        compileflags, 1)
      File "<doctest interfaces[0]>", line 2
        maxima.plot3d("[cos(x)*(3 + y*cos(x/2)), sin(x)*(3 + y*cos(x/2)),\
      y*sin(x/2)]", "[x, -4, 4]", "[y, -4, 4]",\

                                                 ^
    SyntaxError: unexpected character after line continuation character
**********************************************************************
1 items had failures:
   1 of  57 in interfaces
***Test Failed*** 1 failures.
**********************************************************************
1 items had failures:
   1 of   1 in sage.misc.sh.Sh.eval
***Test Failed*** 1 failures.
**********************************************************************
File "sage.doctest.control", line 246, in sage.doctest.control.DocTestController.add_files
Failed example:
    DC = DocTestController(DD, [])
Exception raised:
    Traceback (most recent call last):
      File "/release/merger/sage-5.1.beta5-12415/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 573, in _run
        self.execute(example, compiled, test.globs)
      File "/release/merger/sage-5.1.beta5-12415/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 835, in execute
        exec compiled in globs
      File "<doctest sage.doctest.control.DocTestController.add_files[2]>", line 1, in <module>
        DC = DocTestController(DD, [])
      File "/release/merger/sage-5.1.beta5-12415/local/lib/python2.7/site-packages/sage/doctest/control.py", line 127, in __init__
        self.logfile = open(options.logfile, 'a')
    IOError: [Errno 2] No such file or directory: '/release/merger/sage-5.1.beta5-12415/home/.sage//tmp/test.log'
**********************************************************************
File "sage.doctest.control", line 247, in sage.doctest.control.DocTestController.add_files
Failed example:
    DC.add_files()
Exception raised:
    Traceback (most recent call last):
      File "/release/merger/sage-5.1.beta5-12415/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 573, in _run
        self.execute(example, compiled, test.globs)
      File "/release/merger/sage-5.1.beta5-12415/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 835, in execute
        exec compiled in globs
      File "<doctest sage.doctest.control.DocTestController.add_files[3]>", line 1, in <module>
        DC.add_files()
    NameError: name 'DC' is not defined
**********************************************************************
File "sage.doctest.control", line 249, in sage.doctest.control.DocTestController.add_files
Failed example:
    (os.path.join(os.environ['SAGE_ROOT'], 'devel', 'sage', 'sage'), True) in DC.files
Exception raised:
    Traceback (most recent call last):
      File "/release/merger/sage-5.1.beta5-12415/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 573, in _run
        self.execute(example, compiled, test.globs)
      File "/release/merger/sage-5.1.beta5-12415/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 835, in execute
        exec compiled in globs
      File "<doctest sage.doctest.control.DocTestController.add_files[4]>", line 1, in <module>
        (os.path.join(os.environ['SAGE_ROOT'], 'devel', 'sage', 'sage'), True) in DC.files
    NameError: name 'DC' is not defined
**********************************************************************
File "sage.doctest.control", line 418, in sage.doctest.control.DocTestController.run_doctests
Failed example:
    DC.run_doctests()
Expected:
    Doctesting 1 file.
    sage -t .../sage/rings/homset.py
        [... tests, ... s]
    ------------------------------------------------------------------------
    All tests passed!
    ------------------------------------------------------------------------
    Total time for all tests: ... seconds
        cpu time: ... seconds
        cumulative wall time: ... seconds
Got:
    .
    sage -t /release/merger/sage-5.1.beta5-12415/devel/sage/sage/rings/homset.py
        [15 tests, 0.3 s]
    ------------------------------------------------------------------------
    All tests passed!
    ------------------------------------------------------------------------
    Total time for all tests: 0.8 seconds
        cpu time: 0.1 seconds
        cumulative wall time: 0.3 seconds
**********************************************************************
File "interfaces", line 318, in interfaces
Failed example:
    maxima.plot3d("[cos(x)*(3 + y*cos(x/2)), sin(x)*(3 + y*cos(x/2)),\
      y*sin(x/2)]", "[x, -4, 4]", "[y, -4, 4]",\ 
      '[plot_format, openmath]') # not tested
Exception raised:
    Traceback (most recent call last):
      File "/release/merger/sage-5.1.beta5-12415/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 572, in _run
        compileflags, 1)
      File "<doctest interfaces[0]>", line 2
        maxima.plot3d("[cos(x)*(3 + y*cos(x/2)), sin(x)*(3 + y*cos(x/2)),\
      y*sin(x/2)]", "[x, -4, 4]", "[y, -4, 4]",\

                                                 ^
    SyntaxError: unexpected character after line continuation character
**********************************************************************
1 items had failures:
   1 of  57 in interfaces
***Test Failed*** 1 failures.
**********************************************************************
File "sage.doctest.control", line 479, in sage.doctest.control.DocTestController.cleanup
Failed example:
    DC.run()
Expected:
    Running doctests with ID ...
    Doctesting 1 file.
    sage -t .../rings/infinity.py
        [... tests, ... s]
    ------------------------------------------------------------------------
    All tests passed!
    ------------------------------------------------------------------------
    Total time for all tests: ... seconds
        cpu time: ... seconds
        cumulative wall time: ... seconds
    0
Got:
    Running doctests with ID 2012-06-25-12-57-35-cf5e164b.
    .
    sage -t /release/merger/sage-5.1.beta5-12415/devel/sage/sage/rings/infinity.py
        [220 tests, 12.8 s]
    ------------------------------------------------------------------------
    All tests passed!
    ------------------------------------------------------------------------
    Total time for all tests: 13.8 seconds
        cpu time: 1.0 seconds
        cumulative wall time: 12.8 seconds
    0
**********************************************************************
File "sage.doctest.control", line 619, in sage.doctest.control.DocTestController.run
Failed example:
    DC.run()
Expected:
    Running doctests with ID ...
    Doctesting 1 file.
    sage -t .../sage/sets/non_negative_integers.py
        [... tests, ... s]
    ------------------------------------------------------------------------
    All tests passed!
    ------------------------------------------------------------------------
    Total time for all tests: ... seconds
        cpu time: ... seconds
        cumulative wall time: ... seconds
    0
Got:
    Running doctests with ID 2012-06-25-12-57-49-f475b4d8.
    .
    sage -t /release/merger/sage-5.1.beta5-12415/devel/sage/sage/sets/non_negative_integers.py
        [41 tests, 0.5 s]
    ------------------------------------------------------------------------
    All tests passed!
    ------------------------------------------------------------------------
    Total time for all tests: 1.4 seconds
        cpu time: 0.1 seconds
        cumulative wall time: 0.5 seconds
    0
**********************************************************************
File "sage.doctest.control", line 660, in sage.doctest.control.run_doctests
Failed example:
    run_doctests(sage.rings.infinity)
Expected:
    Doctesting .../sage/rings/infinity.py
    Running doctests with ID ...
    Doctesting 1 file.
    sage -t .../sage/rings/infinity.py
        [... tests, ... s]
    ------------------------------------------------------------------------
    All tests passed!
    ------------------------------------------------------------------------
    Total time for all tests: ... seconds
        cpu time: ... seconds
        cumulative wall time: ... seconds
Got:
    Doctesting /release/merger/sage-5.1.beta5-12415/devel/sage/sage/rings/infinity.py
    Running doctests with ID 2012-06-25-12-57-50-38f106c7.
    .
    sage -t /release/merger/sage-5.1.beta5-12415/devel/sage/sage/rings/infinity.py
        [220 tests, 20.3 s]
    ------------------------------------------------------------------------
    All tests passed!
    ------------------------------------------------------------------------
    Total time for all tests: 22.3 seconds
        cpu time: 1.0 seconds
        cumulative wall time: 20.3 seconds
**********************************************************************
5 items had failures:
   3 of  13 in sage.doctest.control.DocTestController.add_files
   1 of  10 in sage.doctest.control.DocTestController.cleanup
   1 of   6 in sage.doctest.control.DocTestController.run
   1 of   7 in sage.doctest.control.DocTestController.run_doctests
   1 of   1 in sage.doctest.control.run_doctests
***Test Failed*** 7 failures.
**********************************************************************
File "sage.parallel.decorate", line 559, in sage.parallel.decorate.fork
Failed example:
    g()
Expected:
    <BLANKLINE>
    ------------------------------------------------------------------------
    Unhandled SIGSEGV: A segmentation fault occurred in Sage.
    This probably occurred because a *compiled* component of Sage has a bug
    in it and is not properly wrapped with sig_on(), sig_off(). You might
    want to run Sage under gdb with 'sage -gdb' to debug this.
    Sage will now terminate.
    ------------------------------------------------------------------------
    'NO DATA'
Got:
    /release/merger/sage-5.1.beta5-12415/local/lib/libcsage.so(print_backtrace+0x2b)[0x2b39ebbaf66e]
    /release/merger/sage-5.1.beta5-12415/local/lib/libcsage.so(sigdie+0x14)[0x2b39ebbaf69b]
    /release/merger/sage-5.1.beta5-12415/local/lib/libcsage.so(sage_signal_handler+0x20b)[0x2b39ebbaf189]
    /lib/libpthread.so.0[0x2b39e86417d0]
    /lib/libc.so.6(strlen+0x30)[0x2b39e8f532f0]
    /release/merger/sage-5.1.beta5-12415/local/lib/libpython2.7.so.1.0(PyString_FromString+0x1b)[0x2b39e82e486b]
    /release/merger/sage-5.1.beta5-12415/home/.sage//temp/sage.math.washington.edu/28931//spyx//_release_merger_sage_5_1_beta5_12415_home__sage_temp_sage_math_washington_edu_28931_tmp_0_spyx/_release_merger_sage_5_1_beta5_12415_home__sage_temp_sage_math_washington_edu_28931_tmp_0_spyx_0.so[0x2aaaaaab0abd]
    /release/merger/sage-5.1.beta5-12415/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5a63)[0x2b39e833b2a3]
    /release/merger/sage-5.1.beta5-12415/local/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x852)[0x2b39e833d312]
    /release/merger/sage-5.1.beta5-12415/local/lib/libpython2.7.so.1.0[0x2b39e82c1979]
    /release/merger/sage-5.1.beta5-12415/local/lib/libpython2.7.so.1.0(PyObject_Call+0x68)[0x2b39e82942c8]
    /release/merger/sage-5.1.beta5-12415/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x37bf)[0x2b39e8338fff]
    /release/merger/sage-5.1.beta5-12415/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x69c5)[0x2b39e833c205]
    /release/merger/sage-5.1.beta5-12415/local/lib/libpython2.7.so.1.0[0x2b39e82b6b09]
    /release/merger/sage-5.1.beta5-12415/local/lib/libpython2.7.so.1.0[0x2b39e82c952d]
    /release/merger/sage-5.1.beta5-12415/local/lib/libpython2.7.so.1.0[0x2b39e82c9acb]
    /release/merger/sage-5.1.beta5-12415/local/lib/libpython2.7.so.1.0[0x2b39e82f0568]
    /release/merger/sage-5.1.beta5-12415/local/lib/libpython2.7.so.1.0(PyObject_Call+0x68)[0x2b39e82942c8]
    /release/merger/sage-5.1.beta5-12415/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x1299)[0x2b39e8336ad9]
    /release/merger/sage-5.1.beta5-12415/local/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x852)[0x2b39e833d312]
    /release/merger/sage-5.1.beta5-12415/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5ae4)[0x2b39e833b324]
    /release/merger/sage-5.1.beta5-12415/local/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x852)[0x2b39e833d312]
    /release/merger/sage-5.1.beta5-12415/local/lib/libpython2.7.so.1.0(PyEval_EvalCode+0x32)[0x2b39e833d432]
    /release/merger/sage-5.1.beta5-12415/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x714e)[0x2b39e833c98e]
    /release/merger/sage-5.1.beta5-12415/local/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x852)[0x2b39e833d312]
    /release/merger/sage-5.1.beta5-12415/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5ae4)[0x2b39e833b324]
    /release/merger/sage-5.1.beta5-12415/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x69c5)[0x2b39e833c205]
    /release/merger/sage-5.1.beta5-12415/local/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x852)[0x2b39e833d312]
    /release/merger/sage-5.1.beta5-12415/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5ae4)[0x2b39e833b324]
    /release/merger/sage-5.1.beta5-12415/local/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x852)[0x2b39e833d312]
    /release/merger/sage-5.1.beta5-12415/local/lib/libpython2.7.so.1.0[0x2b39e82c1979]
    /release/merger/sage-5.1.beta5-12415/local/lib/libpython2.7.so.1.0(PyObject_Call+0x68)[0x2b39e82942c8]
    /release/merger/sage-5.1.beta5-12415/local/lib/libpython2.7.so.1.0[0x2b39e82a487f]
    /release/merger/sage-5.1.beta5-12415/local/lib/libpython2.7.so.1.0(PyObject_Call+0x68)[0x2b39e82942c8]
    /release/merger/sage-5.1.beta5-12415/local/lib/libpython2.7.so.1.0[0x2b39e82f563a]
    /release/merger/sage-5.1.beta5-12415/local/lib/libpython2.7.so.1.0(PyObject_Call+0x68)[0x2b39e82942c8]
    /release/merger/sage-5.1.beta5-12415/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x37bf)[0x2b39e8338fff]
    /release/merger/sage-5.1.beta5-12415/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x69c5)[0x2b39e833c205]
    /release/merger/sage-5.1.beta5-12415/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x69c5)[0x2b39e833c205]
    /release/merger/sage-5.1.beta5-12415/local/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x852)[0x2b39e833d312]
    /release/merger/sage-5.1.beta5-12415/local/lib/libpython2.7.so.1.0[0x2b39e82c1979]
    /release/merger/sage-5.1.beta5-12415/local/lib/libpython2.7.so.1.0(PyObject_Call+0x68)[0x2b39e82942c8]
    /release/merger/sage-5.1.beta5-12415/local/lib/libpython2.7.so.1.0[0x2b39e82a487f]
    /release/merger/sage-5.1.beta5-12415/local/lib/libpython2.7.so.1.0(PyObject_Call+0x68)[0x2b39e82942c8]
    /release/merger/sage-5.1.beta5-12415/local/lib/libpython2.7.so.1.0[0x2b39e82f523c]
    /release/merger/sage-5.1.beta5-12415/local/lib/libpython2.7.so.1.0[0x2b39e82f0568]
    /release/merger/sage-5.1.beta5-12415/local/lib/libpython2.7.so.1.0(PyObject_Call+0x68)[0x2b39e82942c8]
    /release/merger/sage-5.1.beta5-12415/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x1299)[0x2b39e8336ad9]
    /release/merger/sage-5.1.beta5-12415/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x69c5)[0x2b39e833c205]
    /release/merger/sage-5.1.beta5-12415/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x69c5)[0x2b39e833c205]
    /release/merger/sage-5.1.beta5-12415/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x69c5)[0x2b39e833c205]
    /release/merger/sage-5.1.beta5-12415/local/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x852)[0x2b39e833d312]
    /release/merger/sage-5.1.beta5-12415/local/lib/libpython2.7.so.1.0[0x2b39e82c1979]
    /release/merger/sage-5.1.beta5-12415/local/lib/libpython2.7.so.1.0(PyObject_Call+0x68)[0x2b39e82942c8]
    /release/merger/sage-5.1.beta5-12415/local/lib/libpython2.7.so.1.0[0x2b39e82a487f]
    /release/merger/sage-5.1.beta5-12415/local/lib/libpython2.7.so.1.0(PyObject_Call+0x68)[0x2b39e82942c8]
    /release/merger/sage-5.1.beta5-12415/local/lib/libpython2.7.so.1.0[0x2b39e82f523c]
    /release/merger/sage-5.1.beta5-12415/local/lib/libpython2.7.so.1.0[0x2b39e82f0568]
    /release/merger/sage-5.1.beta5-12415/local/lib/libpython2.7.so.1.0(PyObject_Call+0x68)[0x2b39e82942c8]
    /release/merger/sage-5.1.beta5-12415/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x1299)[0x2b39e8336ad9]
    /release/merger/sage-5.1.beta5-12415/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x69c5)[0x2b39e833c205]
    /release/merger/sage-5.1.beta5-12415/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x69c5)[0x2b39e833c205]
    /release/merger/sage-5.1.beta5-12415/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x69c5)[0x2b39e833c205]
    /release/merger/sage-5.1.beta5-12415/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x69c5)[0x2b39e833c205]
    /release/merger/sage-5.1.beta5-12415/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x69c5)[0x2b39e833c205]
    /release/merger/sage-5.1.beta5-12415/local/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x852)[0x2b39e833d312]
    /release/merger/sage-5.1.beta5-12415/local/lib/libpython2.7.so.1.0(PyEval_EvalCode+0x32)[0x2b39e833d432]
    /release/merger/sage-5.1.beta5-12415/local/lib/libpython2.7.so.1.0(PyRun_FileExFlags+0xc1)[0x2b39e83611b1]
    /release/merger/sage-5.1.beta5-12415/local/lib/libpython2.7.so.1.0(PyRun_SimpleFileExFlags+0x1f9)[0x2b39e8361489]
    /release/merger/sage-5.1.beta5-12415/local/lib/libpython2.7.so.1.0(Py_Main+0xb15)[0x2b39e8373e65]
    /lib/libc.so.6(__libc_start_main+0xf4)[0x2b39e8ef61f4]
    python[0x400679]
    <BLANKLINE>
    ------------------------------------------------------------------------
    Unhandled SIGSEGV: A segmentation fault occurred in Sage.
    This probably occurred because a *compiled* component of Sage has a bug
    in it and is not properly wrapped with sig_on(), sig_off(). You might
    want to run Sage under gdb with 'sage -gdb' to debug this.
    Sage will now terminate.
    ------------------------------------------------------------------------
    'NO DATA'
**********************************************************************
1 items had failures:
   1 of  17 in sage.parallel.decorate.fork
***Test Failed*** 1 failures.
**********************************************************************
File "sage.tests.startup", line 6, in sage.tests.startup
Failed example:
    if os.uname()[1] == 'sage.math.washington.edu':
        print float(os.popen("sage -startuptime>/dev/null; sage -startuptime|grep sage.all").readlines()[0].split()[1]) < 2.0
    else: print True   # nothing when not on sage.math
Expected:
    True
Got:
    False
**********************************************************************
File "sage.rings.polynomial.multi_polynomial_libsingular", line 420, in sage.rings.polynomial.multi_polynomial_libsingular.MPolynomialRing_libsingular.__copy__
Failed example:
    len(ring_refcount_dict) == n   
Expected:
    True
Got:
    False
**********************************************************************
File "sage.rings.polynomial.plural", line 2738, in sage.rings.polynomial.plural.new_NRing
Failed example:
    z*x
Expected:
    x*z
Got:
    Exception KeyError: (The ring pointer 0x2b3a04886e90,) in 'sage.libs.singular.ring.singular_ring_delete' ignored
    x*z
**********************************************************************
File "sage.rings.polynomial.plural", line 2856, in sage.rings.polynomial.plural.SCA
Failed example:
    E = SCA(QQ, ['x', 'y', 'z'], [0, 1], order = 'degrevlex')
Expected nothing
Got:
    Exception KeyError: (The ring pointer 0x2b3a048af1a0,) in 'sage.libs.singular.ring.singular_ring_delete' ignored
**********************************************************************
File "sage.matrix.benchmark", line 11, in sage.matrix.benchmark
Failed example:
    b.report([b.det_ZZ], 'Test', systems=['sage'])
Expected:
    ======================================================================
              Test
    ======================================================================
    ...
    ======================================================================
Got:
    Linux sage.math.washington.edu 2.6.24-28-server #1 SMP Fri Feb 11 18:08:32 UTC 2011 x86_64 GNU/Linux
    ======================================================================
              Test
    ======================================================================
    <BLANKLINE>
    <BLANKLINE>
    ----------------------------------------------------------------------
    Dense integer determinant over ZZ.
        Given an n x n matrix A over ZZ with random entries
        between min and max, inclusive, compute det(A).
    <BLANKLINE>
   **********************************************************************
2 items had failures:
   1 of   8 in sage.rings.polynomial.plural.SCA
   1 of  17 in sage.rings.polynomial.plural.new_NRing
***Test Failed*** 2 failures.
     INPUT:
    <BLANKLINE>
        - ``n`` - matrix dimension (default: ``200``)
        - ``min`` - minimal value for entries of matrix (default: ``1``)
        - ``max`` - maximal value for entries of matrix (default: ``100``)
        - ``system`` - either 'sage' or 'magma' (default: 'sage')
    <BLANKLINE>
        EXAMPLES::
    <BLANKLINE>
            sage: import sage.matrix.benchmark as b
            sage: ts = b.det_ZZ(200)
            sage: tm = b.det_ZZ(200, system='magma')  # optional - magma
               sage
              1.560
    ======================================================================
**********************************************************************
1 items had failures:
   1 of  16 in sage.rings.polynomial.multi_polynomial_libsingular.MPolynomialRing_libsingular.__copy__
***Test Failed*** 1 failures.
**********************************************************************
1 items had failures:
   1 of   2 in sage.tests.startup
***Test Failed*** 1 failures.
**********************************************************************
File "sage.matrix.benchmark", line 38, in sage.matrix.benchmark.report
Failed example:
    b.report([b.det_ZZ], 'Test', systems=['sage'])
Expected:
    ======================================================================
              Test
    ======================================================================
    ...
    ======================================================================
Got:
    Linux sage.math.washington.edu 2.6.24-28-server #1 SMP Fri Feb 11 18:08:32 UTC 2011 x86_64 GNU/Linux
    ======================================================================
              Test
    ======================================================================
    <BLANKLINE>
    <BLANKLINE>
    ----------------------------------------------------------------------
    Dense integer determinant over ZZ.
        Given an n x n matrix A over ZZ with random entries
        between min and max, inclusive, compute det(A).
    <BLANKLINE>
        INPUT:
    <BLANKLINE>
        - ``n`` - matrix dimension (default: ``200``)
        - ``min`` - minimal value for entries of matrix (default: ``1``)
        - ``max`` - maximal value for entries of matrix (default: ``100``)
        - ``system`` - either 'sage' or 'magma' (default: 'sage')
    <BLANKLINE>
        EXAMPLES::
    <BLANKLINE>
            sage: import sage.matrix.benchmark as b
            sage: ts = b.det_ZZ(200)
            sage: tm = b.det_ZZ(200, system='magma')  # optional - magma
               sage
              1.330
    ======================================================================
**********************************************************************
File "sage.doctest.forker", line 925, in sage.doctest.forker.SageDocTestRunner.report_unexpected_exception
Failed example:
    "ArithmeticError" in tb
Expected:
    True
Got:
    False
**********************************************************************
File "sage.doctest.forker", line 1185, in sage.doctest.forker.DocTestWorker.__init__
Failed example:
    run_doctests(sage.rings.big_oh) # indirect doctest
Expected:
    Doctesting .../sage/rings/big_oh.py
    Running doctests with ID ...
    Doctesting 1 file.
    sage -t .../sage/rings/big_oh.py
        [... tests, ... s]
    ------------------------------------------------------------------------
    All tests passed!
    ------------------------------------------------------------------------
    Total time for all tests: ... seconds
        cpu time: ... seconds
        cumulative wall time: ... seconds
Got:
    Doctesting /release/merger/sage-5.1.beta5-12415/devel/sage/sage/rings/big_oh.py
    Running doctests with ID 2012-06-25-12-59-29-e5a039b5.
    .
    sage -t /release/merger/sage-5.1.beta5-12415/devel/sage/sage/rings/big_oh.py
        [17 tests, 1.4 s]
    ------------------------------------------------------------------------
    All tests passed!
    ------------------------------------------------------------------------
    Total time for all tests: 1.9 seconds
        cpu time: 0.4 seconds
        cumulative wall time: 1.4 seconds
**********************************************************************
File "sage.doctest.forker", line 1211, in sage.doctest.forker.DocTestWorker.run
Failed example:
    run_doctests(sage.symbolic.units) # indirect doctest
Expected:
    Doctesting .../sage/symbolic/units.py
    Running doctests with ID ...
    Doctesting 1 file.
    sage -t .../sage/symbolic/units.py
        [... tests, ... s]
    ------------------------------------------------------------------------
    All tests passed!
    ------------------------------------------------------------------------
    Total time for all tests: ... seconds
        cpu time: ... seconds
        cumulative wall time: ... seconds
Got:
    Doctesting /release/merger/sage-5.1.beta5-12415/devel/sage/sage/symbolic/units.py
    Running doctests with ID 2012-06-25-12-59-31-e804e534.
    .
    sage -t /release/merger/sage-5.1.beta5-12415/devel/sage/sage/symbolic/units.py
        [82 tests, 0.4 s]
    ------------------------------------------------------------------------
    All tests passed!
    ------------------------------------------------------------------------
    Total time for all tests: 0.7 seconds
        cpu time: 0.2 seconds
        cumulative wall time: 0.4 seconds
**********************************************************************
3 items had failures:
   1 of   1 in sage.doctest.forker.DocTestWorker.__init__
   1 of   1 in sage.doctest.forker.DocTestWorker.run
   1 of  18 in sage.doctest.forker.SageDocTestRunner.report_unexpected_exception
***Test Failed*** 3 failures.
**********************************************************************
File "sage.tests.cmdline", line 270, in sage.tests.cmdline.test_executable
Failed example:
    ret
Expected:
    0
Got:
    1
**********************************************************************
File "sage.tests.cmdline", line 272, in sage.tests.cmdline.test_executable
Failed example:
    out.find("All tests passed!") >= 0
Expected:
    True
Got:
    False
**********************************************************************
File "sage.tests.cmdline", line 275, in sage.tests.cmdline.test_executable
Failed example:
    ret
Expected:
    0
Got:
    1
**********************************************************************
File "sage.tests.cmdline", line 277, in sage.tests.cmdline.test_executable
Failed example:
    out.find("All tests passed!") >= 0
Expected:
    True
Got:
    False
**********************************************************************
File "sage.tests.cmdline", line 291, in sage.tests.cmdline.test_executable
Failed example:
    ret
Expected:
    128
Got:
    1
**********************************************************************
File "sage.tests.cmdline", line 293, in sage.tests.cmdline.test_executable
Failed example:
    out.find("1 items had failures:") >= 0
Expected:
    True
Got:
    False
**********************************************************************
1 items had failures:
   6 of 183 in sage.tests.cmdline.test_executable
***Test Failed*** 6 failures.
**********************************************************************
File "sage.matrix.benchmark", line 566, in sage.matrix.benchmark.report_GF
Failed example:
    b.report_GF(systems=['sage'])
Expected:
    ======================================================================
    Dense benchmarks over GF with prime 16411
    ======================================================================
    ...
    ======================================================================
Got:
    Linux sage.math.washington.edu 2.6.24-28-server #1 SMP Fri Feb 11 18:08:32 UTC 2011 x86_64 GNU/Linux
    ======================================================================
              Dense benchmarks over GF with prime 16411
    ======================================================================
    <BLANKLINE>
    <BLANKLINE>
    ----------------------------------------------------------------------
    Rank over GF(p):
        Given a n x (n+10) matrix over GF(p) with random entries, compute the rank.
    <BLANKLINE>
        INPUT:
    <BLANKLINE>
        - ``n`` - matrix dimension (default: 300)
        - ``p`` - prime number (default: ``16411``)
        - ``system`` - either 'magma' or 'sage' (default: 'sage')
    <BLANKLINE>
        EXAMPLES::
    <BLANKLINE>
            sage: import sage.matrix.benchmark as b
            sage: ts = b.rank_GF(1000)
            sage: tm = b.rank_GF(1000, system='magma')  # optional - magma
               sage
              0.160
    ----------------------------------------------------------------------
    Rank over GF(p): Given a (n + 10) x n matrix over GF(p) with
        random entries, compute the rank.
    <BLANKLINE>
        INPUT:
    <BLANKLINE>
        - ``n`` - matrix dimension (default: 300)
        - ``p`` - prime number (default: ``16411``)
        - ``system`` - either 'magma' or 'sage' (default: 'sage')
    <BLANKLINE>
        EXAMPLES::
    <BLANKLINE>
            sage: import sage.matrix.benchmark as b
            sage: ts = b.rank2_GF(500)
            sage: tm = b.rank2_GF(500, system='magma')  # optional - magma
               sage
              0.170
    ----------------------------------------------------------------------
    Given a n+1 x n  matrix over GF(p) with random
        entries, compute the nullspace.
    <BLANKLINE>
        INPUT:
    <BLANKLINE>
        - ``n`` - matrix dimension (default: 300)
        - ``p`` - prime number (default: ``16411``)
        - ``system`` - either 'magma' or 'sage' (default: 'sage')
    <BLANKLINE>
        EXAMPLES::
    <BLANKLINE>
            sage: import sage.matrix.benchmark as b
            sage: ts = b.nullspace_GF(300)
            sage: tm = b.nullspace_GF(300, system='magma')  # optional - magma
               sage
              0.820
    ----------------------------------------------------------------------
    Given a n x n matrix over GF with random entries, compute the
        charpoly.
    <BLANKLINE>
        INPUT:
    <BLANKLINE>
        - ``n`` - matrix dimension (default: 100)
        - ``p`` - prime number (default: ``16411``)
        - ``system`` - either 'magma' or 'sage' (default: 'sage')
    <BLANKLINE>
        EXAMPLES::
    <BLANKLINE>
            sage: import sage.matrix.benchmark as b
            sage: ts = b.charpoly_GF(100)
            sage: tm = b.charpoly_GF(100, system='magma')  # optional - magma
               sage
              0.060
    ----------------------------------------------------------------------
    Given an n x n matrix A over GF(p) with random entries, compute
        A * (A+1).
    <BLANKLINE>
        INPUT:
    <BLANKLINE>
        - ``n`` - matrix dimension (default: 100)
        - ``p`` - prime number (default: ``16411``)
        - ``system`` - either 'magma' or 'sage' (default: 'sage')
        - ``times`` - number of experiments (default: ``3``)
    <BLANKLINE>
        EXAMPLES::
    <BLANKLINE>
            sage: import sage.matrix.benchmark as b
            sage: ts = b.matrix_multiply_GF(100, p=19)
            sage: tm = b.matrix_multiply_GF(100, p=19, system='magma')  # optional - magma
               sage
              0.003
    ----------------------------------------------------------------------
    Dense determinant over GF(p).
        Given an n x n matrix A over GF with random entries compute
        det(A).
    <BLANKLINE>
        INPUT:
    <BLANKLINE>
        - ``n`` - matrix dimension (default: 300)
        - ``p`` - prime number (default: ``16411``)
        - ``system`` - either 'magma' or 'sage' (default: 'sage')
    <BLANKLINE>
        EXAMPLES::
    <BLANKLINE>
            sage: import sage.matrix.benchmark as b
            sage: ts = b.det_GF(1000)
            sage: tm = b.det_GF(1000, system='magma')  # optional - magma
               sage
              0.090
    ======================================================================
**********************************************************************
3 items had failures:
   1 of   2 in sage.matrix.benchmark
   1 of   2 in sage.matrix.benchmark.report
   1 of   2 in sage.matrix.benchmark.report_GF
***Test Failed*** 3 failures.

------------------------------------------------------------------------
Doctests interrupted: 0/2067 files tested
------------------------------------------------------------------------
Total time for all tests: 181.4 seconds
    cpu time: 0.0 seconds
    cumulative wall time: 0.0 seconds
Traceback (most recent call last):
  File "/release/merger/sage-5.1.beta5-12415/local/bin/sage-runtests", line 69, in <module>
    err = DC.run()
  File "/release/merger/sage-5.1.beta5-12415/local/lib/python2.7/site-packages/sage/doctest/control.py", line 645, in run
    self.run_doctests()
  File "/release/merger/sage-5.1.beta5-12415/local/lib/python2.7/site-packages/sage/doctest/control.py", line 446, in run_doctests
    self.dispatcher.dispatch()
  File "/release/merger/sage-5.1.beta5-12415/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 1130, in dispatch
    self._parallel_dispatch()
  File "/release/merger/sage-5.1.beta5-12415/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 1072, in _parallel_dispatch
    w.start()
  File "/release/merger/sage-5.1.beta5-12415/local/lib/python/multiprocessing/process.py", line 130, in start
    self._popen = Popen(self)
  File "/release/merger/sage-5.1.beta5-12415/local/lib/python/multiprocessing/forking.py", line 120, in __init__
    self.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory

jdemeyer commented 12 years ago

comment:42

Replying to @jdemeyer:

Planning to make a reviewer patch

...after you guys fix the more serious errors.

roed314 commented 12 years ago

comment:43

Some of these are due to a bug in the change I just made to -p. I'm working on fixing those.

Some look like you didn't apply the patch to the sagenb repo. Did you forget, or is something else going on?

Some are due to the fact that in your tests, os.system calls are happening before the python process gets far enough, so results that were printed to the screen in the wrong order. I'm guessing this may relate to the fact that you're running on /release, since my tests on sage.math didn't show the same behavior.

I'm in the process of fixing stuff.

roed314 commented 12 years ago

comment:44

There's some code in the current doctesting code to output xml results, which I'm not currently duplicating. Is this used somewhere?

roed314 commented 12 years ago

comment:45

Currently sage -t --new is broken. I'm trying to figure out why.

roed314 commented 12 years ago

comment:46

Alright, I'm going to start running tests. There are still some remaining issues, but the patches are in a state where someone else might want to look at them. :-)

Remaining problems:

13145 needs work, generating some doctest failures related to deallocating multivariate polynomials
hg_sage seems to be broken, causing the --new option to fail
valgrinding sage -t doesn't work; I'm not sure whether it did before
The new code doesn't support xml output

I'll see if I find more when running sage -t --all.

jdemeyer commented 12 years ago

comment:47

Replying to @roed314:

Some are due to the fact that in your tests, os.system calls are happening before the python process gets far enough, so results that were printed to the screen in the wrong order.

Use sys.stdout.flush() (and/or the same for stderr) before os.system() to overcome this. The difference is probably that my output was sent to a pipe, as opposed to directly to the terminal. In the latter case, flushing is automatic.

True, I might not have applied all patches correctly.

kini commented 12 years ago

comment:48

It doesn't make sense to patch the old notebook. Please work with the flask notebook instead (i.e. make this ticket depend on #11080 and preferably #13121).

roed314 commented 12 years ago

comment:49

Replying to @kini:

It doesn't make sense to patch the old notebook. Please work with the flask notebook instead (i.e. make this ticket depend on #11080 and preferably #13121).

I agree that it doesn't make much sense to patch the old notebook, but if this can get a positive review quickly I'd rather not wait on #11080 and #13121. The changes to the old notebook are pretty minor.

roed314 commented 12 years ago

comment:50

Design question: when running tests in parallel, failed examples are printed as they arise, meaning that examples from different files are intermingled. Is this a problem that needs to be fixed?

roed314 commented 12 years ago

comment:51

The only tests I have failing on sage.math are sage.tests.startup and sage.rings.polynomial.multi_polynomial_libsingular (due to #13145).

jdemeyer commented 12 years ago

comment:52

Replying to @roed314:

Design question: when running tests in parallel, failed examples are printed as they arise, meaning that examples from different files are intermingled. Is this a problem that needs to be fixed?

It might not be truly a problem, but I prefer the current model where the tests for one file are grouped.

sagemath / sage

Update doctesting framework #12415

13145 needs work, generating some doctest failures related to deallocating multivariate polynomials