Update doctesting framework

robertwb commented 12 years ago

There are several improvements that would be good to make, including but not limited to:

Forking rather than starting up a new Sage each time.
Customizable comparison rather than mutating doctests into new doctests (e.g. followup to #10952).
Eliminate need to write temporary files for every doctested file.
Constant headaches caused by the difference in doctesting library vs. non-library files.
The ability to drop into a debugger if a doctest raises an exception.

Robert's code is at

http://code.google.com/p/sagemath-timer/

Apply attachment: 12415_framework.patch, attachment: 12415_doctest_fixes.patch, attachment: 12415_doc.patch, attachment: 12415_review.patch, attachment: 12415_test.patch, attachment: 12415_review_review.patch, attachment: 12415_review3.patch, attachment: 12415_manifest.patch, attachment: 12415_rebase_58.patch

Apply attachment: 12415_script.patch and attachment: 12415_script_review.patch to the scripts repo

Apply attachment: 12415_spkg_bin_sage.patch to the root repo

For follow-up or other doctest-related tickets see #11337.

Depends on #13147 Depends on #13146 Depends on #13145 Depends on #12723 Depends on #12392 Depends on #12393 Depends on #12395 Depends on #12396 Depends on #12397 Depends on #12381 Depends on #12382 Depends on #12383 Depends on #12384 Depends on #11871 Depends on #13195 Depends on #13121 Depends on #13748 Depends on #13899 Depends on #12719 Depends on #5155 Depends on #14070 Depends on #14079 Depends on #14150 Depends on #14158 Depends on #14182 Depends on #14184 Depends on #14054 Depends on #14063 Depends on #13605 Depends on #14111 Depends on #14254 Depends on #14242 Depends on #14253

CC: @kini @ohanar @jhpalmieri

Component: doctest framework

Author: David Roe, Robert Bradshaw, Jeroen Demeyer

Reviewer: Jeroen Demeyer, David Roe

Merged: sage-5.9.beta0

Issue created by migration from https://trac.sagemath.org/ticket/12415

roed314 commented 12 years ago

comment:101

I updated the sagenb patch

kini commented 12 years ago

Description changed:

--- 
+++ 
@@ -16,6 +16,6 @@

 **Apply** [attachment: 12415_script.patch](https://github.com/sagemath/sage-prod/files/10654697/12415_script.patch.gz) to the scripts repo

-**Apply** [attachment: 12415_sagenb_fixes_vs_11080.patch](https://github.com/sagemath/sage/files/ticket12415/12415_sagenb_fixes_vs_11080.patch.gz) to the sagenb repo (fixes doctest errors)
+**Apply** [attachment: 12415_sagenb_fixes_vs_11080.patch](https://github.com/sagemath/sage/files/ticket12415/12415_sagenb_fixes_vs_11080.patch.gz) to the sagenb repo (fixes doctest errors) or wait for #13121

 **Apply** [attachment: 12415_spkg_bin_sage.patch](https://github.com/sagemath/sage-prod/files/10654696/12415_spkg_bin_sage.patch.gz) to the root repo

kini commented 12 years ago

comment:102

I imported the patch: https://github.com/sagemath/sagenb/pull/84

I'll cut a sagenb 0.10.2 release with this fix in it (and put it on #13121). Jeroen, are you aware of any other tickets with bugfix-type patches to sagenb which I could merge at the same time?

kini commented 12 years ago

comment:103

Actually, since this doctest fix actually breaks doctests with the current doctesting framework, should I leave this patch for sagenb 0.11 or something? How close would you say this ticket is to positive review? I notice it's set to milestone 5.3, whereas I'm hoping that #13121 can get into 5.2, so maybe that is the way to go.

jdemeyer commented 12 years ago

comment:104

Replying to @kini:

Actually, since this doctest fix actually breaks doctests with the current doctesting framework, should I leave this patch for sagenb 0.11 or something?

Yes please.

How close would you say this ticket is to positive review?

I think it's mostly finished, but since this is a big change, I'm certainly not merging it in sage-5.2. Also, there is a good chance that various small issues will come up here.

I notice it's set to milestone 5.3, whereas I'm hoping that #13121 can get into 5.2, so maybe that is the way to go.

Yes.

jdemeyer commented 12 years ago

comment:105

Just one failure remaining related to this ticket:

**********************************************************************
File "devel/sage/sage/doctest/control.py", line 258, in sage.doctest.control.DocTestController.add_files
Failed example:
    DC = DocTestController(DD, [])
Expected nothing
Got:
    Unable to open logfile at /release/merger/sage-5.1-12415/home/.sage//tmp/test.log
    Proceeding without logging.
sage -t devel/sage/sage/ext/gmp.pxi
    [0 tests, 0.0 s]
**********************************************************************
1 items had failures:
   1 of  15 in sage.doctest.control.DocTestController.add_files
***Test Failed*** 1 failures.
sage -t devel/sage/sage/doctest/control.py
    [115 tests, 3.6 s]

But also the formatting of the above failure looks strange:

1) Why does

sage -t devel/sage/sage/ext/gmp.pxi
    [0 tests, 0.0 s]

appear in the middle of the failure?

2) Could you format the output as

sage -t devel/sage/sage/doctest/control.py
**********************************************************************
[...failure...]
**********************************************************************
1 items had failures:
   1 of  15 in sage.doctest.control.DocTestController.add_files
***Test Failed*** 1 failures.
    [115 tests, 3.6 s]

I would even shorten the last two lines to

    [115 tests, 1 failed, 3.6 s]

And I would add a function like

def count_noun(number, noun):
    if number == 1:
        return "1 %s"%noun
    else:
        return "%d %ss"%(number,noun)

So you can do

print count_noun(n, 'item'), "had failures"

jdemeyer commented 12 years ago

comment:106

The KeyboardInterrupt example http://boxen.math.washington.edu/home/jdemeyer/doctest/keyboardinterrupt.py still doesn't work correctly:

Running doctests with ID 2012-07-13-10-53-02-ed232e22.
Doctesting 1 file.
sage -t /home/jdemeyer/doctest/keyboardinterrupt.py
    Unexpected KeyboardInterrupt raised in file
********************************************************************************
Tests run before interrupt:
sage: raise KeyboardInterrupt ## line 2 ##

********************************************************************************
------------------------------------------------------------------------
sage -t /home/jdemeyer/doctest/keyboardinterrupt.py # Unhandled KeyboardInterrupt
------------------------------------------------------------------------
Total time for all tests: 1.1 seconds
    cpu time: 0.0 seconds
    cumulative wall time: 0.0 seconds

The expected result is that the test passes. I don't mind if you don't fix this (currently, it's also broken), but I thought you should know...

jdemeyer commented 12 years ago

comment:107

Concerning the logfile error: I think you are making the mistaken assumption that $SAGE_TESTDIR is an existing directory. But I also find it strange that a doctest writes to a logfile inside SAGE_TESTDIR, I don't really like that.

jdemeyer commented 12 years ago

comment:108

When testing sage/tests/interrupt.pyx verbosely, I regularly get errors of the form

[...]
Trying (line 637):    test_try_finally_signal()
Expecting:
    Traceback (most recent call last):
    ...
    RuntimeError: Aborted
ok [0.23s]
Trying (line 100631):    sig_on_count()
Expecting:
    0
ok [0.00s]
sage -t --long devel/sage/sage/tests/interrupt.pyx
Traceback (most recent call last):
  File "/release/merger/sage-5.1-12415/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 1634, in __call__
  File "/release/merger/sage-5.1-12415/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 725, in run
  File "/release/merger/sage-5.1-12415/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 556, in _run
AttributeError: Example instance has no attribute 'options'

------------------------------------------------------------------------
sage -t --long devel/sage/sage/tests/interrupt.pyx # AttributeError in loading
------------------------------------------------------------------------
Total time for all tests: 12.7 seconds
    cpu time: 0.0 seconds
    cumulative wall time: 0.0 seconds

This looks like a bug in the doctesting framework (unless somehow the interrupt tests corrupt the doctest process?)

roed314 commented 12 years ago

comment:109

Replying to @jdemeyer:

The KeyboardInterrupt example http://boxen.math.washington.edu/home/jdemeyer/doctest/keyboardinterrupt.py still doesn't work correctly:

Running doctests with ID 2012-07-13-10-53-02-ed232e22.
Doctesting 1 file.
sage -t /home/jdemeyer/doctest/keyboardinterrupt.py
    Unexpected KeyboardInterrupt raised in file
********************************************************************************
Tests run before interrupt:
sage: raise KeyboardInterrupt ## line 2 ##

********************************************************************************
------------------------------------------------------------------------
sage -t /home/jdemeyer/doctest/keyboardinterrupt.py # Unhandled KeyboardInterrupt
------------------------------------------------------------------------
Total time for all tests: 1.1 seconds
    cpu time: 0.0 seconds
    cumulative wall time: 0.0 seconds

The expected result is that the test passes. I don't mind if you don't fix this (currently, it's also broken), but I thought you should know...

That's because your expected output has an extra colon. If you delete the trailing colon it passes tests.

With the extra colon it's an unexpected KeyboardInterrupt.

roed314 commented 12 years ago

comment:110

Replying to @jdemeyer:

Just one failure remaining related to this ticket:

**********************************************************************
File "devel/sage/sage/doctest/control.py", line 258, in sage.doctest.control.DocTestController.add_files
Failed example:
    DC = DocTestController(DD, [])
Expected nothing
Got:
    Unable to open logfile at /release/merger/sage-5.1-12415/home/.sage//tmp/test.log
    Proceeding without logging.
sage -t devel/sage/sage/ext/gmp.pxi
    [0 tests, 0.0 s]
**********************************************************************
1 items had failures:
   1 of  15 in sage.doctest.control.DocTestController.add_files
***Test Failed*** 1 failures.
sage -t devel/sage/sage/doctest/control.py
    [115 tests, 3.6 s]

But also the formatting of the above failure looks strange:

1) Why does

sage -t devel/sage/sage/ext/gmp.pxi
    [0 tests, 0.0 s]

appear in the middle of the failure?

2) Could you format the output as

sage -t devel/sage/sage/doctest/control.py
**********************************************************************
[...failure...]
**********************************************************************
1 items had failures:
   1 of  15 in sage.doctest.control.DocTestController.add_files
***Test Failed*** 1 failures.
    [115 tests, 3.6 s]

I would even shorten the last two lines to

    [115 tests, 1 failed, 3.6 s]

And I would add a function like

def count_noun(number, noun):
    if number == 1:
        return "1 %s"%noun
    else:
        return "%d %ss"%(number,noun)

So you can do

print count_noun(n, 'item'), "had failures"

Cool. I will work on these.

I'm going to be mostly (or entirely) out of communication for the next few days. Hopefully I'll have some changes available on Monday.

jhpalmieri commented 12 years ago

comment:111

From a discussion on sage-combinat-devel: would it be possible to optionally print extra information after doctesting: how many optional doctests were skipped, and perhaps how they were labeled? ("skipped 5 doctests marked "optional -- bug")

roed314 commented 12 years ago

comment:112

Replying to @jhpalmieri:

From a discussion on sage-combinat-devel: would it be possible to optionally print extra information after doctesting: how many optional doctests were skipped, and perhaps how they were labeled? ("skipped 5 doctests marked "optional -- bug")

Yeah, that shouldn't be a problem. But I'd like to put a feature freeze on this ticket and open a new ticket for it: I'm already regretting adding the new debugging feature since I'm struggling with debugging its interaction with KeyboardInterrupts.

I just opened #13278 for this.

kini commented 12 years ago

comment:113

Ping - should this still be needs_work?

jdemeyer commented 12 years ago

comment:114

There's still the interrupt.pyx failures which need to be investigated (probably by me since I know most about interrupts).

roed314 commented 12 years ago

comment:115

If you could look at those it would be great. My computer is currently being repaired, but I can try to take another shot at this next week.

roed314 commented 11 years ago

comment:116

Alright, I've rebased this against 5.4.rc1 and made the suggested changes in #13147. There's still an intermittent failure in sage.tests.interrupt and I'm at a loss for what's causing it. Help would be appreciated.

roed314 commented 11 years ago

comment:117

I've put the output before segfault from a case where interrupt.pyx failed at http://pastebin.com/Yvs8Ev68 (expiring in 1 month).

roed314 commented 11 years ago

comment:118

I've moved the debug functionality that triggers on an incorrect result to #13610 since I was struggling to get the interrupt handling working correctly (IPython embedded shells intercept KeyboardInterrupts). With this change the only work issue I'm aware of on this ticket is the intermittent failure in interrupt.pyx. Testing just interrupt.pyx doesn't trigger this failure (or at least does so rarely); run sage -t sage/tests/parigp.py sage/tests/interrupt.pyx for example.

roed314 commented 11 years ago

Work Issues: intermittent interrupt.pyx failure

roed314 commented 11 years ago

comment:119

Ok, I found another issue which I'm not going to fix tonight. The code that parses a file and generates the tests has a bug somewhere that means some files with tests are not getting tested. I need to track down the bug and add some doctests to make sure this kind of thing doesn't happen again.

roed314 commented 11 years ago

Description changed:

--- 
+++ 
@@ -12,10 +12,9 @@

 and does most of the above (plus more which has been moved to followup tickets #12720 and #12722)

-**Apply** [attachment: 12415_stderr_vs_51b5.patch](https://github.com/sagemath/sage/files/ticket12415/12415_stderr_vs_51b5.patch.gz), [attachment: 12415_doctest_fixes.patch](https://github.com/sagemath/sage-prod/files/10654706/12415_doctest_fixes.patch.gz), [attachment: 12415_framework.patch](https://github.com/sagemath/sage-prod/files/10654698/12415_framework.patch.gz), [attachment: 12415_doc.patch](https://github.com/sagemath/sage-prod/files/10654702/12415_doc.patch.gz)
+**Apply** [attachment: 12415_stderr_vs_51b5.patch](https://github.com/sagemath/sage/files/ticket12415/12415_stderr_vs_51b5.patch.gz), [attachment: 12415_framework.patch](https://github.com/sagemath/sage-prod/files/10654698/12415_framework.patch.gz), [attachment: 12415_framework_fixes.patch](https://github.com/sagemath/sage/files/ticket12415/12415_framework_fixes.patch.gz), [attachment: 12415_doctest_fixes.patch](https://github.com/sagemath/sage-prod/files/10654706/12415_doctest_fixes.patch.gz), [attachment: 12415_more_doctest_fixes.patch](https://github.com/sagemath/sage/files/ticket12415/12415_more_doctest_fixes.patch.gz), [attachment: 12415_doc.patch](https://github.com/sagemath/sage-prod/files/10654702/12415_doc.patch.gz)

 **Apply** [attachment: 12415_script.patch](https://github.com/sagemath/sage-prod/files/10654697/12415_script.patch.gz) to the scripts repo

-**Apply** [attachment: 12415_sagenb_fixes_vs_11080.patch](https://github.com/sagemath/sage/files/ticket12415/12415_sagenb_fixes_vs_11080.patch.gz) to the sagenb repo (fixes doctest errors) or wait for #13121
+**Apply** [attachment: 12415_spkg_bin_sage.patch](https://github.com/sagemath/sage-prod/files/10654696/12415_spkg_bin_sage.patch.gz) to the root repo

-**Apply** [attachment: 12415_spkg_bin_sage.patch](https://github.com/sagemath/sage-prod/files/10654696/12415_spkg_bin_sage.patch.gz) to the root repo

roed314 commented 11 years ago

Changed dependencies from #13147,#13146, #13145, #12723, #12392, #12393, #12395, #12396, #12397, #12381, #12382, #12383, #12384, #11871, #13195 to #13147,#13146, #13145, #12723, #12392, #12393, #12395, #12396, #12397, #12381, #12382, #12383, #12384, #11871, #13195, #13121

roed314 commented 11 years ago

comment:120

Alright, I fixed a bunch of bugs in the parsing of files. I added two new patches since the old ones were getting big. Sorry for the huge size of this ticket: some of the changes in attachment: 12415_more_doctest_fixes.patch can be split off into new tickets, but many of them need to be made along with the switch to the new testing code.

roed314 commented 11 years ago

comment:121

Here's my current status on the interrupt.pyx problem.

It occurs only if interrupt.pyx is not the first file being tested (so sage -t rings/big_oh.py tests/interrupt.pyx will show it for example)
The segfault is actually caused by the test at line 694 of interrupt.pyx: sage: test_sig_block()

I tried looking into the signal handling code but got lost. Any ideas?

roed314 commented 11 years ago

Changed work issues from intermittent interrupt.pyx failure to interrupt.pyx failure

jdemeyer commented 11 years ago

comment:123

I can have a look at the interrupt code, but not now...

roed314 commented 11 years ago

comment:124

Fair enough. Do you have any idea when you might have a chance? Or any suggestions for debugging it myself?

jhpalmieri commented 11 years ago

Description changed:

--- 
+++ 
@@ -12,7 +12,7 @@

 and does most of the above (plus more which has been moved to followup tickets #12720 and #12722)

-**Apply** [attachment: 12415_stderr_vs_51b5.patch](https://github.com/sagemath/sage/files/ticket12415/12415_stderr_vs_51b5.patch.gz), [attachment: 12415_framework.patch](https://github.com/sagemath/sage-prod/files/10654698/12415_framework.patch.gz), [attachment: 12415_framework_fixes.patch](https://github.com/sagemath/sage/files/ticket12415/12415_framework_fixes.patch.gz), [attachment: 12415_doctest_fixes.patch](https://github.com/sagemath/sage-prod/files/10654706/12415_doctest_fixes.patch.gz), [attachment: 12415_more_doctest_fixes.patch](https://github.com/sagemath/sage/files/ticket12415/12415_more_doctest_fixes.patch.gz), [attachment: 12415_doc.patch](https://github.com/sagemath/sage-prod/files/10654702/12415_doc.patch.gz)
+**Apply** [attachment: 12415_stderr_vs_51b5.patch](https://github.com/sagemath/sage/files/ticket12415/12415_stderr_vs_51b5.patch.gz), [attachment: 12415_framework.patch](https://github.com/sagemath/sage-prod/files/10654698/12415_framework.patch.gz), [attachment: 12415_framework_fixes.patch](https://github.com/sagemath/sage/files/ticket12415/12415_framework_fixes.patch.gz), [attachment: 12415_doctest_fixes-rebased.patch](https://github.com/sagemath/sage/files/ticket12415/12415_doctest_fixes-rebased.patch.gz), [attachment: 12415_more_doctest_fixes.patch](https://github.com/sagemath/sage/files/ticket12415/12415_more_doctest_fixes.patch.gz), [attachment: 12415_doc.patch](https://github.com/sagemath/sage-prod/files/10654702/12415_doc.patch.gz)

 **Apply** [attachment: 12415_script.patch](https://github.com/sagemath/sage-prod/files/10654697/12415_script.patch.gz) to the scripts repo

jhpalmieri commented 11 years ago

comment:126

I have no idea what's going on, but to follow up on David's comment: on an OS X 10.8 machine, if I run sage -t rings/big_oh.py tests/interrupt.pyx, I get an error at line 744. If I make either of the following two changes, the error goes away:

diff --git a/sage/tests/interrupt.pyx b/sage/tests/interrupt.pyx
--- a/sage/tests/interrupt.pyx
+++ b/sage/tests/interrupt.pyx
@@ -691,8 +691,6 @@ def test_sig_block(long delay = DEFAULT_
     TESTS::

         sage: from sage.tests.interrupt import *
-        sage: test_sig_block()
-        42
     """
     cdef volatile_int v = 0

or

diff --git a/sage/tests/interrupt.pyx b/sage/tests/interrupt.pyx
--- a/sage/tests/interrupt.pyx
+++ b/sage/tests/interrupt.pyx
@@ -741,9 +741,6 @@ def test_signal_during_malloc(long delay

     TESTS::

-        sage: from sage.tests.interrupt import *
-        sage: for i in range(4):  # Several times to reduce chances of false positive
-        ...       test_signal_during_malloc()
     """
     signal_after_delay(SIGINT, delay)
     try:

For the second one, I still get the error if I keep line 744,

sage: from sage.tests.interrupt import *

but remove the actual doctest. Does this have any significance for anyone?

roed314 commented 11 years ago

comment:127

The problem seems to be caused by

sage: test_sig_block()
42

but it only shows up if enough time is taken up by later tests. So you can delete

sage: from sage.tests.interrupt import *
sage: for i in range(4):  # Several times to reduce chances of false positive
...       test_signal_during_malloc()

and add something else, and you still get the error.

roed314 commented 11 years ago

comment:128

By commenting out most of the tests in that file, I can get the following error message:

Fatal Python error: GC object already tracked

I still get a SIGABRT, but now the bad exit code is -6 rather than -11.

roed314 commented 11 years ago

comment:129

Alright. I have a minimal example that doesn't rely on the doctesting framework. Of course, I still don't know how to resolve the issue, but at least it should be easier to work on. See #13748.

roed314 commented 11 years ago

Changed dependencies from #13147,#13146, #13145, #12723, #12392, #12393, #12395, #12396, #12397, #12381, #12382, #12383, #12384, #11871, #13195, #13121 to #13147,#13146, #13145, #12723, #12392, #12393, #12395, #12396, #12397, #12381, #12382, #12383, #12384, #11871, #13195, #13121,#13748

jhpalmieri commented 11 years ago

comment:131

If the issue isn't related to the new doctesting framework, but is a pre-existing condition revealed by it, then it shouldn't be a dependency, but a separate issue. If that was the only issue with this ticket, then this ticket could be merged, right?

roed314 commented 11 years ago

comment:132

Replying to @jhpalmieri:

If the issue isn't related to the new doctesting framework, but is a pre-existing condition revealed by it, then it shouldn't be a dependency, but a separate issue. If that was the only issue with this ticket, then this ticket could be merged, right?

I have no problem with that interpretation. But in some sense it is related to the new doctesting framework since we weren't using Python's multiprocessing module before.

So, does anyone want to give this ticket a positive review?

jdemeyer commented 11 years ago

Description changed:

--- 
+++ 
@@ -12,7 +12,7 @@

 and does most of the above (plus more which has been moved to followup tickets #12720 and #12722)

-**Apply** [attachment: 12415_stderr_vs_51b5.patch](https://github.com/sagemath/sage/files/ticket12415/12415_stderr_vs_51b5.patch.gz), [attachment: 12415_framework.patch](https://github.com/sagemath/sage-prod/files/10654698/12415_framework.patch.gz), [attachment: 12415_framework_fixes.patch](https://github.com/sagemath/sage/files/ticket12415/12415_framework_fixes.patch.gz), [attachment: 12415_doctest_fixes-rebased.patch](https://github.com/sagemath/sage/files/ticket12415/12415_doctest_fixes-rebased.patch.gz), [attachment: 12415_more_doctest_fixes.patch](https://github.com/sagemath/sage/files/ticket12415/12415_more_doctest_fixes.patch.gz), [attachment: 12415_doc.patch](https://github.com/sagemath/sage-prod/files/10654702/12415_doc.patch.gz)
+**Apply** [attachment: 12415_stderr_vs_51b5.patch](https://github.com/sagemath/sage/files/ticket12415/12415_stderr_vs_51b5.patch.gz), [attachment: 12415_framework.patch](https://github.com/sagemath/sage-prod/files/10654698/12415_framework.patch.gz), [attachment: 12415_framework_fixes.patch](https://github.com/sagemath/sage/files/ticket12415/12415_framework_fixes.patch.gz), [attachment: 12415_doctest_fixes.patch](https://github.com/sagemath/sage-prod/files/10654706/12415_doctest_fixes.patch.gz), [attachment: 12415_doc.patch](https://github.com/sagemath/sage-prod/files/10654702/12415_doc.patch.gz)

 **Apply** [attachment: 12415_script.patch](https://github.com/sagemath/sage-prod/files/10654697/12415_script.patch.gz) to the scripts repo

jdemeyer commented 11 years ago

comment:134

Rebased and merged the two doctest patches.

jdemeyer commented 11 years ago

Description changed:

--- 
+++ 
@@ -12,7 +12,7 @@

 and does most of the above (plus more which has been moved to followup tickets #12720 and #12722)

-**Apply** [attachment: 12415_stderr_vs_51b5.patch](https://github.com/sagemath/sage/files/ticket12415/12415_stderr_vs_51b5.patch.gz), [attachment: 12415_framework.patch](https://github.com/sagemath/sage-prod/files/10654698/12415_framework.patch.gz), [attachment: 12415_framework_fixes.patch](https://github.com/sagemath/sage/files/ticket12415/12415_framework_fixes.patch.gz), [attachment: 12415_doctest_fixes.patch](https://github.com/sagemath/sage-prod/files/10654706/12415_doctest_fixes.patch.gz), [attachment: 12415_doc.patch](https://github.com/sagemath/sage-prod/files/10654702/12415_doc.patch.gz)
+**Apply** [attachment: 12415_stderr_vs_51b5.patch](https://github.com/sagemath/sage/files/ticket12415/12415_stderr_vs_51b5.patch.gz), [attachment: 12415_framework.patch](https://github.com/sagemath/sage-prod/files/10654698/12415_framework.patch.gz), [attachment: 12415_doctest_fixes.patch](https://github.com/sagemath/sage-prod/files/10654706/12415_doctest_fixes.patch.gz), [attachment: 12415_doc.patch](https://github.com/sagemath/sage-prod/files/10654702/12415_doc.patch.gz)

 **Apply** [attachment: 12415_script.patch](https://github.com/sagemath/sage-prod/files/10654697/12415_script.patch.gz) to the scripts repo

jdemeyer commented 11 years ago

comment:136

Merged the two framework patches, removed the change to sage/tests/interrupt.pyx

jdemeyer commented 11 years ago

comment:137

I still don't like the DocTestWorker.annihilate() method which parses ps output to figure out which processes to kill.

roed314 commented 11 years ago

comment:138

Robert wrote that code. :-) I think the issue is that some doctests may spawn new processes that can survive their death. Do you have an alternate approach to suggest?

I'm very happy that you found the problem in #13748; I will try to take a look at this while I'm at the Joint meetings next week. In general I have very little time this month....

jdemeyer commented 11 years ago

Changed dependencies from #13147,#13146, #13145, #12723, #12392, #12393, #12395, #12396, #12397, #12381, #12382, #12383, #12384, #11871, #13195, #13121,#13748 to #13147,#13146, #13145, #12723, #12392, #12393, #12395, #12396, #12397, #12381, #12382, #12383, #12384, #11871, #13195, #13121,#13748, #13899

jdemeyer commented 11 years ago

Changed work issues from interrupt.pyx failure to none

jdemeyer commented 11 years ago

comment:140

There are various new doctest errors, all of them look easy to fix. But also two errors in the doctest framework itself:

sage -t --long devel/sage/sage/doctest/sources.py
**********************************************************************
File "devel/sage/sage/doctest/sources.py", line 177, in sage.doctest.sources.DocTestSource._process_doc
Failed example:
    doctests == manual_doctests
Expected:
    True
Got:
    False
**********************************************************************
File "devel/sage/sage/doctest/sources.py", line 655, in sage.doctest.sources.FileDocTestSource._test_enough_doctests
Failed example:
    for path, dirs, files in itertools.chain(os.walk(sage_loc), os.walk(doc_loc)): # long time
        path = os.path.relpath(path)
        for F in files:
            _, ext = os.path.splitext(F)
            if ext in ('.py', '.pyx', '.sage', '.spyx', '.rst', '.tex'):
                filename = os.path.join(path, F)
                FDS = FileDocTestSource(filename, True, True, True, False)
                FDS._test_enough_doctests(verbose=False)
Expected:
    There are 3 unexpected tests being run in .../sage/doctest/parsing.py
    There are 1 tests in .../sage/ext/c_lib.pyx that are not being run
    There are 2 tests in .../sage/server/notebook/worksheet.py that are not being run
    There are 5 tests in .../doc/en/tutorial/interfaces.rst that are not being run
Got:
    There are 2 tests in devel/sage/sage/server/notebook/worksheet.py that are not being run
    There are 2 unexpected tests being run in devel/sage/sage/misc/cython.py
    There are 1 tests in devel/sage/sage/ext/c_lib.pyx that are not being run
    There are 3 unexpected tests being run in devel/sage/sage/doctest/parsing.py
    There are 5 tests in devel/sage/doc/en/tutorial/interfaces.rst that are not being run
    There are 1130 tests in devel/sage/doc/output/latex/en/reference/reference.tex that are not being run
    There are 51 tests in devel/sage/doc/output/latex/en/tutorial/SageTutorial.tex that are not being run
    There are 4 tests in devel/sage/doc/output/latex/en/thematic_tutorials/thematic_tutorials.tex that are not being run
    There are 6 tests in devel/sage/doc/output/latex/en/prep/prep_tutorials.tex that are not being run
    There are 2 tests in devel/sage/doc/output/latex/en/numerical_sage/numerical_sage.tex that are not being run
    There are 15 tests in devel/sage/doc/output/latex/en/developer/developer.tex that are not being run
    There are 16 tests in devel/sage/doc/output/latex/en/constructions/constructions.tex that are not being run
    There are 51 tests in devel/sage/doc/output/latex/ru/tutorial/SageTutorial_ru.tex that are not being run
    There are 51 tests in devel/sage/doc/output/latex/fr/tutorial/tutorial-fr.tex that are not being run
    There are 51 tests in devel/sage/doc/output/latex/de/tutorial/SageTutorial-de.tex that are not being run
**********************************************************************

Also, the directory devel/sagenb/sagenb/data should probably not be tested, as there are failures because of TAB characters in the MathJax rst documentation.

jdemeyer commented 11 years ago

Changed dependencies from #13147,#13146, #13145, #12723, #12392, #12393, #12395, #12396, #12397, #12381, #12382, #12383, #12384, #11871, #13195, #13121,#13748, #13899 to #13147,#13146, #13145, #12723, #12392, #12393, #12395, #12396, #12397, #12381, #12382, #12383, #12384, #11871, #13195, #13121, #13748, #13899

jdemeyer commented 11 years ago

comment:142

Do you guys consider it a feature or a bug that

"""
sage: 1e16     # relative tol 1e-10
10^16
"""

doesn't work? See #12815.

jdemeyer commented 11 years ago

comment:143

What I'm missing in this patchbomb is a high-level overview of how everything works. To me, it looks like a large lump of code (which seems to work pretty well) which is hard to understand.

jdemeyer commented 11 years ago

comment:144

Typo:

Comparison ist just comparison of the underlying lists.

jdemeyer commented 11 years ago

comment:145

Replying to @roed314:

Robert wrote that code. :-) I think the issue is that some doctests may spawn new processes that can survive their death. Do you have an alternate approach to suggest?

There is some commented-out code mentioning process groups, it would be a good idea to make that work.

But I'm also a bit lost on how the fork()ing actually works and how many Sage sessions are involved.

sagemath / sage

Update doctesting framework #12415