Update doctesting framework

robertwb commented 12 years ago

There are several improvements that would be good to make, including but not limited to:

Forking rather than starting up a new Sage each time.
Customizable comparison rather than mutating doctests into new doctests (e.g. followup to #10952).
Eliminate need to write temporary files for every doctested file.
Constant headaches caused by the difference in doctesting library vs. non-library files.
The ability to drop into a debugger if a doctest raises an exception.

Robert's code is at

http://code.google.com/p/sagemath-timer/

Apply attachment: 12415_framework.patch, attachment: 12415_doctest_fixes.patch, attachment: 12415_doc.patch, attachment: 12415_review.patch, attachment: 12415_test.patch, attachment: 12415_review_review.patch, attachment: 12415_review3.patch, attachment: 12415_manifest.patch, attachment: 12415_rebase_58.patch

Apply attachment: 12415_script.patch and attachment: 12415_script_review.patch to the scripts repo

Apply attachment: 12415_spkg_bin_sage.patch to the root repo

For follow-up or other doctest-related tickets see #11337.

Depends on #13147 Depends on #13146 Depends on #13145 Depends on #12723 Depends on #12392 Depends on #12393 Depends on #12395 Depends on #12396 Depends on #12397 Depends on #12381 Depends on #12382 Depends on #12383 Depends on #12384 Depends on #11871 Depends on #13195 Depends on #13121 Depends on #13748 Depends on #13899 Depends on #12719 Depends on #5155 Depends on #14070 Depends on #14079 Depends on #14150 Depends on #14158 Depends on #14182 Depends on #14184 Depends on #14054 Depends on #14063 Depends on #13605 Depends on #14111 Depends on #14254 Depends on #14242 Depends on #14253

CC: @kini @ohanar @jhpalmieri

Component: doctest framework

Author: David Roe, Robert Bradshaw, Jeroen Demeyer

Reviewer: Jeroen Demeyer, David Roe

Merged: sage-5.9.beta0

Issue created by migration from https://trac.sagemath.org/ticket/12415

roed314 commented 11 years ago

comment:192

Replying to @jdemeyer:

Replying to @roed314:

I agree that the line describing which file is being tested should come at the beginning.

OK, so you agree with
sage -t file.py
<messages>
    [1 test, 1 failure, 1.7 s]
I would move the part of report() which prints the "sage -t ..." header to a new function report_start() (for example) and then correct the logic in forker.py to call that function at the appropriate time. Does that sound good?

Sounds good. Presumably it will get printed to the msgfile before all of the other messages and then actually printed to the screen in whatever way is appropriate.

jdemeyer commented 11 years ago

Description changed:

--- 
+++ 
@@ -12,7 +12,7 @@

 and does most of the above (plus more which has been moved to followup tickets #12720 and #12722)

-**Apply** [attachment: 12415_framework.patch](https://github.com/sagemath/sage-prod/files/10654698/12415_framework.patch.gz), [attachment: 12415_doctest_fixes.patch](https://github.com/sagemath/sage-prod/files/10654706/12415_doctest_fixes.patch.gz), [attachment: 12415_doc.patch](https://github.com/sagemath/sage-prod/files/10654702/12415_doc.patch.gz), [attachment: 12415_review.patch](https://github.com/sagemath/sage-prod/files/10654704/12415_review.patch.gz)
+**Apply** [attachment: 12415_framework.patch](https://github.com/sagemath/sage-prod/files/10654698/12415_framework.patch.gz), [attachment: 12415_doctest_fixes.patch](https://github.com/sagemath/sage-prod/files/10654706/12415_doctest_fixes.patch.gz), [attachment: 12415_doctest_review.patch](https://github.com/sagemath/sage-prod/files/10654699/12415_doctest_review.patch.gz), [attachment: 12415_doc.patch](https://github.com/sagemath/sage-prod/files/10654702/12415_doc.patch.gz), [attachment: 12415_review.patch](https://github.com/sagemath/sage-prod/files/10654704/12415_review.patch.gz)

 **Apply** [attachment: 12415_script.patch](https://github.com/sagemath/sage-prod/files/10654697/12415_script.patch.gz) to the scripts repo

jdemeyer commented 11 years ago

comment:193

Could you please review attachment: 12415_doctest_review.patch? It contains all my reviewer changes for doctests in the Sage library (i.e. all changes not for the doctest framework).

roed314 commented 11 years ago

comment:194

Replying to @jdemeyer:

Could you please review attachment: 12415_doctest_review.patch? It contains all my reviewer changes for doctests in the Sage library (i.e. all changes not for the doctest framework).

I'm running tests on OS X 10.6.8 with sage 5.7.beta1.

Your changes to sage.calculus.test_sympy break doctests on my machine: they were correct before the change. Presumably these results are system dependent and should be changed somehow.
I get a bad exit in sage.misc.interpreter at line 132 (in get_test_shell()): it prints out a bunch of help text from IPython, ending with

[SageTerminalApp] Bad config encountered during initialization:
[SageTerminalApp] Unrecognized flag: '--all'

I'm not really sure what's happening here: perhaps it's related to the IPython upgrade?

The same thing happens in sage.misc.sage_extension at line 22.
There's still an error in sagenb.notebook.worksheet line 196: sagenb.notebook has no attribute misc. I've seen this before and it should be easy to fix.
There are problems running some tests in sage.doctest.control that may go away with your new code. If not I have fixes for them.

How is attachment: 12415_review.patch coming along?

jdemeyer commented 11 years ago

comment:195

Replying to @roed314:

Replying to @jdemeyer:

Could you please review attachment: 12415_doctest_review.patch? It contains all my reviewer changes for doctests in the Sage library (i.e. all changes not for the doctest framework).

I'm running tests on OS X 10.6.8 with sage 5.7.beta1.

Your changes to sage.calculus.test_sympy break doctests on my machine: they were correct before the change. Presumably these results are system dependent and should be changed somehow.

The difference is between plain ASCII characters and some fancy Unicode characters for the display. Not sure what to do with this.

I get a bad exit in sage.misc.interpreter at line 132 (in get_test_shell()):

Did you apply #14070?

There's still an error in sagenb.notebook.worksheet line 196: sagenb.notebook has no attribute misc. I've seen this before and it should be easy to fix.

Good, then please fix it.

How is attachment: 12415_review.patch coming along?

I think it works well for normal testing. I still have to think about --debug and things like --gdb. It seems it will be necessary to have a serial mode for example for --gdb, so I have to put that back in. I would also like to force --serial when --debug is used, mainly to simplify the code.

But first I'm going to look at #14079 on OS X.

roed314 commented 11 years ago

comment:196

Your changes to sage.calculus.test_sympy break doctests on my machine: they were correct before the change. Presumably these results are system dependent and should be changed somehow.

The difference is between plain ASCII characters and some fancy Unicode characters for the display. Not sure what to do with this.

I fixed it by temporarily turning off unicode in pretting printing. I updated the patch with this change.

I get a bad exit in sage.misc.interpreter at line 132 (in get_test_shell()):

Did you apply #14070?

Nope: applying #14070 fixed the problem.

There's still an error in sagenb.notebook.worksheet line 196: sagenb.notebook has no attribute misc. I've seen this before and it should be easy to fix.

Good, then please fix it.

Okay, the easy fix I remembered is to add a line to sagenb.notebook.worksheet:

         EXAMPLES: We test the constructor via an indirect doctest::

             sage: nb = sagenb.notebook.notebook.Notebook(tmp_dir()+'.sagenb')
+            sage: import sagenb.notebook.misc
             sage: sagenb.notebook.misc.notebook = nb
             sage: W = nb.create_new_worksheet('Test with unicode ěščřžýáíéďĎ', 'admin')
             sage: W

I haven't made any changes to sagenb for a long time. I assume this would require a new spkg?

How is attachment: 12415_review.patch coming along?

I think it works well for normal testing. I still have to think about --debug and things like --gdb. It seems it will be necessary to have a serial mode for example for --gdb, so I have to put that back in. I would also like to force --serial when --debug is used, mainly to simplify the code.

Makes sense.

But first I'm going to look at #14079 on OS X.

Good luck.

jdemeyer commented 11 years ago

comment:197

Replying to @roed314:

I haven't made any changes to sagenb for a long time. I assume this would require a new spkg?

No, sagenb is now officially a separate project. I have sent a patch to "upstream" sagenb.

jdemeyer commented 11 years ago

Description changed:

--- 
+++ 
@@ -14,7 +14,7 @@

 **Apply** [attachment: 12415_framework.patch](https://github.com/sagemath/sage-prod/files/10654698/12415_framework.patch.gz), [attachment: 12415_doctest_fixes.patch](https://github.com/sagemath/sage-prod/files/10654706/12415_doctest_fixes.patch.gz), [attachment: 12415_doctest_review.patch](https://github.com/sagemath/sage-prod/files/10654699/12415_doctest_review.patch.gz), [attachment: 12415_doc.patch](https://github.com/sagemath/sage-prod/files/10654702/12415_doc.patch.gz), [attachment: 12415_review.patch](https://github.com/sagemath/sage-prod/files/10654704/12415_review.patch.gz)

-**Apply** [attachment: 12415_script.patch](https://github.com/sagemath/sage-prod/files/10654697/12415_script.patch.gz) to the scripts repo
+**Apply** [attachment: 12415_script.patch](https://github.com/sagemath/sage-prod/files/10654697/12415_script.patch.gz) and [attachment: 12415_script_review.patch](https://github.com/sagemath/sage-prod/files/10654700/12415_script_review.patch.gz) to the scripts repo

 **Apply** [attachment: 12415_spkg_bin_sage.patch](https://github.com/sagemath/sage-prod/files/10654696/12415_spkg_bin_sage.patch.gz) to the root repo

jdemeyer commented 11 years ago

comment:198

New major update of my code. Now adds support for --debug.

I removed SagePdb since the motivation of the output-redirection in the Python doctesting framework seems to be doctesting Pdb itself, not running a debugger inside the doctesting framework. For the new doctesting framework, we just need to stop spoofing before entering the debugger and start spoofing after the debugger is finished and that's all.

jdemeyer commented 11 years ago

Changed author from David Roe, Robert Bradshaw to David Roe, Robert Bradshaw, Jeroen Demeyer

jdemeyer commented 11 years ago

comment:199

I fixed a few doctests errors, now the only remaining failure is

sage -t --long devel/sage/sage/doctest/sources.py
**********************************************************************
File "devel/sage/sage/doctest/sources.py", line 653, in sage.doctest.sources.FileDocTestSource._test_enough_doctests
Failed example:
    for path, dirs, files in itertools.chain(os.walk(sage_loc), os.walk(doc_loc)): # long time
        path = os.path.relpath(path)
        for F in files:
            _, ext = os.path.splitext(F)
            if ext in ('.py', '.pyx', '.sage', '.spyx', '.rst', '.tex'):
                filename = os.path.join(path, F)
                FDS = FileDocTestSource(filename, True, True, True, False)
                FDS._test_enough_doctests(verbose=False)
Expected:
    There are 3 unexpected tests being run in .../sage/doctest/parsing.py
    There are 1 tests in .../sage/ext/c_lib.pyx that are not being run
    There are 2 tests in .../sage/server/notebook/worksheet.py that are not being run
    There are 5 tests in .../doc/en/tutorial/interfaces.rst that are not being run
Got:
    doctest:229: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
    There are 2 tests in devel/sage/sage/server/notebook/worksheet.py that are not being run
    There are 2 unexpected tests being run in devel/sage/sage/misc/cython.py
    There are 9 tests in devel/sage/sage/graphs/graph_plot.py that are not being run
    There are 1 tests in devel/sage/sage/ext/c_lib.pyx that are not being run
    There are 3 unexpected tests being run in devel/sage/sage/doctest/parsing.py
    There are 5 tests in devel/sage/doc/en/tutorial/interfaces.rst that are not being run
    There are 1152 tests in devel/sage/doc/output/latex/en/reference/reference.tex that are not being run
    There are 51 tests in devel/sage/doc/output/latex/en/tutorial/SageTutorial.tex that are not being run
    There are 4 tests in devel/sage/doc/output/latex/en/thematic_tutorials/thematic_tutorials.tex that are not being run
    There are 6 tests in devel/sage/doc/output/latex/en/prep/prep_tutorials.tex that are not being run
    There are 2 tests in devel/sage/doc/output/latex/en/numerical_sage/numerical_sage.tex that are not being run
    There are 15 tests in devel/sage/doc/output/latex/en/developer/developer.tex that are not being run
    There are 16 tests in devel/sage/doc/output/latex/en/constructions/constructions.tex that are not being run
    There are 51 tests in devel/sage/doc/output/latex/ru/tutorial/SageTutorial_ru.tex that are not being run
    There are 51 tests in devel/sage/doc/output/latex/fr/tutorial/tutorial-fr.tex that are not being run
    There are 51 tests in devel/sage/doc/output/latex/de/tutorial/SageTutorial-de.tex that are not being run
**********************************************************************

And there are still messages about TABs in sagenb/data.

jdemeyer commented 11 years ago

comment:200

I have been thinking about the encoding warnings (and encodings in doctests in general). I think there are two options:

either we ignore the Cython warnings and continue working with pure strings,
or we make the doctester completely Unicode-aware.

I don't really see any middle ground. I'm inclined to go for the first option initially, but eventually migrating to the second option in a different ticket.

But I'm open to opinions on this.

roed314 commented 11 years ago

comment:201

Replying to @jdemeyer:

I have been thinking about the encoding warnings (and encodings in doctests in general). I think there are two options:

either we ignore the Cython warnings and continue working with pure strings,

or we make the doctester completely Unicode-aware.

I don't really see any middle ground. I'm inclined to go for the first option initially, but eventually migrating to the second option in a different ticket.

But I'm open to opinions on this.

Sounds like a good plan to me. I prefer to avoid unicode when I can.

jdemeyer commented 11 years ago

comment:202

With these patches + the patched sagenb (https://github.com/sagemath/sagenb/pull/84), I get just one sporadic failure:

sage -t --long devel/sage/sage/homology/simplicial_complex.py
**********************************************************************
File "devel/sage/sage/homology/simplicial_complex.py", line 2188, in sage.homology.simplicial_complex.SimplicialComplex.is_cohen_macaulay
Failed example:
    S.is_cohen_macaulay(ncpus=3)
Expected:
    False
Got:
    Exception OSError: (10, 'No child processes') in <generator object __call__ at 0x646f1e0> ignored
    False
**********************************************************************

I have not investigated, but I assume it must be some strange interaction between the DocTestWorker processes and the cohen_macaulay subprocesses.

Since bsd.math has been down for a while (now it is back up), I haven't been able to look into the OS X issues.

I guess the only remaining issue is --gdb (and --valgrind...) support.

jdemeyer commented 11 years ago

Changed dependencies from #13147,#13146, #13145, #12723, #12392, #12393, #12395, #12396, #12397, #12381, #12382, #12383, #12384, #11871, #13195, #13121, #13748, #13899, #12719, #5155, #14070, #14079 to #13147,#13146, #13145, #12723, #12392, #12393, #12395, #12396, #12397, #12381, #12382, #12383, #12384, #11871, #13195, #13121, #13748, #13899, #12719, #5155, #14070, #14079, #14150

jdemeyer commented 11 years ago

Changed reviewer from Jeroen Demeyer to Jeroen Demeyer, David Roe

jdemeyer commented 11 years ago

Description changed:

--- 
+++ 
@@ -10,11 +10,10 @@

 http://code.google.com/p/sagemath-timer/

-and does most of the above (plus more which has been moved to followup tickets #12720 and #12722)
-
 **Apply** [attachment: 12415_framework.patch](https://github.com/sagemath/sage-prod/files/10654698/12415_framework.patch.gz), [attachment: 12415_doctest_fixes.patch](https://github.com/sagemath/sage-prod/files/10654706/12415_doctest_fixes.patch.gz), [attachment: 12415_doctest_review.patch](https://github.com/sagemath/sage-prod/files/10654699/12415_doctest_review.patch.gz), [attachment: 12415_doc.patch](https://github.com/sagemath/sage-prod/files/10654702/12415_doc.patch.gz), [attachment: 12415_review.patch](https://github.com/sagemath/sage-prod/files/10654704/12415_review.patch.gz)

 **Apply** [attachment: 12415_script.patch](https://github.com/sagemath/sage-prod/files/10654697/12415_script.patch.gz) and [attachment: 12415_script_review.patch](https://github.com/sagemath/sage-prod/files/10654700/12415_script_review.patch.gz) to the scripts repo

 **Apply** [attachment: 12415_spkg_bin_sage.patch](https://github.com/sagemath/sage-prod/files/10654696/12415_spkg_bin_sage.patch.gz) to the root repo

+Follow-up tickets: #12720, #12722, #14152, #14153.

roed314 commented 11 years ago

comment:206

So is this ready for review by me?

jdemeyer commented 11 years ago

comment:207

Parallel testing still doesn't work on OS X. 1-process testing (-p 1) works.

roed314 commented 11 years ago

comment:208

Do you know why parallel testing doesn't work?

jdemeyer commented 11 years ago

Changed dependencies from #13147,#13146, #13145, #12723, #12392, #12393, #12395, #12396, #12397, #12381, #12382, #12383, #12384, #11871, #13195, #13121, #13748, #13899, #12719, #5155, #14070, #14079, #14150 to #13147,#13146, #13145, #12723, #12392, #12393, #12395, #12396, #12397, #12381, #12382, #12383, #12384, #11871, #13195, #13121, #13748, #13899, #12719, #5155, #14070, #14079, #14150, #14158

jdemeyer commented 11 years ago

comment:210

What's the point of running optional tests without optional "sage" tests? Because that would badly fail with code like

sage: a = some_object()
sage: a.some_standard_method()
sage: a.use_mypackage()  # optional: mypackage

(only the last line would be executed)

Are all doctests supposed to be written like

sage: a = some_object()
sage: a.some_standard_method()

sage: a = some_object()  # optional: mypackage
sage: a.use_mypackage()  # optional: mypackage

jdemeyer commented 11 years ago

Description changed:

--- 
+++ 
@@ -10,7 +10,7 @@

 http://code.google.com/p/sagemath-timer/

-**Apply** [attachment: 12415_framework.patch](https://github.com/sagemath/sage-prod/files/10654698/12415_framework.patch.gz), [attachment: 12415_doctest_fixes.patch](https://github.com/sagemath/sage-prod/files/10654706/12415_doctest_fixes.patch.gz), [attachment: 12415_doctest_review.patch](https://github.com/sagemath/sage-prod/files/10654699/12415_doctest_review.patch.gz), [attachment: 12415_doc.patch](https://github.com/sagemath/sage-prod/files/10654702/12415_doc.patch.gz), [attachment: 12415_review.patch](https://github.com/sagemath/sage-prod/files/10654704/12415_review.patch.gz)
+**Apply** [attachment: 12415_framework.patch](https://github.com/sagemath/sage-prod/files/10654698/12415_framework.patch.gz), [attachment: 12415_doctest_fixes.patch](https://github.com/sagemath/sage-prod/files/10654706/12415_doctest_fixes.patch.gz), [attachment: 12415_doctest_review.patch](https://github.com/sagemath/sage-prod/files/10654699/12415_doctest_review.patch.gz), [attachment: 12415_doc.patch](https://github.com/sagemath/sage-prod/files/10654702/12415_doc.patch.gz), [attachment: 12415_review.patch](https://github.com/sagemath/sage-prod/files/10654704/12415_review.patch.gz), [attachment: 12415_test.patch](https://github.com/sagemath/sage-prod/files/10654703/12415_test.patch.gz)

 **Apply** [attachment: 12415_script.patch](https://github.com/sagemath/sage-prod/files/10654697/12415_script.patch.gz) and [attachment: 12415_script_review.patch](https://github.com/sagemath/sage-prod/files/10654700/12415_script_review.patch.gz) to the scripts repo

jdemeyer commented 11 years ago

Changed dependencies from #13147,#13146, #13145, #12723, #12392, #12393, #12395, #12396, #12397, #12381, #12382, #12383, #12384, #11871, #13195, #13121, #13748, #13899, #12719, #5155, #14070, #14079, #14150, #14158 to #13147,#13146, #13145, #12723, #12392, #12393, #12395, #12396, #12397, #12381, #12382, #12383, #12384, #11871, #13195, #13121, #13748, #13899, #12719, #5155, #14070, #14079, #14150, #14158, #14182

roed314 commented 11 years ago

comment:213

Replying to @jdemeyer:

What's the point of running optional tests without optional "sage" tests? Because that would badly fail with code like
sage: a = some_object()
sage: a.some_standard_method()
sage: a.use_mypackage()  # optional: mypackage
(only the last line would be executed)

Are all doctests supposed to be written like
sage: a = some_object()
sage: a.some_standard_method()

sage: a = some_object()  # optional: mypackage
sage: a.use_mypackage()  # optional: mypackage

I agree that the current implementation is kinda dumb. But I could see wanting to run sage -t --all --optional gap and have only files with optional gap tests be run.

jdemeyer commented 11 years ago

comment:214

Replying to @roed314:

I agree that the current implementation is kinda dumb. But I could see wanting to run sage -t --all --optional gap and have only files with optional gap tests be run.

So this is basically an option waiting for a better future implementation?

roed314 commented 11 years ago

comment:215

Replying to @jdemeyer:

Replying to @roed314:

I agree that the current implementation is kinda dumb. But I could see wanting to run sage -t --all --optional gap and have only files with optional gap tests be run.

So this is basically an option waiting for a better future implementation?

Yeah, that's how I see it. I don't really run optional tests very often, but it seems like it could be useful to quickly run the tests that directly depend on gap when you update an spkg or something. But I'm not really attached to the particulars of the interface, so feel free to change it.

jhpalmieri commented 11 years ago

comment:216

A question about this new framework: at #13928, it was pointed out that if you (for whatever reason) put the Sage directory into a directory whose path contains "/." (like "/home/user/.Sage-stuff/sage-5.8.beta0/"), then it will refuse to doctest any files in the Sage library. Will that still be true with this new setup? Basically, I'm wondering if this ticket supersedes #13928.

jdemeyer commented 11 years ago

comment:217

Replying to @jhpalmieri:

A question about this new framework: at #13928, it was pointed out that if you (for whatever reason) put the Sage directory into a directory whose path contains "/." (like "/home/user/.Sage-stuff/sage-5.8.beta0/")

Hiding Sage from the boss?

then it will refuse to doctest any files in the Sage library. Will that still be true with this new setup?

No, see sage/doctest/control.py, line 456.

jhpalmieri commented 11 years ago

comment:218

Nathann has a reason for putting his entire Sage installation in a hidden directory. I can understand why we would skip testing files in directories within the Sage library that start with "." (like ".hg"), but why not just have that limitation and let the user put their Sage directory anywhere?

jdemeyer commented 11 years ago

Changed dependencies from #13147,#13146, #13145, #12723, #12392, #12393, #12395, #12396, #12397, #12381, #12382, #12383, #12384, #11871, #13195, #13121, #13748, #13899, #12719, #5155, #14070, #14079, #14150, #14158, #14182 to #13147,#13146, #13145, #12723, #12392, #12393, #12395, #12396, #12397, #12381, #12382, #12383, #12384, #11871, #13195, #13121, #13748, #13899, #12719, #5155, #14070, #14079, #14150, #14158, #14182, #14184

jdemeyer commented 11 years ago

comment:220

Replying to @jhpalmieri:

Nathann has a reason for putting his entire Sage installation in a hidden directory. I can understand why we would skip testing files in directories within the Sage library that start with "." (like ".hg"), but why not just have that limitation

I believe that's precisely what we're doing, unless I am mistaken.

jdemeyer commented 11 years ago

comment:221

Sorry John, I misread your original question.

jhpalmieri commented 11 years ago

comment:222

Okay, good, then I suggest that we close #13928 as a duplicate of this one, or at least close #13928 when this one is merged.

6bdad4c1-1e26-4f2f-a442-a01a2292c181 commented 11 years ago

comment:223

I can understand why we would skip testing files in directories within the Sage library that start with "." (like ".hg"), but why not just have that limitation

I believe that's precisely what we're doing, unless I am mistaken.

Sorry, but I don't get it O_o

After this patch is applied, will all files be skipped if SAGE_ROOT contains a /. somewhere, even if it is not a /.hg ?

Nathann

6bdad4c1-1e26-4f2f-a442-a01a2292c181 commented 11 years ago

comment:224

Okay, good, then I suggest that we close #13928 as a duplicate of this one, or at least close #13928 when this one is merged.

Gloops. That answers my question too :-)

Then I will set #13928 to "positive review/duplicate", as I can safely have faith in our release manager and this patch :-)

Nathann

jhpalmieri commented 11 years ago

comment:225

What goes wrong with parallel testing on OS X? I've installed the patches on an OS X 10.8.2 box, and running ./sage -tp devel/sage/sage/homology devel/sage/sage/parallel/ works fine, correctly detecting two cores, and completing testing faster than ./sage -t .... Using ./sage -tp 3 ... runs even slightly faster.

roed314 commented 11 years ago

comment:226

Replying to @jhpalmieri:

What goes wrong with parallel testing on OS X? I've installed the patches on an OS X 10.8.2 box, and running ./sage -tp devel/sage/sage/homology devel/sage/sage/parallel/ works fine, correctly detecting two cores, and completing testing faster than ./sage -t .... Using ./sage -tp 3 ... runs even slightly faster.

Jeroen said at #14079 that the parallel testing problem on OS X should be fixed now.

jdemeyer commented 11 years ago

comment:227

So far, OS X has not been my priority. I think it has to do with the way terminals are handled (try ./sage -t ... </dev/null 2>&1 | cat for example, completely bypassing terminal I/O)

jdemeyer commented 11 years ago

comment:228

Concerning terminals, I also noted (on Linux) that piping output to less (./sage -t ... | less) doesn't quite work properly.

jhpalmieri commented 11 years ago

comment:229

Without applying any of the patches here, if a user modifies their configuration, say via #14188, some doctests may fail. I assume the same will be true after applying the patches here. My question: should make ptest (etc.) run with the --nodotsage setting, while sage -t ... uses the user's configuration? Or should both use --nodotsage, or neither use it?

roed314 commented 11 years ago

comment:231

Replying to @jhpalmieri:

Without applying any of the patches here, if a user modifies their configuration, say via #14188, some doctests may fail. I assume the same will be true after applying the patches here. My question: should make ptest (etc.) run with the --nodotsage setting, while sage -t ... uses the user's configuration? Or should both use --nodotsage, or neither use it?

I would say that both should use it: I don't see why the user might need their custom configuration when doctesting.

jhpalmieri commented 11 years ago

comment:232

I have a request/suggestion: move DOCTEST_MODE out of plot.py and into a file in the doctest directory. Right now, importing from plot.py can be lead to circular imports, etc. Also, it never belonged in plot.py in the first place. Can we do that on this ticket, or should it be on a followup?

jpflori commented 11 years ago

comment:233

Replying to @roed314:

Do you ever use sage -t with valgrind? I'm not sure exactly what's wrong with sage -t --valgrind <FILENAME>.

It used to work and I used it at some point to deal with the nice random failures caused by the introduction of weak caches of #715. Note the optional spkg we currently propose is really old (and surely dysfunctional on most systems), but there's an update rotting at #13060. I wouldn't mind making valgrind working again a follow-up ticket as I'm not actively using it right now.

jdemeyer commented 11 years ago

comment:234

Concerning DOT_SAGE: it's not as easy as you think. The problem is that the doctesting framework needs DOT_SAGE in a few places, for example to store the timing information. So running

sage --nodotsage -t ...

is not an option.

Perhaps changing $IPYTHONDIR is sufficient? That could be done within the doctesting framework.

jdemeyer commented 11 years ago

Description changed:

--- 
+++ 
@@ -16,4 +16,4 @@

 **Apply** [attachment: 12415_spkg_bin_sage.patch](https://github.com/sagemath/sage-prod/files/10654696/12415_spkg_bin_sage.patch.gz) to the root repo

-Follow-up tickets: #12720, #12722, #14152, #14153.
+Follow-up tickets: #12720, #12722, #14152, #14153, #14203.

jdemeyer commented 11 years ago

comment:236

Replying to @jhpalmieri:

I have a request/suggestion: move DOCTEST_MODE out of plot.py and into a file in the doctest directory. Right now, importing from plot.py can be lead to circular imports, etc. Also, it never belonged in plot.py in the first place. Can we do that on this ticket, or should it be on a followup?

This ticket is already big enough, so I created the follow-up #14203.

jdemeyer commented 11 years ago

Attachment: 12415_doctest_review.patch.gz

jdemeyer commented 11 years ago

comment:237

Attachment: 12415_script_review.patch.gz

jdemeyer commented 11 years ago

Description changed:

--- 
+++ 
@@ -16,4 +16,4 @@

 **Apply** [attachment: 12415_spkg_bin_sage.patch](https://github.com/sagemath/sage-prod/files/10654696/12415_spkg_bin_sage.patch.gz) to the root repo

-Follow-up tickets: #12720, #12722, #14152, #14153, #14203.
+Follow-up or other doctest-related tickets: #12720, #12722, #13610, #13063, #14152, #14153, #14203, #12281.

sagemath / sage

Update doctesting framework #12415