jdemeyer
commented
11 years ago comment:1
I am working on a patch.
Closed jdemeyer closed 11 years ago
jdemeyer
commented
11 years ago I am working on a patch.
jdemeyer
commented
11 years ago Author: Jeroen Demeyer
jhpalmieri
commented
11 years ago In the line
pidfile = os.path.join(DOT_SAGE, 'tmp', 'cleaner-%s.pid'%HOSTNAME)would it make sense to use 'temp' instead of 'tmp', so all of the sage-cleaner information is in the same directory?
(Is there any good reason for having both directories tmp and temp? At some point, we should clean this up.)
83660e46-0051-498b-a8c1-f7a7bd232b5a
commented
11 years ago THERE ARE STILL SOME INSTANCES OF UPPERCASE SAGE.
I don't like the mixture of tmp, temp, TMP and TEMP or whatever.
temp and tmp shouldn't be hardcoded.
"Deleting .../spawned_processes" gets printed even if .../spawned_processes is not a directory (not e), and also rmtree() is attempted, ignoring any OSError.
I don't like "%s" % i where i is (or should be) an int.
Command line option handling is certainly suboptimal. (argv[1] could be a float as well, there's no usage, and additional parameters simply get ignored.)
Likewise, if DOT_SAGE isn't set, an ugly KeyError is raised, instead of printing a meaningful message.
83660e46-0051-498b-a8c1-f7a7bd232b5a
commented
11 years ago Changed keywords from none to orphans
83660e46-0051-498b-a8c1-f7a7bd232b5a
commented
11 years ago Reviewer: Punarbasu Purkayastha, Leif Leonhardy
83660e46-0051-498b-a8c1-f7a7bd232b5a
commented
11 years ago Ooops.
83660e46-0051-498b-a8c1-f7a7bd232b5a
commented
11 years ago Changed reviewer from Punarbasu Purkayastha, Leif Leonhardy to John Palmieri, Leif Leonhardy
jdemeyer
commented
11 years ago Leif: the subject of this ticket is not "Fix all possible issues in sage-cleaner", the goal is to just make it work.
jdemeyer
commented
11 years ago Replying to @jhpalmieri:
Is there any good reason for having both directories
tmpandtemp?
Absolutely not, but at least the directory used in sage-cleaner should match the directory used in the Sage library.
jhpalmieri
commented
11 years ago I am having a problem with this patch, but I don't know why. I am repeatedly running sage -tp 2 devel/sage/sage/homology/s*. Without the patch applied, everything is fine. With the patch, after the first successful run, I get this (I also removed the &>/dev/null after the call to sage-cleaner in spkg/bin/sage):
$ sage -tp 2 devel/sage/sage/homology/s*
SAGE_TMP_ROOT = /Users/palmieri/.sage/temp/jpalmieri538
Starting sage-cleaner
Checking PIDs []
sage-cleaner is finished
Running doctests with ID 2013-03-20-11-16-30-613ad8e5.
Sorting sources by runtime so that slower doctests are run first....
Doctesting 3 files using 2 threads.
sage -t devel/sage/sage/homology/simplicial_complex_morphism.py
[187 tests, 0.3 s]
sage -t devel/sage/sage/homology/simplicial_complex_homset.py
[49 tests, 0.1 s]
sage -t devel/sage/sage/homology/simplicial_complex.py
Time out after testing finished
**********************************************************************
Tests run before process timed out:
...
**********************************************************************
----------------------------------------------------------------------
sage -t devel/sage/sage/homology/simplicial_complex.py # Time out after testing finished
----------------------------------------------------------------------
Total time for all tests: 26.9 seconds
cpu time: 0.4 seconds
cumulative wall time: 0.5 secondsWithout the patch, the file simplicial_complex.py passes tests in under 7 seconds; cpu time and cumulative wall time are each around 6 or 7 seconds. With the patch, the time out is pretty consistent for each run after the first. I'm seeing this on two different OS X 10.8 machines.
Also, if I delete the directory DOT_SAGE/tmp, then the pidfile won't get written at all during doctesting or when running sage. What script is responsible for creating that directory? If this directory is missing, I don't get the time-out any more, so the problem is really with sage-cleaner.
On the bright side, it seems to be cleaning out the appropriate files, directories, and processes.
jhpalmieri
commented
11 years ago Maybe the problem is that time.sleep(wait + 3) used to be called before the first call to cleanup(), and now it's not. So now cleanup() can be called before any doctesting processes have started, so it returns zero, so sage-cleaner thinks there is nothing to be cleaned up and exits right away. I'm not sure why that results in a timeout in the doctesting, but anyway, this change fixes the problem for me:
diff --git a/sage-cleaner b/sage-cleaner
--- a/sage-cleaner
+++ b/sage-cleaner
@@ -115,6 +115,7 @@
else:
wait = 10
+ time.sleep(wait)
# Initial cleanup, ignore time
running_sages = cleanup()
cleanup_time = 0.0I don't know if the initial sleep should be for wait or a smaller number. Does it matter?
jhpalmieri
commented
11 years ago Replying to @jdemeyer:
Replying to @jhpalmieri:
Is there any good reason for having both directories
tmpandtemp?Absolutely not, but at least the directory used in
sage-cleanershould match the directory used in the Sage library.
Where is DOTSAGE/tmp used? Certainly the temp version is used in the definition of SAGE_TMP, so that is ubiquitous. For the file cleaner-HOSTNAME.pid, is that referenced anywhere in the Sage library? It looks like we could put that file in temp instead of tmp safely. Indeed, I can't find any other references to DOTSAGE/tmp...
novoselt
commented
11 years ago I've applied this patch to SAGE_ROOT/local/bin, ran "make ptestlong" and still got two ecl processes eating RAM when tests were finished without errors.
jdemeyer
commented
11 years ago jhpalmieri: I can totally reproduce problems with devel/sage/sage/homology/simplicial_complex.py, but they are different from yours and unrelated to sage-cleaner: #14323.
jhpalmieri
commented
11 years ago If you look at the output from the doctest command, I find it suspicious that the message "sage-cleaner is finished" appears before any doctests are run.
jdemeyer
commented
11 years ago Replying to @jhpalmieri:
I find it suspicious that the message "sage-cleaner is finished" appears before any doctests are run.
Sure, but I don't see how that could lead to doctest failures. It could be that your problem is also an instance of #14323, let's first fix that.
83660e46-0051-498b-a8c1-f7a7bd232b5a
commented
11 years ago Replying to @jhpalmieri:
Maybe the problem is that
time.sleep(wait + 3)used to be called before the first call tocleanup(), and now it's not. So nowcleanup()can be called before any doctesting processes have started, so it returns zero, so sage-cleaner thinks there is nothing to be cleaned up and exits right away. [...] I don't know if the initial sleep should be forwaitor a smaller number. Does it matter?
I wouldn't rely on some wall time.
Unfortunately the cleaner cannot wait() for its parent, but it could poll whether it is still running (or -- much better I think -- wait for another signal), and do nothing until its parent has "vanished" (terminated, and no other process with the same PID is meanwhile running -- that's also a potential, although probably unlikely, problem with left-over Sage cleaner and spawned_processes PID files; in the latter case, even unrelated processes of the same user could get killed.)
If there's already another Sage cleaner instance running, it could of course exit immediately (modulo the problem mentioned before).
jhpalmieri
commented
11 years ago I also get the same problem with devel/sage/sage/homology/examples.py, but that file uses libGAP, too. If I doctest both of these, I only get a failure in examples.py, while the other passes doctests. Adding my patch from my comment above makes the failures go away. I'm still not sure if these problems are related to #14323 or not. If they are, then at least I can reliably produce the failures, and maybe that will help to track them down.
jhpalmieri
commented
11 years ago In case it helps, I can duplicate this problem on bsd.math.washington.edu.
jdemeyer
commented
11 years ago John: please check whether the new spkg at #14323 fixes your problem.
jdemeyer
commented
11 years ago Replying to @jhpalmieri:
In case it helps, I can duplicate this problem on bsd.math.washington.edu.
I can indeed reproduce the problem and #14323 doesn't help.
jhpalmieri
commented
11 years ago On boxen, with the spkg from #14323, I see that the issue from that ticket seems to be fixed, but then I can reproduce the issue here.
jdemeyer
commented
11 years ago I found the problem with expect.py: the doctest process starts the sage-cleaner, somehow leading to trouble.
jdemeyer
commented
11 years ago Attachment: 14055_cleaner_sagelib.patch.gz
jdemeyer
commented
11 years ago Description changed:
---
+++
@@ -1 +1,5 @@
`sage-cleaner` got completely broken due to incompatible filenames between `sage-cleaner` and the Sage library.
+
+**Apply**:
+1. [attachment: 14055_cleaner_sagelib.patch](https://github.com/sagemath/sage-prod/files/10657053/14055_cleaner_sagelib.patch.gz) to the Sage library.
+2. [attachment: 14055_cleaner.patch](https://github.com/sagemath/sage-prod/files/10657054/14055_cleaner.patch.gz) to `local/bin`.Changed keywords from orphans to none
jhpalmieri
commented
11 years ago Once or twice, after running Sage, quitting, and starting it again, or doing doctests and then running Sage, I've seen this (again, I have allowed sage-cleaner to print its output to the screen):
----------------------------------------------------------------------
| Sage Version 5.9.beta1, Release Date: 2013-03-22 |
| Type "notebook()" for the browser-based notebook interface. |
| Type "help()" for help. |
----------------------------------------------------------------------
**********************************************************************
* *
* Warning: this is a prerelease version, and it may be unstable. *
* *
**********************************************************************
SAGE_TMP_ROOT = /Users/palmieri/.sage/temp/Macintosh_001b639d44a1.local
sage-cleaner is already running with PID 78006, exiting
Checking PIDs []
cleanup() #13 took 0.00s
sage-cleaner is finished
sage: Sure enough, in this case sage-cleaner isn't running while Sage is. Is this a problem? (It does start up if asked to by interfaces/cleaner.py.)
I can, not very reliably, reproduce this by starting Sage, letting the count go above 10, quit Sage, wait a second or two, and then restart it.
Otherwise, I think I'm happy with this.
jhpalmieri
commented
11 years ago See #14342 for a sort of follow-up (an attempt to stop using DOT_SAGE/tmp altogether).
83660e46-0051-498b-a8c1-f7a7bd232b5a
commented
11 years ago Replying to @novoselt:
I've applied this patch to SAGE_ROOT/local/bin, ran "make ptestlong" and still got two ecl processes eating RAM when tests were finished without errors.
Did you meanwhile also try the new versions? (There's now a patch to the Sage library as well.)
novoselt
commented
11 years ago Same story: compile 5.9.beta1, apply both patches here, run "make ptestlong" and there are two ecl processes. I also got failures
sage -t --long devel/sage/sage/homology/simplicial_complex.py # 5 doctests failed
sage -t --long devel/sage/sage/libs/gap/libgap.pyx # 5 doctests failed
sage -t --long devel/sage/sage/libs/gap/element.pyx # 3 doctests failedReplying to @novoselt:
Same story: compile 5.9.beta1, apply both patches here, run "make ptestlong" and there are two ecl processes. I also got failures
sage -t --long devel/sage/sage/homology/simplicial_complex.py # 5 doctests failed sage -t --long devel/sage/sage/libs/gap/libgap.pyx # 5 doctests failed sage -t --long devel/sage/sage/libs/gap/element.pyx # 3 doctests failed
Please show which tests are failing.
novoselt
commented
11 years ago novoselt@sage:~/sage-5.9.beta1$ ./sage -t --long devel/sage/sage/homology/simplicial_complex.py # 5 doctests failed
Running doctests with ID 2013-03-28-13-53-51-e8efe0bf.
Doctesting 1 file.
sage -t --long devel/sage/sage/homology/simplicial_complex.py
**********************************************************************
File "devel/sage/sage/homology/simplicial_complex.py", line 3134, in sage.homology.simplicial_complex.SimplicialComplex.automorphism_group
Failed example:
S.automorphism_group().is_isomorphic(SymmetricGroup(4))
Exception raised:
Traceback (most recent call last):
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 459, in _run
self.execute(example, compiled, test.globs)
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 815, in execute
exec compiled in globs
File "<doctest sage.homology.simplicial_complex.SimplicialComplex.automorphism_group[1]>", line 1, in <module>
S.automorphism_group().is_isomorphic(SymmetricGroup(Integer(4)))
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/homology/simplicial_complex.py", line 3155, in automorphism_group
[f.tuple() for f in self.facets()]])
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/graphs/generic_graph.py", line 16543, in automorphism_group
output.append(PermutationGroup([perm_group_elt(aa) for aa in a]))
File "refinement_graphs.pyx", line 941, in sage.groups.perm_gps.partn_ref.refinement_graphs.perm_group_elt (sage/groups/perm_gps/partn_ref/refinement_graphs.c:32796)
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/groups/perm_gps/permgroup.py", line 653, in __call__
return self._element_class()(x, self, check=check)
File "permgroup_element.pyx", line 452, in sage.groups.perm_gps.permgroup_element.PermutationGroupElement.__init__ (sage/groups/perm_gps/permgroup_element.c:4490)
File "sage_object.pyx", line 474, in sage.structure.sage_object.SageObject._gap_ (sage/structure/sage_object.c:4544)
File "sage_object.pyx", line 450, in sage.structure.sage_object.SageObject._interface_ (sage/structure/sage_object.c:4144)
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/interfaces/interface.py", line 199, in __call__
return cls(self, x, name=name)
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/interfaces/expect.py", line 1285, in __init__
raise TypeError, x
TypeError: list.remove(x): x not in list
**********************************************************************
File "devel/sage/sage/homology/simplicial_complex.py", line 3138, in sage.homology.simplicial_complex.SimplicialComplex.automorphism_group
Failed example:
P.automorphism_group().is_isomorphic(AlternatingGroup(5))
Exception raised:
Traceback (most recent call last):
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 459, in _run
self.execute(example, compiled, test.globs)
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 815, in execute
exec compiled in globs
File "<doctest sage.homology.simplicial_complex.SimplicialComplex.automorphism_group[3]>", line 1, in <module>
P.automorphism_group().is_isomorphic(AlternatingGroup(Integer(5)))
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/homology/simplicial_complex.py", line 3155, in automorphism_group
[f.tuple() for f in self.facets()]])
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/graphs/generic_graph.py", line 16543, in automorphism_group
output.append(PermutationGroup([perm_group_elt(aa) for aa in a]))
File "refinement_graphs.pyx", line 941, in sage.groups.perm_gps.partn_ref.refinement_graphs.perm_group_elt (sage/groups/perm_gps/partn_ref/refinement_graphs.c:32796)
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/groups/perm_gps/permgroup.py", line 653, in __call__
return self._element_class()(x, self, check=check)
File "permgroup_element.pyx", line 452, in sage.groups.perm_gps.permgroup_element.PermutationGroupElement.__init__ (sage/groups/perm_gps/permgroup_element.c:4490)
File "sage_object.pyx", line 474, in sage.structure.sage_object.SageObject._gap_ (sage/structure/sage_object.c:4544)
File "sage_object.pyx", line 450, in sage.structure.sage_object.SageObject._interface_ (sage/structure/sage_object.c:4144)
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/interfaces/interface.py", line 199, in __call__
return cls(self, x, name=name)
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/interfaces/expect.py", line 1280, in __init__
self._name = parent._create(value, name=name)
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/interfaces/interface.py", line 389, in _create
self.set(name, value)
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/interfaces/gap.py", line 1322, in set
out = self._eval_line(cmd, allow_use_file=True)
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/interfaces/gap.py", line 730, in _eval_line
self._start()
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/interfaces/gap.py", line 1175, in _start
Expect._start(self, "Failed to start GAP.")
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/interfaces/expect.py", line 443, in _start
self._expect.expect(self._prompt)
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/pexpect.py", line 916, in expect
return self.expect_list(compiled_pattern_list, timeout, searchwindowsize)
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/pexpect.py", line 967, in expect_list
c = self.read_nonblocking (self.maxread, timeout)
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/pexpect.py", line 548, in read_nonblocking
r, w, e = select.select([self.child_fd], [], [], timeout)
error: (4, 'Interrupted system call')
**********************************************************************
File "devel/sage/sage/homology/simplicial_complex.py", line 3142, in sage.homology.simplicial_complex.SimplicialComplex.automorphism_group
Failed example:
Z.automorphism_group().is_isomorphic(CyclicPermutationGroup(2))
Exception raised:
Traceback (most recent call last):
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 459, in _run
self.execute(example, compiled, test.globs)
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 815, in execute
exec compiled in globs
File "<doctest sage.homology.simplicial_complex.SimplicialComplex.automorphism_group[5]>", line 1, in <module>
Z.automorphism_group().is_isomorphic(CyclicPermutationGroup(Integer(2)))
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/homology/simplicial_complex.py", line 3155, in automorphism_group
[f.tuple() for f in self.facets()]])
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/graphs/generic_graph.py", line 16543, in automorphism_group
output.append(PermutationGroup([perm_group_elt(aa) for aa in a]))
File "refinement_graphs.pyx", line 941, in sage.groups.perm_gps.partn_ref.refinement_graphs.perm_group_elt (sage/groups/perm_gps/partn_ref/refinement_graphs.c:32796)
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/groups/perm_gps/permgroup.py", line 653, in __call__
return self._element_class()(x, self, check=check)
File "permgroup_element.pyx", line 452, in sage.groups.perm_gps.permgroup_element.PermutationGroupElement.__init__ (sage/groups/perm_gps/permgroup_element.c:4490)
File "sage_object.pyx", line 474, in sage.structure.sage_object.SageObject._gap_ (sage/structure/sage_object.c:4544)
File "sage_object.pyx", line 450, in sage.structure.sage_object.SageObject._interface_ (sage/structure/sage_object.c:4144)
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/interfaces/interface.py", line 199, in __call__
return cls(self, x, name=name)
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/interfaces/expect.py", line 1280, in __init__
self._name = parent._create(value, name=name)
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/interfaces/interface.py", line 389, in _create
self.set(name, value)
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/interfaces/gap.py", line 1322, in set
out = self._eval_line(cmd, allow_use_file=True)
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/interfaces/gap.py", line 736, in _eval_line
expect_eof= (self._quit_string() in line))
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/interfaces/gap.py", line 597, in _execute_line
x = E.expect_list(self._compiled_full_pattern)
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/interfaces/interface.py", line 516, in __getattr__
return ParentWithBase.__getattribute__(self, attrname)
File "parent.pyx", line 675, in sage.structure.parent.Parent.__getattr__ (sage/structure/parent.c:6215)
AttributeError: 'Gap' object has no attribute '_compiled_full_pattern'
**********************************************************************
File "devel/sage/sage/homology/simplicial_complex.py", line 3144, in sage.homology.simplicial_complex.SimplicialComplex.automorphism_group
Failed example:
group, dict = Z.automorphism_group(translation=True)
Exception raised:
Traceback (most recent call last):
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 459, in _run
self.execute(example, compiled, test.globs)
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 815, in execute
exec compiled in globs
File "<doctest sage.homology.simplicial_complex.SimplicialComplex.automorphism_group[6]>", line 1, in <module>
group, dict = Z.automorphism_group(translation=True)
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/homology/simplicial_complex.py", line 3155, in automorphism_group
[f.tuple() for f in self.facets()]])
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/graphs/generic_graph.py", line 16543, in automorphism_group
output.append(PermutationGroup([perm_group_elt(aa) for aa in a]))
File "refinement_graphs.pyx", line 941, in sage.groups.perm_gps.partn_ref.refinement_graphs.perm_group_elt (sage/groups/perm_gps/partn_ref/refinement_graphs.c:32796)
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/groups/perm_gps/permgroup.py", line 653, in __call__
return self._element_class()(x, self, check=check)
File "permgroup_element.pyx", line 452, in sage.groups.perm_gps.permgroup_element.PermutationGroupElement.__init__ (sage/groups/perm_gps/permgroup_element.c:4490)
File "sage_object.pyx", line 474, in sage.structure.sage_object.SageObject._gap_ (sage/structure/sage_object.c:4544)
File "sage_object.pyx", line 450, in sage.structure.sage_object.SageObject._interface_ (sage/structure/sage_object.c:4144)
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/interfaces/interface.py", line 199, in __call__
return cls(self, x, name=name)
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/interfaces/expect.py", line 1280, in __init__
self._name = parent._create(value, name=name)
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/interfaces/interface.py", line 389, in _create
self.set(name, value)
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/interfaces/gap.py", line 1322, in set
out = self._eval_line(cmd, allow_use_file=True)
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/interfaces/gap.py", line 736, in _eval_line
expect_eof= (self._quit_string() in line))
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/interfaces/gap.py", line 597, in _execute_line
x = E.expect_list(self._compiled_full_pattern)
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/interfaces/interface.py", line 516, in __getattr__
return ParentWithBase.__getattribute__(self, attrname)
File "parent.pyx", line 675, in sage.structure.parent.Parent.__getattr__ (sage/structure/parent.c:6215)
AttributeError: 'Gap' object has no attribute '_compiled_full_pattern'
**********************************************************************
File "devel/sage/sage/homology/simplicial_complex.py", line 3145, in sage.homology.simplicial_complex.SimplicialComplex.automorphism_group
Failed example:
Set([dict[s] for s in Z.vertices()])
Exception raised:
Traceback (most recent call last):
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 459, in _run
self.execute(example, compiled, test.globs)
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 815, in execute
exec compiled in globs
File "<doctest sage.homology.simplicial_complex.SimplicialComplex.automorphism_group[7]>", line 1, in <module>
Set([dict[s] for s in Z.vertices()])
TypeError: 'type' object has no attribute '__getitem__'
**********************************************************************
1 item had failures:
5 of 9 in sage.homology.simplicial_complex.SimplicialComplex.automorphism_group
[409 tests, 5 failures, 10.0 s]
----------------------------------------------------------------------
sage -t --long devel/sage/sage/homology/simplicial_complex.py # 5 doctests failed
----------------------------------------------------------------------
Total time for all tests: 10.3 seconds
cpu time: 7.9 seconds
cumulative wall time: 10.0 secondsnovoselt@sage:~/sage-5.9.beta1$ ./sage -t --long devel/sage/sage/libs/gap/libgap.pyx # 5 doctests failed
Running doctests with ID 2013-03-28-13-55-04-e18989af.
Doctesting 1 file.
sage -t --long devel/sage/sage/libs/gap/libgap.pyx
**********************************************************************
File "devel/sage/sage/libs/gap/libgap.pyx", line 23, in sage.libs.gap.libgap
Failed example:
b = gap('10')
Exception raised:
Traceback (most recent call last):
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 459, in _run
self.execute(example, compiled, test.globs)
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 815, in execute
exec compiled in globs
File "<doctest sage.libs.gap.libgap[5]>", line 1, in <module>
b = gap('10')
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/interfaces/interface.py", line 199, in __call__
return cls(self, x, name=name)
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/interfaces/expect.py", line 1285, in __init__
raise TypeError, x
TypeError: list.remove(x): x not in list
**********************************************************************
File "devel/sage/sage/libs/gap/libgap.pyx", line 24, in sage.libs.gap.libgap
Failed example:
timeit('b*b') # random output
Exception raised:
Traceback (most recent call last):
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 459, in _run
self.execute(example, compiled, test.globs)
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 815, in execute
exec compiled in globs
File "<doctest sage.libs.gap.libgap[6]>", line 1, in <module>
timeit('b*b') # random output
File "sage_timeit_class.pyx", line 114, in sage.misc.sage_timeit_class.SageTimeit.__call__ (sage/misc/sage_timeit_class.c:931)
File "sage_timeit_class.pyx", line 78, in sage.misc.sage_timeit_class.SageTimeit.eval (sage/misc/sage_timeit_class.c:731)
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/misc/sage_timeit.py", line 236, in sage_timeit
if timer.timeit(number) >= 0.2:
File "/home/novoselt/sage-5.9.beta1/local/lib/python/timeit.py", line 195, in timeit
timing = self.inner(it, self.timer)
File "<magic-timeit>", line 6, in inner
NameError: global name 'b' is not defined
**********************************************************************
File "devel/sage/sage/libs/gap/libgap.pyx", line 52, in sage.libs.gap.libgap
Failed example:
generators = libgap.AlternatingGroup(4).GeneratorsOfGroup().sage()
Exception raised:
Traceback (most recent call last):
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 459, in _run
self.execute(example, compiled, test.globs)
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 815, in execute
exec compiled in globs
File "<doctest sage.libs.gap.libgap[13]>", line 1, in <module>
generators = libgap.AlternatingGroup(Integer(4)).GeneratorsOfGroup().sage()
File "element.pyx", line 1617, in sage.libs.gap.element.GapElement_List.sage (sage/libs/gap/element.c:9500)
File "element.pyx", line 1673, in sage.libs.gap.element.GapElement_Permutation.sage (sage/libs/gap/element.c:9699)
File "permgroup_element.pyx", line 452, in sage.groups.perm_gps.permgroup_element.PermutationGroupElement.__init__ (sage/groups/perm_gps/permgroup_element.c:4490)
File "sage_object.pyx", line 474, in sage.structure.sage_object.SageObject._gap_ (sage/structure/sage_object.c:4544)
File "sage_object.pyx", line 450, in sage.structure.sage_object.SageObject._interface_ (sage/structure/sage_object.c:4144)
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/interfaces/interface.py", line 199, in __call__
return cls(self, x, name=name)
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/interfaces/expect.py", line 1280, in __init__
self._name = parent._create(value, name=name)
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/interfaces/interface.py", line 389, in _create
self.set(name, value)
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/interfaces/gap.py", line 1322, in set
out = self._eval_line(cmd, allow_use_file=True)
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/interfaces/gap.py", line 730, in _eval_line
self._start()
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/interfaces/gap.py", line 1175, in _start
Expect._start(self, "Failed to start GAP.")
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/interfaces/expect.py", line 443, in _start
self._expect.expect(self._prompt)
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/pexpect.py", line 916, in expect
return self.expect_list(compiled_pattern_list, timeout, searchwindowsize)
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/pexpect.py", line 967, in expect_list
c = self.read_nonblocking (self.maxread, timeout)
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/pexpect.py", line 548, in read_nonblocking
r, w, e = select.select([self.child_fd], [], [], timeout)
error: (4, 'Interrupted system call')
**********************************************************************
File "devel/sage/sage/libs/gap/libgap.pyx", line 53, in sage.libs.gap.libgap
Failed example:
generators # a Sage list of Sage permutations!
Exception raised:
Traceback (most recent call last):
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 459, in _run
self.execute(example, compiled, test.globs)
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 815, in execute
exec compiled in globs
File "<doctest sage.libs.gap.libgap[14]>", line 1, in <module>
generators # a Sage list of Sage permutations!
NameError: name 'generators' is not defined
**********************************************************************
File "devel/sage/sage/libs/gap/libgap.pyx", line 55, in sage.libs.gap.libgap
Failed example:
PermutationGroup(generators).cardinality() # computed in Sage
Exception raised:
Traceback (most recent call last):
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 459, in _run
self.execute(example, compiled, test.globs)
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 815, in execute
exec compiled in globs
File "<doctest sage.libs.gap.libgap[15]>", line 1, in <module>
PermutationGroup(generators).cardinality() # computed in Sage
NameError: name 'generators' is not defined
**********************************************************************
1 item had failures:
5 of 36 in sage.libs.gap.libgap
[63 tests, 5 failures, 5.3 s]
----------------------------------------------------------------------
sage -t --long devel/sage/sage/libs/gap/libgap.pyx # 5 doctests failed
----------------------------------------------------------------------
Total time for all tests: 5.4 seconds
cpu time: 5.1 seconds
cumulative wall time: 5.3 secondsnovoselt@sage:~/sage-5.9.beta1$ ./sage -t --long devel/sage/sage/libs/gap/element.pyx # 3 doctests failed
Running doctests with ID 2013-03-28-13-55-47-94cfc7ed.
Doctesting 1 file.
sage -t --long devel/sage/sage/libs/gap/element.pyx
**********************************************************************
File "devel/sage/sage/libs/gap/element.pyx", line 1400, in sage.libs.gap.element.GapElement_Function._sage_doc_
Failed example:
'constructs the cyclic group' in f._sage_doc_()
Exception raised:
Traceback (most recent call last):
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 459, in _run
self.execute(example, compiled, test.globs)
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 815, in execute
exec compiled in globs
File "<doctest sage.libs.gap.element.GapElement_Function._sage_doc_[1]>", line 1, in <module>
'constructs the cyclic group' in f._sage_doc_()
File "element.pyx", line 1407, in sage.libs.gap.element.GapElement_Function._sage_doc_ (sage/libs/gap/element.c:8928)
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/interfaces/gap.py", line 1287, in help
tmp_to_use = self._local_tmpfile()
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/interfaces/expect.py", line 581, in _local_tmpfile
self.__local_tmpfile = os.path.join(SAGE_TMP_INTERFACE, 'tmp' + str(self.pid()))
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/interfaces/expect.py", line 298, in pid
self._start()
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/interfaces/gap.py", line 1180, in _start
expect.failed_to_start.remove(self.name())
ValueError: list.remove(x): x not in list
**********************************************************************
File "devel/sage/sage/libs/gap/element.pyx", line 1666, in sage.libs.gap.element.GapElement_Permutation.sage
Failed example:
perm_gap.sage()
Exception raised:
Traceback (most recent call last):
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 459, in _run
self.execute(example, compiled, test.globs)
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 815, in execute
exec compiled in globs
File "<doctest sage.libs.gap.element.GapElement_Permutation.sage[1]>", line 1, in <module>
perm_gap.sage()
File "element.pyx", line 1673, in sage.libs.gap.element.GapElement_Permutation.sage (sage/libs/gap/element.c:9699)
File "permgroup_element.pyx", line 452, in sage.groups.perm_gps.permgroup_element.PermutationGroupElement.__init__ (sage/groups/perm_gps/permgroup_element.c:4490)
File "sage_object.pyx", line 474, in sage.structure.sage_object.SageObject._gap_ (sage/structure/sage_object.c:4544)
File "sage_object.pyx", line 450, in sage.structure.sage_object.SageObject._interface_ (sage/structure/sage_object.c:4144)
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/interfaces/interface.py", line 199, in __call__
return cls(self, x, name=name)
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/interfaces/expect.py", line 1280, in __init__
self._name = parent._create(value, name=name)
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/interfaces/interface.py", line 389, in _create
self.set(name, value)
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/interfaces/gap.py", line 1322, in set
out = self._eval_line(cmd, allow_use_file=True)
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/interfaces/gap.py", line 730, in _eval_line
self._start()
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/interfaces/gap.py", line 1175, in _start
Expect._start(self, "Failed to start GAP.")
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/sage/interfaces/expect.py", line 443, in _start
self._expect.expect(self._prompt)
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/pexpect.py", line 916, in expect
return self.expect_list(compiled_pattern_list, timeout, searchwindowsize)
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/pexpect.py", line 967, in expect_list
c = self.read_nonblocking (self.maxread, timeout)
File "/home/novoselt/sage-5.9.beta1/local/lib/python2.7/site-packages/pexpect.py", line 548, in read_nonblocking
r, w, e = select.select([self.child_fd], [], [], timeout)
error: (4, 'Interrupted system call')
**********************************************************************
File "devel/sage/sage/libs/gap/element.pyx", line 1668, in sage.libs.gap.element.GapElement_Permutation.sage
Failed example:
type(_)
Expected:
<type 'sage.groups.perm_gps.permgroup_element.PermutationGroupElement'>
Got:
<type 'sage.libs.gap.element.GapElement_Permutation'>
**********************************************************************
2 items had failures:
1 of 3 in sage.libs.gap.element.GapElement_Function._sage_doc_
2 of 4 in sage.libs.gap.element.GapElement_Permutation.sage
[256 tests, 3 failures, 5.5 s]
----------------------------------------------------------------------
sage -t --long devel/sage/sage/libs/gap/element.pyx # 3 doctests failed
----------------------------------------------------------------------
Total time for all tests: 5.6 seconds
cpu time: 5.3 seconds
cumulative wall time: 5.5 secondsPlease apply #14323 (if you haven't) and try again.
novoselt
commented
11 years ago 83660e46-0051-498b-a8c1-f7a7bd232b5a
commented
11 years ago Replying to @novoselt:
14323 fixes the doctest failures, but not ecl processes...
I'm now among the lucky few who get these greedy, immortal ECL processes:
With Sage 5.9.beta2 (which includes #14323), GCC 4.8.0, on Ubuntu 10.04.4 x86_64, during make testlong (as opposed to make ptestlong -- others previously reported only the latter caused problems).
Two ECL processes show up while testing sage/interfaces/lisp.py (as I had guessed), quickly start eating up memory, and keep running even after the file has been tested without any errors.
jhpalmieri
commented
11 years ago What needs to be done here. I asked a question earlier, but otherwise, are there issues with this?
jdemeyer
commented
11 years ago That question is really a race condition between exiting one Sage cleaner and starting another. I'll think about it.
83660e46-0051-498b-a8c1-f7a7bd232b5a
commented
11 years ago Replying to @nexttime:
Replying to @novoselt:
14323 fixes the doctest failures, but not ecl processes...
I'm now among the lucky few who get these greedy, immortal ECL processes:
With Sage 5.9.beta2 (which includes #14323), GCC 4.8.0, on Ubuntu 10.04.4 x86_64, during
maketestlong (as opposed tomake ptestlong-- others previously reported only the latter caused problems).Two ECL processes show up while testing
sage/interfaces/lisp.py(as I had guessed), quickly start eating up memory, and keep running even after the file has been tested without any errors.
Same on Ubuntu 10.04.4 x86, GCC 4.7.2, and ptestlong. I also got an orphaned Maxima process (running at 100%) in the first run. (Sage 5.9.beta2, no patches from here applied.)
83660e46-0051-498b-a8c1-f7a7bd232b5a
commented
11 years ago So I've continued experimenting with Sage 5.8 vs. 5.9.beta2 (same system, same compiler and flags), and it turns out that the rogue ECL processes are solely caused by the new doctesting framework, so we should probably open another ticket for that issue. (I did the tests with the Sage cleaner "disabled" in both releases, too, with the same results: 5.8, with the old doctesting framework: No problems at all. 5.9.beta2: ECL orphans every time I run doctests from the Makefile, no problems when run from the command line, i.e., without using make.)
Minimal receipt to reproduce the orphans here:
testlisp:
./sage -t devel/sage/sage/interfaces/lisp.pyRunning make testlisp.
The (un)setting of MAKE, using make -j1, ./sage -t vs. ./sage -tp etc. doesn't matter, nor does using $(PIPE) (or spkg/bin/pipestatus), or teeing the output with a "normal" pipe; it just works when running ./sage -t ... from the command line, while I get the busy orphans as soon as I run the exact same command(s) from the Makefile.
(Note that also ECL didn't change between 5.8 and 5.9.beta2.)
So fixing the Sage cleaner alone may alleviate the problem, but won't really solve it.
83660e46-0051-498b-a8c1-f7a7bd232b5a
commented
11 years ago Replying to @nexttime:
[...] Minimal receipt to reproduce the orphans here:
testlisp: ./sage -t devel/sage/sage/interfaces/lisp.pyRunning
make testlisp.The (un)setting of
MAKE, usingmake -j1,./sage -tvs../sage -tpetc. doesn't matter, nor does using$(PIPE)(orspkg/bin/pipestatus), orteeing the output with a "normal" pipe; it just works when running./sage -t ...from the command line, while I get the busy orphans as soon as I run the exact same command(s) from the Makefile.
Did a little more testing ...
With (Ubuntu 10.04.4's) GNU Make 3.81, I do get the orphans, while I don't get them with (vanilla) GNU Make 3.82. (I haven't found any particularly interesting change in 3.82's NEWS.)
Still, at least John Cremona reported getting the orphans on Ubuntu 12.04 as well, and I strongly doubt Precise still ships GNU Make 3.81 (released 2006, while 3.82 was released in 2010, although Ubuntu might [still] patch its version, haven't checked that). And we currently don't require the latest version of GNU Make either.
83660e46-0051-498b-a8c1-f7a7bd232b5a
commented
11 years ago Replying to @nexttime:
With (Ubuntu 10.04.4's) GNU Make 3.81, I do get the orphans, while I don't get them with (vanilla) GNU Make 3.82. (I haven't found any particularly interesting change in 3.82's NEWS.)
Still, at least John Cremona reported getting the orphans on Ubuntu 12.04 as well, and I strongly doubt Precise still ships GNU Make 3.81 (released 2006, while 3.82 was released in 2010, although Ubuntu might [still] patch its version, haven't checked that). And we currently don't require the latest version of GNU Make either.
Oh, indeed at least Ubuntu 12.04.1 LTS still has (some?) GNU Make 3.81 ... m)
jdemeyer
commented
11 years ago The ECL problem is fixed and needs review at #14426.
jhpalmieri
commented
11 years ago Here is my view on this ticket:
sage-cleaner used to be completely broken.sage-cleaner could be fixed up, but they can wait.So I'm willing to give this a positive review. Any dissenting views?
83660e46-0051-498b-a8c1-f7a7bd232b5a
commented
11 years ago Replying to @jhpalmieri:
Here is my view on this ticket:
sage-cleanerused to be completely broken.
I actually didn't have problems with it recently (more precisely, for meanwhile a few years), until 5.9.x.
- this ticket mostly fixes it.
- the only flaw I see is a race condition, but that looks pretty rare. Fixing it can wait.
Agreed, although I haven't closely looked at that yet. (It's pretty clear that there is a race condition, but I'm not sure how likely it is in practice.)
- there are undoubtedly other ways in which
sage-cleanercould be fixed up, but they can wait.So I'm willing to give this a positive review. Any dissenting views?
Hmmm, in principleTM the Sage cleaner (with the patches from here) should have killed the ECL processes, but Andrey reported it did not, so (regardless of #12426) something seems to be still broken.
83660e46-0051-498b-a8c1-f7a7bd232b5a
commented
11 years ago Replying to @nexttime:
Hmmm, in principleTM the Sage cleaner (with the patches from here) should have killed the ECL processes, but Andrey reported it did not, so (regardless of #12426) something seems to be still broken.
Ooops, of course regardless of #12426, but I meant #14426.
jhpalmieri
commented
11 years ago Replying to @nexttime:
Replying to @jhpalmieri:
Here is my view on this ticket:
sage-cleanerused to be completely broken.I actually didn't have problems with it recently (more precisely, for meanwhile a few years), until 5.9.x.
I haven't seen problems with processes, but the subdirectories of DOT_SAGE/temp/ have been polluted with many directories dating back to 5.8 at least, because sage-cleaner was looking in the wrong directory.
Hmmm, in principleTM the Sage cleaner (with the patches from here) should have killed the ECL processes, but Andrey reported it did not, so (regardless of #12426) something seems to be still broken.
In your testing, could you tell whether the correct PID (for the runaway process) was written to the correct file? How did the runaway process escape the cleaner?
83660e46-0051-498b-a8c1-f7a7bd232b5a
commented
11 years ago Replying to @jhpalmieri:
Replying to @nexttime: In your testing, could you tell whether the correct PID (for the runaway process) was written to the correct file? How did the runaway process escape the cleaner?
That's a question to Andrey (just in case that wasn't clear).
novoselt
commented
11 years ago Replying to @nexttime:
Replying to @jhpalmieri:
Replying to @nexttime: In your testing, could you tell whether the correct PID (for the runaway process) was written to the correct file? How did the runaway process escape the cleaner?
That's a question to Andrey (just in case that wasn't clear).
It was not ;-) I have no idea how to check it, but if you provide me with instructions on what files to look at, I'll be happy to do it. ECL processes appear consistently for me until #14426, so reproduction is trivial.
sage-cleanergot completely broken due to incompatible filenames betweensage-cleanerand the Sage library.Apply:
local/bin.local/bin.Update ECL spkg with http://boxen.math.washington.edu/home/vbraun/spkg/ecl-12.12.1.p3.spkg
CC: @ppurka @nbruin
Component: scripts
Author: Jeroen Demeyer, Volker Braun
Reviewer: John Palmieri, Leif Leonhardy, Volker Braun
Merged: sage-5.10.beta2
Issue created by migration from https://trac.sagemath.org/ticket/14055