Closed 1d7ec08f-60ae-4512-91a6-8324c06eab9f closed 13 years ago
Reviewer: Martin Raum
Attachment: trac_11027-schur-decomposition.patch.gz
Some discussion is necessary: As a doctest I get
Expected:
[ 0.006 - 0.001*I 0.0425 + 0.1018*I -0.6446 + 0.2201*I 0.3981 + 0.6045*I]
[ 0.0875 - 0.005*I 0.1984 + 0.5537*I -0.5022 + 0.1944*I -0.3414 - 0.4896*I]
[ 0.3572 - 0.0159*I 0.2444 + 0.685*I 0.4499 - 0.1643*I 0.1983 + 0.2728*I]
[ 0.929 - 0.0377*I -0.111 - 0.317*I -0.1212 + 0.044*I -0.0466 - 0.0627*I]
Got:
[ 0.006 - 0.001*I 0.0283 + 0.1066*I -0.6784 + 0.0611*I 0.3686 + 0.6229*I]
[ 0.0875 - 0.005*I 0.1216 + 0.5755*I -0.534 + 0.0699*I -0.3175 - 0.5055*I]
[ 0.3572 - 0.0159*I 0.1494 + 0.7118*I 0.476 - 0.053*I 0.185 + 0.282*I]
[ 0.929 - 0.0377*I -0.067 - 0.3291*I -0.1282 + 0.0141*I -0.0435 - 0.0649*I]
**********************************************************************
File "/scratch/mraum/sagedev/devel/sage-dev/sage/matrix/matrix_double_dense.pyx", line 1382:
sage: T.round(4)
Expected:
[ 30.7332 + 4648.5418*I -2342.252 + 767.2915*I -477.0197 - 1460.6287*I -532.7363 + 318.6099*I]
[ 0 -0.1841 - 159.0576*I 151.3315 + 0.458*I -15.2369 - 67.0242*I]
[ 0 0 -0.5239 + 11.158*I 6.2073 - 1.1208*I]
[ 0 0 0 -0.0252 - 0.6422*I]
Got:
[ 30.7332 + 4648.5418*I -2424.5875 + 443.008*I -117.3203 - 1532.0639*I -547.4078 + 292.6858*I]
[ 0 -0.1841 - 159.0576*I 150.4761 + 16.0737*I -21.0591 - 65.4288*I]
[ 0 0 -0.5239 + 11.158*I 5.8811 - 2.2803*I]
[ 0 0 0 -0.0252 - 0.6422*I]
Mathematica gives:
In[1]:= m = m = Table[Table[0.0 + i + 4 * j, {i, 0, 3}], {j, 0, 3}] + Table[Table[(i + 4 * j)^3 * I, {i, 0, 3}], {j, 0, 3}]
Out[1]= {{0., 1. + 1. I, 2. + 8. I, 3. + 27. I},
> {4. + 64. I, 5. + 125. I, 6. + 216. I, 7. + 343. I},
> {8. + 512. I, 9. + 729. I, 10. + 1000. I, 11. + 1331. I},
> {12. + 1728. I, 13. + 2197. I, 14. + 2744. I, 15. + 3375. I}}
In[2]:= {matU, matT} = SchurDecomposition[m]
Out[2]= {{{0.0060036 - 0.000979619 I, -0.0373719 + 0.10381 I,
> -0.0166147 + 0.680909 I, 0.370437 + 0.621815 I},
> {0.0875304 - 0.00504818 I, -0.227102 + 0.542591 I,
> 0.0085438 + 0.538502 I, -0.318925 - 0.504541 I},
> {0.357191 - 0.0158906 I, -0.281727 + 0.670563 I,
> 0.0016419 - 0.478947 I, 0.185814 + 0.281457 I},
> {0.928988 - 0.0376762 I, 0.131963 - 0.308882 I,
> -0.000646133 + 0.128983 I, -0.0436872 - 0.064763 I}},
> {{30.7332 + 4648.54 I, -2246.55 - 1013.86 I, -1535.45 - 58.1104 I,
> -546.563 + 294.26 I}, {0. + 0. I, -0.184122 - 159.058 I,
> -56.6912 - 140.312 I, -54.6402 - 41.6996 I},
> {0. + 0. I, 0. + 0. I, -0.523917 + 11.158 I, 2.95198 + 5.57432 I},
> {0. + 0. I, 0. + 0. I, 0. + 0. I, -0.0251565 - 0.642223 I}}}
That is three different results, probably depending on the machine you run it on.
Replying to @sagetrac-mraum:
Some discussion is necessary
Definitely! That is disturbing. I don't know if SciPy
has a random element, or it is that platform dependent.
This decomposition is not unique (mathematically). I cannot determine if SciPy
is suppose to return the same result across platforms. I'll dig deeper.
In any event, your three examples all seem correct, in that the diagonals of T all have the eigenvalues of the matrix. Run in Sage, the doctests should have checked all the key properties (unitary, upper-triangular, decomposition).
Did you check Mathematica's result for unitary and decomposition?
Rob
A = matrix(RDF, [[-7., 5., 11., -4., 13.],
[-11., -3., 11., 8., -19.],
[-6., 3., -5., 0., -12.],
[-4., -12., -14., 8., -8.],
[11., 0., 9., 6., 10.]])
Q, T = A.schur(base_ring=CDF)
T.round(4)
Unknown hardware, etc
http://abel.ee.ucla.edu/cvxopt/userguide/lapack.html
[ 5.67e+00+j1.69e+01 -2.13e+01+j2.85e+00 1.40e+00+j5.88e+00 -4.19e+00+j2.05e-01 3.19e+00-j1.01e+01]
[ 0.00e+00-j0.00e+00 5.67e+00-j1.69e+01 1.09e+01+j5.93e-01 -3.29e+00-j1.26e+00 -1.26e+01+j7.80e+00]
[ 0.00e+00-j0.00e+00 0.00e+00-j0.00e+00 1.27e+01+j3.43e-17 -6.83e+00+j2.18e+00 5.31e+00-j1.69e+00]
[ 0.00e+00-j0.00e+00 0.00e+00-j0.00e+00 0.00e+00-j0.00e+00 -1.31e+01-j0.00e+00 -2.60e-01-j0.00e+00]
[ 0.00e+00-j0.00e+00 0.00e+00-j0.00e+00 0.00e+00-j0.00e+00 0.00e+00-j0.00e+00 -7.86e+00-j0.00e+00]
sage-4.7.alpha2, sage.math
[ 5.6668 - 16.9373*I -8.6458 + 7.8734*I -7.1561 + 14.5703*I 5.7811 + 2.688*I -8.5999 - 5.1071*I]
[ 0 5.6668 + 16.9373*I -10.0921 + 9.9632*I -1.5618 - 5.8776*I 7.4924 + 8.7084*I]
[ 0 0 12.6587 1.6512 + 1.049*I 11.1778 + 4.4387*I]
[ 0 0 0 -13.136 -0.1856 + 0.0353*I]
[ 0 0 0 0 -7.8563]
sage4.7.alpha2, Intel i7-2600
[ 5.6668 - 16.9373*I -8.6458 + 7.8734*I -7.1561 + 14.5703*I 5.7811 + 2.688*I -8.5999 - 5.1071*I]
[ 0 5.6668 + 16.9373*I -10.0921 + 9.9632*I -1.5618 - 5.8776*I 7.4924 + 8.7084*I]
[ 0 0 12.6587 1.6512 + 1.049*I 11.1778 + 4.4387*I]
[ 0 0 0 -13.136 -0.1856 + 0.0353*I]
[ 0 0 0 0 -7.8563]
Mathematica 6.0, sage.math
{
{5.66679 - 16.9373 I, -10.6919 + 4.73528 I, -7.15609 + 14.5703 I, 5.84829 + 2.53856 I, -8.59995 - 5.10712 I},
{0. + 0. I, 5.66679 + 16.9373 I, -6.42363 + 12.6433 I, -3.46984 - 4.99463 I, 9.86177 + 5.89218 I},
{0. + 0. I, 0. + 0. I, 12.6587 + 1.16017 10^-15 I, 1.67762 + 1.0062 I, 11.1778 + 4.4386 I},
{0. + 0. I, 0. + 0. I, 0. + 0. I, -13.136 + 5.90551 10^-16 I, -0.186487 + 0.0305276 I},
{0. + 0. I, 0. + 0. I, 0. + 0. I, 0. + 0. I, -7.85626 + 1.08747 10^-17 I}
}
Results test out as correct (did not have a unitary matrix for the first example), but obvious variability. The failing doctest has a large condition number (~1000), but so does another test which is not failing. Notice that super-diagonal starts to agree at the right side.
It would be easy to fix doctests, by focusing on what the matrices should do, not on what they are. The eigenvalues could be stripped from the diagonal of the upper-triangular matrix to test as a list, since they should not vary.
This function seems too useful elsewhere (testing Hermitian, normal, etc) to abandon it. Thoughts on moving forward? We could
Abandon failing test.
Fix tests to concentrate on properties and not on uniqueness.
Solicit testing on sage-devel to see how variable this is across systems and processors.
I wouldn't support abandoning the test. But of cause it tests something, that is not guaranteed by the (mathematical) definitions. My suggestion is: 1) Emphasize in the documentation that the decomposition cannot be assumed to be unique. In particular, it can even vary on different machines. 2) Actually the crucial: U is unitary, A is upper tridiagonal, the eigenvalues are in "right" order. 3) One can also test the Frobenius norm. That's not very deep, but it is a further invariant for the nilpotent part of A.
Your thoughts?
Replying to @rbeezer:
- Fix tests to concentrate on properties and not on uniqueness.
+1
- Solicit testing on sage-devel to see how variable this is across systems and processors.
Also, you might write to the scipy list and ask how much variability we should expect on different platforms.
Hi Martin and Jason,
Thanks - those are all great suggestions. I'll probably get back to this tomorrow.
Rob
Description changed:
---
+++
@@ -1 +1,3 @@
Wraps the `schur()` function from `SciPy`.
+
+Apply trac_11027-schur-decomposition-v2.patch
Attachment: trac_11027-schur-decomposition-v2.patch.gz
v2 patch adjusts the tests to not rely on exact values in the unitary matrix, nor above the diagonal in the upper-triangular matrix. It adds a few more checks on the diagonal entries, which are the eigenvalues of the matrix. There are also a couple of new checks on the 2-norm of the similar matrices, one for the real case and one for the complex case.
This passes tests on my desktop (Ubuntu, i7-2600) and on sage.math.
Jason - would you mind testing this on a Mac before we turn it loose?
Rob
That's very good. And now all tests pass.
I just noticed, and include this as a comment: Shouldn't we think of Rstyfying the matrix_double_dense file? Currently all your nice documentation doesn't show up in the reference manual. In particular, the schur method, that is somewhat unique to this class, won't be advertised.
Author: Rob Beezer
Thanks, Martin! Yes, I went through this whole file once, but then started changing and adding whole routines and realized I was walking all over my formatting changes/updates. So I thought it would be best to wait just a little bit.
Once changes settle down I will go through and do some documentation (and minor code) clean-up and add this to the docs. It'll go up on #8046 once it happens. I've been checking my new documentation in the notebook as I go, so it should all format properly.
(I may add an exact Schur method, just for instructional use.)
Merged: sage-4.7.alpha4
Replying to @rbeezer:
Jason - would you mind testing this on a Mac before we turn it loose?
It looks like this didn't happen, and I'm getting doctest failures. On a Mac (OS X 10.6.7, Intel):
File "/Applications/sage_builds/clean/sage-4.7.alpha4/devel/sage-main/sage/matrix/matrix_double_dense.pyx", line 1538:
sage: T.round(4)
Expected:
[-13.5698 0.0 0.0 0.0]
[ 0.0 -0.8508 -0.0 -0.0]
[ 0.0 0.0 7.7664 0.0]
[ 0.0 0.0 0.0 11.6542]
Got:
[-13.5698 0.0 0.0 0.0]
[ 0.0 -0.8508 0.0 0.0]
[ 0.0 0.0 7.7664 0.0]
[ 0.0 0.0 0.0 11.6542]
On t2.math.washington.edu (Solaris on sparc):
File "/scratch/palmieri/clean/sage-4.7.alpha4/devel/sage/sage/matrix/matrix_double_dense.pyx", line 1538:
sage: T.round(4)
Expected:
[-13.5698 0.0 0.0 0.0]
[ 0.0 -0.8508 -0.0 -0.0]
[ 0.0 0.0 7.7664 0.0]
[ 0.0 0.0 0.0 11.6542]
Got:
[-13.5698 0.0 0.0 0.0]
[ 0.0 -0.8508 0.0 -0.0]
[ 0.0 0.0 7.7664 -0.0]
[ 0.0 0.0 0.0 11.6542]
Unmerging this in sage-4.7.alpha5...
Changed merged from sage-4.7.alpha4 to none
On OpenSolaris 06/2009 (quad core Xeon):
File "/export/home/drkirkby/sage-4.7.alpha4/devel/sage/sage/matrix/matrix_double_dense.pyx", line 1538:
sage: T.round(4)
Expected:
[-13.5698 0.0 0.0 0.0]
[ 0.0 -0.8508 -0.0 -0.0]
[ 0.0 0.0 7.7664 0.0]
[ 0.0 0.0 0.0 11.6542]
Got:
[-13.5698 0.0 0.0 0.0]
[ 0.0 -0.8508 0.0 -0.0]
[ 0.0 0.0 7.7664 0.0]
[ 0.0 0.0 0.0 11.6542]
**********************************************************************
Replying to @jhpalmieri:
Replying to @rbeezer:
Jason - would you mind testing this on a Mac before we turn it loose?
It looks like this didn't happen, and I'm getting doctest failures. On a Mac (OS X 10.6.7, Intel):
Thanks, John and Dave.
I'm more concerned about #10837, so if you see those variable results, please let me know about that.
John - I have an account on Skynet, so I should do some testing there, I see. Do you have any pointers to etiquette/practices there? Do I build in my home directory? Should I "nice" things or limit the number of cores?
Rob
Some clarification of how this got into sage-4.7.alpha4: when I build a new Sage alpha release, it is tested on the buildbot which tests on many machines, including (most of) the Skynet machines. So this doctest problems did show up. However, in my original sage-4.7.alpha4, I also had #10837 and I mistook those doctest failures to come from #10837 instead. So I unmerged #10837 and went ahead to release sage-4.7.alpha4 without testing again. Experience shows that testing again after unmerging a patch from a candidate alpha is usually not necessary and only slows down things. In this case, it would have caught the errors from #11027.
Long story short: even if you don't test yourself, a patch will not get in a final Sage release if it fails doctests on Skynet.
Replying to @jdemeyer:
Some clarification of how this got into sage-4.7.alpha4:
Thanks, Jeroen. No problems here.
I'd misunderstood and thought #10837 was in as well, so this helped clear that up for me.
I just need to get into the habit of testing on a wider variety of platforms if I'm going to tackle these numerical procedures. sage.math, of course, but I should add SkyNet
to my repertoire also.
Sorry for the trouble - I should have realized these -0.0 entries were a bad sign.
As always, thanks for all your careful work as release manager.
Rob
Replying to @rbeezer:
John - I have an account on Skynet, so I should do some testing there, I see. Do you have any pointers to etiquette/practices there? Do I build in my home directory? Should I "nice" things or limit the number of cores?
Ignore the request, John. I think I have found my way. Maybe.
Rob
Well, it looks like this was just carelessness on my part, and not any kind of platform-variability problem. On other doctests, which seem to pass for the various testers, I have code like
sage: T = T.zero_at(1.0e-12).change_ring(RDF)
sage: T.round(6)
The first line was missing on this one problem test - the doctest patch just adds in such a line. Likely a cut/paste when there should have been a copy/paste.
With the new patch, this passes tests on my 64-bit Ubuntu machine and on sage.math. I'm in the process of testing on SkyNet/cleo
. I'm confident this fix should be OK, but "once burned, twice shy."
This is ready for review, but I'd like to have the cleo result before it gets flipped to positive review.
Description changed:
---
+++
@@ -1,3 +1,6 @@
Wraps the `schur()` function from `SciPy`.
-Apply trac_11027-schur-decomposition-v2.patch
+**Apply:**
+1. [attachment: trac_11027-schur-decomposition-v2.patch](https://github.com/sagemath/sage-prod/files/10652480/trac_11027-schur-decomposition-v2.patch.gz)
+2. [attachment: trac_11027-schur-decomposition-doctest.patch](https://github.com/sagemath/sage-prod/files/10652481/trac_11027-schur-decomposition-doctest.patch.gz)
+
Replying to @rbeezer:
This is ready for review, but I'd like to have the cleo result before it gets flipped to positive review.
I couldn't get Sage past building PARI on cleo, and this seems to be a problem with a few tickets floating around, and I didn't see a quick fix (ie one I could understand). So I'm going to abandon that and let the buildbots do any further checks. As mentioned, i'm pretty confident about this fixing the problem, across platforms.
It's a one-line patch, and this decomposition will be really useful for lots of checks (like Hermitian, unitary, etc), so this is ready to go for a review.
Hi Rob,
I think it's going to have to be more than a one-line patch, maybe three or four lines: in addition to adding the call to zero_at
, the matrix entries in the doctest which are currently -0.0
need to be changed to 0.0
, don't they?
Replying to @jhpalmieri:
Hi Rob,
I think it's going to have to be more than a one-line patch, maybe three or four lines: in addition to adding the call to
zero_at
, the matrix entries in the doctest which are currently-0.0
need to be changed to0.0
, don't they?
Aargh! Yes, of course. I think I was doctesting the wrong file since I've had my head into matrix2.pyx heavily the past few days.
I'll get it right in the morning. Thanks for the gentle correction.
Rob
Attachment: trac_11027-schur-decomposition-doctest.patch.gz
I've replaced the "doctest" patch with a new one. With two lines now.
Changed reviewer from Martin Raum to Martin Raum, John Palmieri
This now works on Solaris, OS X, sage.math.
Replying to @jhpalmieri:
This now works on Solaris, OS X, sage.math.
Hi John,
Thanks for sticking with this mess and the extra testing. It'll be a valuable one to have - I had not appreciated how useful this is.
I've been looking at making an exact version for instructional purposes as well.
Rob
Merged: sage-4.7.1.alpha0
Attempting to appease the buildbot's prerequisites for #10848.
Apply trac_11027-schur-decomposition-v2.patch, trac_11027-schur-decomposition-doctest.patch
Attachment: trac_11027-schur-decomposition-v3.patch.gz
Patch rebased to sage-4.7
Description changed:
---
+++
@@ -1,6 +1,6 @@
Wraps the `schur()` function from `SciPy`.
**Apply:**
-1. [attachment: trac_11027-schur-decomposition-v2.patch](https://github.com/sagemath/sage-prod/files/10652480/trac_11027-schur-decomposition-v2.patch.gz)
+1. [attachment: trac_11027-schur-decomposition-v3.patch](https://github.com/sagemath/sage-prod/files/10652482/trac_11027-schur-decomposition-v3.patch.gz)
2. [attachment: trac_11027-schur-decomposition-doctest.patch](https://github.com/sagemath/sage-prod/files/10652481/trac_11027-schur-decomposition-doctest.patch.gz)
OS X 10.4 PPC returns eigenvalues in a different order:
sage -t -long "devel/sage/sage/matrix/matrix_double_dense.pyx"
**********************************************************************
File "/Users/jdemeyer/sage-4.7.1.alpha0/devel/sage/sage/matrix/matrix_double_dense.pyx", line 1571:
sage: A.eigenvalues()
Expected:
[-0.789... + 2.336...*I, -0.789... - 2.336...*I, -5.710... + 8.382...*I, -5.710... - 8.382...*I]
Got:
[-5.71017354352 + 8.38264062467*I, -5.71017354352 - 8.38264062467*I, -0.789826456477 + 2.33693044103*I, -0.789826456477 - 2.33693044103*I]
**********************************************************************
Changed merged from sage-4.7.1.alpha0 to none
ping any ideas?
Replying to @jdemeyer:
ping any ideas?
Hi Jeroen. Thanks for the interest.
Ideas? Lots:
(a) Doctesting this numerical stuff is hard.
(b) We should not be relying on Apple's vector libraries.
(c) Organizing Sage Days 31 has me distracted.
But seriously,
(i) I've thinking about taking on eigenvalues and eigenvectors, generally. There is a lot of inconsistency floating around. Ordering eigenvalues would be one thing to tackle. Which is the real reason I have not moved on this one.
(ii) Doctest in question is about verifying the 2x2 blocks along the diagonal of the real Schur decomposition. It would probably be better to not force the eigenvalues of the 2x2 blocks to match the usual Sage output, and instead just sort the "plain" eigenvalues to match the sorted list from the 2x2 blocks. That would be an easy fix.
(iii) The ordering of eigenvalues under "old" OS X might need to be investigated separately and fixed up. On Linux and Sage 4.7 I get:
sage: A = matrix(RDF, 4, 4, [[1, 0, -3, -1],
... [4, -16, -7, 0],
... [1, 21, 1, -2],
... [26, -1, -2, 1]])
sage: A.eigenvalues()
[-0.789826456477 + 2.33693044103*I, -0.789826456477 - 2.33693044103*I, -5.71017354352 + 8.38264062467*I, -5.71017354352 - 8.38264062467*I]
which is different from what you are reporting on OS X 10.4 PPC. Probably happening at the NumPy
level or further down.
I'll likely make the change mentioned above in a few days, perhaps while at Sage Days. On vacation at the moment.
Rob
I've made the change mentioned in (ii) above, and incorporated it into the "doctest" patch, which is now version 2.
Doctest works properly under Linux and conveys the same information as before. It should be more platform-independent, so hopefully it'll perform OK on OSX PPC.
Description changed:
---
+++
@@ -2,5 +2,5 @@
**Apply:**
1. [attachment: trac_11027-schur-decomposition-v3.patch](https://github.com/sagemath/sage-prod/files/10652482/trac_11027-schur-decomposition-v3.patch.gz)
-2. [attachment: trac_11027-schur-decomposition-doctest.patch](https://github.com/sagemath/sage-prod/files/10652481/trac_11027-schur-decomposition-doctest.patch.gz)
+2. [attachment: trac_11027-schur-decomposition-doctest-v2.patch](https://github.com/sagemath/sage-prod/files/10652483/trac_11027-schur-decomposition-doctest-v2.patch.gz)
Replying to @rbeezer:
Replying to @jdemeyer:
ping any ideas?
Hi Jeroen. Thanks for the interest.
To be honest, I am not particularly interested in this patch. I just think it would be a pity to lose this work. There are already too many tickets which are 90% finished, let this not become one of them.
Replying to @jdemeyer:
I just think it would be a pity to lose this work. There are already too many tickets which are 90% finished, let this not become one of them.
That qualifies as interest. ;-) I may be slow, but rarely give up. Except for #10765.
Patchbot gives this a green light. What it really needs is a test run on OS X 10.4 PPC. Anybody willing (and able)?
Replying to @rbeezer:
What it really needs is a test run on OS X 10.4 PPC. Anybody willing (and able)?
It works on OS X 10.4 PPC.
Replying to @jdemeyer:
It works on OS X 10.4 PPC.
Thanks very much, Jeroen!
I think this is all ready for somebody to give it a (quick) positive review. Hint, hint. ;-)
Okay, I have looked at attachment: trac_11027-schur-decomposition-doctest-v2.patch and it makes sense. In fact, the new patch is cleaner than the old one. Since the main patch is already positively reviewed, this gives... positive_review!
Changed reviewer from Martin Raum, John Palmieri to Martin Raum, John Palmieri, Jeroen Demeyer
Merged: sage-4.7.1.alpha3
Replying to @jdemeyer:
this gives... positive_review!
Jeroen - thanks for finishing this one off. (And for your interest!)
Wraps the
schur()
function fromSciPy
.Apply:
CC: @jasongrout @kcrisman
Component: linear algebra
Author: Rob Beezer
Reviewer: Martin Raum, John Palmieri, Jeroen Demeyer
Merged: sage-4.7.1.alpha3
Issue created by migration from https://trac.sagemath.org/ticket/11027