cygwin metaticket: port Sage to Microsoft Windows (via Cygwin): stage 1 -- make building Sage automatic

williamstein commented 15 years ago

The goal of this ticket is that a person can:

Install Cygwin and certain standard Cygwin packages (listed below).
Extract the Sage tarball and type "make"

and have everything build automatically with no errors.

The goal is not that the resulting build works or Sage starts up (this is #13841), but if that happens as well, this will be great. Adding checks for all these prereqs, if necessary, will also be part of a future ticket.

More info

Most recent trials and a lot more archived status detail is at http://trac.sagemath.org/sage_trac/wiki/CygwinPort

Current instructions (work on Windows XP and Windows 7 with latest Cygwin)

As below with Sage 5.9.beta0

Cygwin prereqs

Here is what to install from Cygwin - use the usual stable binaries.

make, perl, m4, binutils
either gcc4-core alone and then use the optional gcc-4.7.2 spkg (the standard gcc-4.6.3 spkg won't be able to compile ecl) or gcc4-core and gcc4-g++ and gcc4-fortran whose versions must match
- Currently, we also need to pull libmpfr4 for obscure reasons upstream that will likely be fixed soon - we hope
lapack, liblapack-devel (this should automatically pull liblapack0, in fact, just installing liblapack-devel should already) - though we should be able to build ATLAS

Other instructions:

Just to make sure, avoid building in home directories of Windows domain users, as they are treated in a special way by Windows (and Cygwin).
It's a good idea that all the pathnames do not contain capital letters (Windows is case-insensitive in this way, unlike Unices), spaces, etc., see #13343
Similarly, do not test without making sure that SAGE_TESTDIR does not contain spaces.
Also, don't forget to export SAGE_PORT=yes (only needed the first time you issue make though, not after failures (memleak, rebase...) and relaunching the build)!

Spkgs

Install the following spkgs ahead of time, e.g. in SAGE_ROOT/spkg/standard/ before compiling

Currently none

Patches

You may have to add patches during the build of the Sage library. Once it fails, do (assuming you are in SAGE_ROOT)

./sage -sh
cd devel/sage
<import the patches - you may need to make ~/.hgrc>
cd ../..
./sage -b
<assuming all goes well here>
touch spkg/installed/sage-5.6.rc0 # or whatever the version number is
exit

which will bring you back to your normal shell.

Patches currently needed for:

Currently none

This should then work 'out of the box', modulo rebasing issues (see CygwinPort). See the wiki for other details as well.

There is (or was) a very old binary of Cygwin available here:

http://sage.math.washington.edu/home/wstein/tmp/sage-4.1-cygwin-i686-CYGWIN_NT-5.1.tar.gz

Depends on #14031 Depends on #14465

CC: @dimpase @mwhansen @jpflori @kcrisman @simon-king-jena

Component: porting: Cygwin

Keywords: sd31 sd32

Reviewer: Jean-Pierre Flori, Dmitrii Pasechnik, Karl-Dieter Crisman, Mike Hansen, William Stein, Luis Felipe Tabera Alonso

Issue created by migration from https://trac.sagemath.org/ticket/6743

jpflori commented 11 years ago

comment:94

Replying to @kcrisman:

By the way, JP, did you end up needing libiconv or not?

Not sure yet if the one provided by cygwin is now sufficient. I'll test this weekend.

What is sure is that building the libiconv spkg on a minimal cygwin was a real pain. But I'll provide that as well, for "completeness".

kcrisman commented 11 years ago

Description changed:

--- 
+++ 
@@ -61,6 +61,9 @@

 Patches currently needed for:
 * #13806 (on Win 7, at least)
+
+After the build is completed, for doctests it would be helpful to apply patches from (following the developer guide)
+* #13324 (ECL)
 * #13364 (Maxima, if you use the spkg there)

 This should then work 'out of the box', modulo rebasing issues (see [CygwinPort](../wiki/CygwinPort)).  See [the wiki](../wiki/CygwinPort) for other details as well.

kcrisman commented 11 years ago

Description changed:

--- 
+++ 
@@ -42,7 +42,6 @@
 Optionally:
 * #9167 (ECL - optional, as only needed for starting Sage)
 * #9543 (cephes - optional, as it removes cephes)
-* #13364 (Maxima - purely optional but might as well)

 ### Patches
 You may have to add patches during the build of the Sage library.  Once it fails, do (assuming you are in `SAGE_ROOT`)
@@ -64,7 +63,6 @@

 After the build is completed, for doctests it would be helpful to apply patches from (following the developer guide)
 * #13324 (ECL)
-* #13364 (Maxima, if you use the spkg there)

 This should then work 'out of the box', modulo rebasing issues (see [CygwinPort](../wiki/CygwinPort)).  See [the wiki](../wiki/CygwinPort) for other details as well.

kcrisman commented 11 years ago

comment:97

I'll test this weekend.

I see now that we only use the iconv spkg on Solaris, Cygwin, and (?) HP-UX, so maybe in the meantime we don't need it.

What is sure is that building the libiconv spkg on a minimal cygwin was a real pain.

Or maybe we do...

But I'll provide that as well, for "completeness".

jpflori commented 11 years ago

comment:98

The easiest way to build the libiconv spkg is to install the Cygwin libiconv package, although I don't think it is the most proper solution as it add a (kind of lame) new dependency.

Apart from that, it is also possible that now, building our own spkg is not necessary anymore, Cygwin package seem to tend to become better as time passes (see the libm example, we now do not need to build our own cephes!).

jdemeyer commented 11 years ago

comment:99

Dear Cygwin testers, could you please test the following two PARI packages, whether these build (if possible with SAGE_CHECK=yes) and whether ./sage --gp seems to work.

http://boxen.math.washington.edu/home/jdemeyer/spkg/pari-2.5.3.p2.spkg (a plain bug-fix package, #13921)
http://boxen.math.washington.edu/home/jdemeyer/spkg/pari-2.5.3.p3.spkg (some Cygwin-specific workarounds removed)

kcrisman commented 11 years ago

Description changed:

--- 
+++ 
@@ -38,6 +38,7 @@
 * #13324 (ECL)
 * #13137 (MPIR)
 * #13804 (fplll, needed only on Win 7)
+* #13954 (GAP)

 Optionally:
 * #9167 (ECL - optional, as only needed for starting Sage)

kcrisman commented 11 years ago

comment:101

http://boxen.math.washington.edu/home/jdemeyer/spkg/pari-2.5.3.p2.spkg (a plain bug-fix package, #13921)

Looks like JP experienced success with this, given that he compiled 5.6.rc0. I'm currently doing this one.

http://boxen.math.washington.edu/home/jdemeyer/spkg/pari-2.5.3.p3.spkg (some Cygwin-specific workarounds removed)

Making him aware of this one - I assume there isn't a ticket yet :)

jpflori commented 11 years ago

Description changed:

--- 
+++ 
@@ -15,34 +15,31 @@

 ### Cygwin prereqs
 Here is what to install from Cygwin - use the usual stable binaries.
-* `file`, `patch`
-* `liblapack`, `liblapack0`, `liblapack-devel`
-* `libiconv`, `openssl`, `openssl-devel`
-* `libgc-devel` - although it's not needed to build, it is to start
-* `zlib-devel` - apparently needed on Win 7
-* `libncurses-devel`
 * `make`, `perl`, `m4`
-* Use all `gcc` and `g++` and `fortran`; the versions **must** match, and `gcc4` is the type we want
+* either `gcc4-core`  alone and then use the optional gcc-4.7.2 spkg (the standard gcc-4.6.3 spkg won't be able to compile ecl) or `gcc4-core` and `gcc4-g++` and `gcc4-fortran` whose versions **must** match
+* `libiconv` - or a tweaked iconv spkg from #13912
+* `liblapack`, `liblapack0`, `liblapack-devel` - though we should be able to build ATLAS
+* `libncurses-devel` - needed by singular to build, not investigated yet.

 We need to add a test for **all** of the above packages on Cygwin which the prereq script does not already check, or confirm that they are not needed, before making the Cygwin port official.

 Other instructions:
 * Just to make sure, avoid building in home directories of Windows domain users, as they are treated in a special way by Windows (and Cygwin). 
-* It's a good idea that all the pathnames do not contain capital letters (Windows is case-insensitive in this way, unlike Unices), spaces, etc.
+* It's a good idea that all the pathnames do not contain capital letters (Windows is case-insensitive in this way, unlike Unices), spaces, etc., see #13343
 * Similarly, do not *test* without making sure that `SAGE_TESTDIR` does not contain spaces.
-* Also, **don't forget to** `export SAGE_PORT=yes`!
+* Also, **don't forget to** `export SAGE_PORT=yes` (only needed the first time you issue make though, not after failures (memleak, rebase...) and relaunching the build)!

 ### Spkgs
 Install the following spkgs ahead of time, e.g. in `SAGE_ROOT/spkg/standard/` before compiling
 * #11635 (NTL)
 * #13324 (ECL)
 * #13137 (MPIR)
-* #13804 (fplll, needed only on Win 7)
+* #13804 (fplll)
 * #13954 (GAP)

 Optionally:
 * #9167 (ECL - optional, as only needed for starting Sage)
-* #9543 (cephes - optional, as it removes cephes)
+* #9543 (cephes - optional, as it removes cephes which fails to build undetected anyway currently)

 ### Patches
 You may have to add patches during the build of the Sage library.  Once it fails, do (assuming you are in `SAGE_ROOT`)
@@ -54,13 +51,13 @@
 cd ../..
 ./sage -b
 <assuming all goes well here>
-touch spkg/installed/sage-5.5.rc0 # or whatever the version number is
+touch spkg/installed/sage-5.6.rc0 # or whatever the version number is
 exit

which will bring you back to your normal shell.

Patches currently needed for: - #13806 (on Win 7, at least) + #13806

After the build is completed, for doctests it would be helpful to apply patches from (following the developer guide)

13324 (ECL)

jpflori commented 11 years ago

comment:103

Not sure what exactly the pari p3 spkg is but the changes look sane (if the first hacks are indeed not needed anymore, the second one definitely looks useless now).

kcrisman commented 11 years ago

comment:104

http://boxen.math.washington.edu/home/jdemeyer/spkg/pari-2.5.3.p2.spkg (a plain bug-fix package, #13921)

With SAGE_CHECK=yes, I get a successful installation and almost successful tests. However, the factoring example in the description of #13921 works fine. Here's the only failure:

*** ../src/test/32/program  2012-09-25 17:10:47.000000000 -0400
--- gp.out  2013-01-15 13:26:27.531250000 -0500
***************
*** 129,137 ****
  400 1.632424285532931448171405619
  ? install(addii,GG)
  ? addii(1,2)
! 3
  ? kill(addii)
  ? getheap
! [23, 3317]
  ? print("Total time spent: ",gettime);
! Total time spent: 24
--- 129,139 ----
  400 1.632424285532931448171405619
  ? install(addii,GG)
  ? addii(1,2)
!   ***   at top-level: addii(1,2)
!   ***                 ^----------
!   *** addii: bug in PARI/GP (Segmentation Fault), please report
  ? kill(addii)
  ? getheap
! [22, 3310]
  ? print("Total time spent: ",gettime);
! Total time spent: 31

http://boxen.math.washington.edu/home/jdemeyer/spkg/pari-2.5.3.p3.spkg (some Cygwin-specific workarounds removed)

I'll check this next.

kcrisman commented 11 years ago

comment:105

Replying to @jpflori: Just checking - looks like you removed zlib-devel, libgc-devel, file, and patch from the prereqs - but are these fixes in yet? Add these if necessary above. What happened to libgc-devel and file?

13914
13844

kcrisman commented 11 years ago

comment:106

http://boxen.math.washington.edu/home/jdemeyer/spkg/pari-2.5.3.p3.spkg (some Cygwin-specific workarounds removed)

I'll check this next.

It built and installed fine, with pretty much the same exact error in the tests, and factored the huge number fine in the console. If all else is well, these spkgs can be included. Well, I guess one of them is :)

kcrisman commented 11 years ago

comment:107

Not sure what exactly the pari p3 spkg is but the changes look sane (if the first hacks are indeed not needed anymore, the second one definitely looks useless now).

I can't comment on whether these really are or are not necessary, but apparently not if self-tests are passing? I haven't had a chance to test any files that use Pari, and I'll probably get forking errors anyway :(

kcrisman commented 11 years ago

Description changed:

--- 
+++ 
@@ -36,6 +36,10 @@
 * #13137 (MPIR)
 * #13804 (fplll)
 * #13954 (GAP)
+* #13914 (zlib)
+  * This has not been tested on XP, but should work; otherwise install zlib-devel on Cygwin
+* #13844 (patch)
+  * This has not been tested on XP, but should work; otherwise install patch on Cygwin

 Optionally:
 * #9167 (ECL - optional, as only needed for starting Sage)

jpflori commented 11 years ago

comment:109

Replying to @kcrisman:

Replying to @jpflori: Just checking - looks like you removed zlib-devel, libgc-devel, file, and patch from the prereqs - but are these fixes in yet? Add these if necessary above. What happened to libgc-devel and file?

13914

13844

I removed them because I was able to build and run Sage without installing them; Quoting from CygwinPort the Cygwin packages I have installed:

$ cygcheck -c
Cygwin Package Information
Package              Version              Status
_autorebase          000192-1             OK
_update-info-dir     01100-1              OK
alternatives         1.3.30c-10           OK
base-cygwin          3.1-1                OK
base-files           4.1-1                OK
bash                 4.1.10-4             OK
binutils             2.22.51-2            OK
bzip2                1.0.6-2              OK
coreutils            8.15-1               OK
crypt                1.2-1                OK
csih                 0.9.6-1              OK
cygrunsrv            1.40-2               OK
cygutils             1.4.10-2             OK
cygwin               1.7.17-1             OK
cygwin-doc           1.7-1                OK
dash                 0.5.7-1              OK
diffutils            3.2-1                OK
dos2unix             6.0.2-1              OK
editrights           1.01-2               OK
file                 5.11-1               OK
findutils            4.5.9-2              OK
gawk                 4.0.2-1              OK
gcc4-core            4.5.3-3              OK
gettext              0.18.1.1-2           OK
grep                 2.6.3-1              OK
groff                1.21-2               OK
gzip                 1.4-1                OK
ipc-utils            1.0-1                OK
lapack               3.4.2-1              OK
less                 444-1                OK
libasn1_8            1.5.2-4              OK
libattr1             2.4.46-1             OK
libbz2_1             1.0.6-2              OK
libcharset1          1.14-2               OK
libcloog0            0.15.7-1             OK
libcom_err2          1.42.6-1             OK
libdb4.5             4.5.20.2-3           OK
libedit0             20120311-1           OK
libexpat1            2.1.0-1              OK
libffi4              4.5.3-3              OK
libgcc1              4.5.3-3              OK
libgdbm4             1.8.3-20             OK
libgfortran3         4.5.3-3              OK
libgmp3              4.3.2-1              OK
libgmpxx4            4.3.2-1              OK
libgomp1             4.5.3-3              OK
libgssapi3           1.5.2-4              OK
libheimbase1         1.5.2-4              OK
libheimntlm0         1.5.2-4              OK
libhx509_5           1.5.2-4              OK
libiconv             1.14-2               OK
libiconv2            1.14-2               OK
libintl8             0.18.1.1-2           OK
libkafs0             1.5.2-4              OK
libkrb5_26           1.5.2-4              OK
liblapack-devel      3.4.2-1              OK
liblapack0           3.4.2-1              OK
liblzma5             5.0.2_20110517-1     OK
libmpc1              0.8-1                OK
libmpfr1             2.4.1-4              OK
libmpfr4             3.0.1-1              OK
libncurses-devel     5.7-18               OK
libncurses10         5.7-18               OK
libncurses9          5.7-16               OK
libncursesw10        5.7-18               OK
libopenssl100        1.0.1c-2             OK
libpcre0             8.21-2               OK
libpopt0             1.6.4-4              OK
libppl               0.10.2-1             OK
libreadline7         6.1.2-3              OK
libroken18           1.5.2-4              OK
libsigsegv2          2.10-1               OK
libsqlite3_0         3.7.13-1             OK
libssp0              4.5.3-3              OK
libstdc++6           4.5.3-3              OK
libwind0             1.5.2-4              OK
libwrap0             7.6-21               OK
libxml2              2.9.0-1              OK
login                1.10-10              OK
m4                   1.4.16-1             OK
make                 3.82.90-1            OK
man                  1.6g-1               OK
mintty               1.1.2-1              OK
nano                 2.2.5-1              OK
openssh              6.1p1-1              OK
perl                 5.14.2-3             OK
perl_vendor          5.14.2-3             OK
rebase               4.3.0-1              OK
run                  1.1.13-1             OK
sed                  4.2.1-2              OK
tar                  1.26-1               OK
terminfo             5.7_20091114-14      OK
texinfo              4.13-4               OK
tzcode               2012j-1              OK
w32api               9999-1               OK
w32api-headers       3.0b_svn5496-1       OK
w32api-runtime       3.0b_svn5496-1       OK
which                2.20-2               OK
xz                   5.0.2_20110517-1     OK
zlib0                1.2.7-1              OK

you can check they were indeed not installed.

Of course, you don't need patch and zlib-devel if you use the fixed spkg I posted at #13844 and #13914, not sure about libgc-devel, but hey it was not there and I had no problems yet. Of course I did not test everything so if it appears it is really needed, I'll put it back it as a prereq to "run" Sage.

jpflori commented 11 years ago

comment:110

Replying to @kcrisman:

Not sure what exactly the pari p3 spkg is but the changes look sane (if the first hacks are indeed not needed anymore, the second one definitely looks useless now).

I can't comment on whether these really are or are not necessary, but apparently not if self-tests are passing? I haven't had a chance to test any files that use Pari, and I'll probably get forking errors anyway :(

If both you oon XP and I on 7 are able to "build" PARI, then these changes should definitely get in. They remove useless Cygwin specific "hacks", so this is a good thing.

And as I said, I think the second one is indeed useless, I barely have doubts about that, as it must have been fixed by a patch I suggested upstream.

As far as the second one is concerned, the fact you could build PARI leaves little doubt as well.

kcrisman commented 11 years ago

comment:111

Replying to @jpflori:

Replying to @kcrisman:

Replying to @jpflori: Just checking - looks like you removed zlib-devel, libgc-devel, file, and patch from the prereqs - but are these fixes in yet? Add these if necessary above. What happened to libgc-devel and file?

13914

13844

I removed them because I was able to build and run Sage without installing them;

My point was that you didn't add the two tickets here, without which assuredly patch would not have worked, which is used a lot! I added them to the list above, as you can check.

But file is in your list in comment:109, so I'm not sure why you removed it from the list. Did it install with something else automatically?

Of course, you don't need patch and zlib-devel if you use the fixed spkg I posted at #13844 and #13914, not sure about libgc-devel, but hey it was not there and I had no problems yet. Of course I did not test everything so if it appears it is really needed, I'll put it back it as a prereq to "run" Sage.

It's conceivable that libgc-devel is, but I think that #9617 fixed that.

jpflori commented 11 years ago

comment:112

Replying to @kcrisman:

I removed them because I was able to build and run Sage without installing them;

My point was that you didn't add the two tickets here, without which assuredly patch would not have worked, which is used a lot! I added them to the list above, as you can check.

Good point.

But file is in your list in comment:109, so I'm not sure why you removed it from the list. Did it install with something else automatically?

Oops, I was not careful. But indeed I did not explicitely add it, so either it was there with the basic Cygwin install, or it was pulled by something else, I'll tend to say it's the former solution as every time I installed something I had a quick look at what was pulled, but I might have missed it as well...

jdemeyer commented 11 years ago

comment:113

Replying to @kcrisman:

I assume there isn't a ticket yet :)

Not yet. I just noticed this Cygwin-specific stuff when working on #13921 and wondered whether it could be removed. Apparently yes.

jpflori commented 11 years ago

comment:114

I ran ptest over the night and the result is not bad at all, especially that I did not include all the fixes we have (lcalc, rpy2 ar least), some of them are time outs (potentially just because Cygwin is sloooooooooooow), some of them passed when reran (e.g. the maxima.py one) and some of them are still the PARI lacking memory (cos I cannot devise how to convince Cygwin to get more than 500000000 bytes).

The result is at http://boxen.math.washington.edu/home/jpflori/ptest-5.6.rc0-cygwin.log

I've got a previous one at http://boxen.math.washington.edu/home/jpflori/ptest-5.5.rc0-cygwin.log which is not that bad as well.

jpflori commented 11 years ago

comment:115

Not really sure anymore about the libncurses-devel dependency as I'm rebuilding singular on top of my 5.6.rc0 after removing libncurses-devel and it does not complain (yet).

jpflori commented 11 years ago

comment:116

New ptest log for 5.7.beta1 at http://boxen.math.washington.edu/home/jpflori/ptest-5.7.beta1-cygwin.log

The failing tests are:

The following tests failed:

    sage -t  -force_lib devel/sagenb-main/sagenb/misc/misc.py # Exception from doctest framework
    sage -t  -force_lib devel/sage/sage/functions/other.py # 1 doctests failed
    sage -t  -force_lib devel/sage/sage/geometry/lattice_polytope.py # 96 doctests failed
    sage -t  -force_lib devel/sage/sage/graphs/genus.pyx # 2 doctests failed
    sage -t  -force_lib devel/sage/sage/gsl/ode.pyx # 3 doctests failed
    sage -t  -force_lib devel/sage/sage/libs/lcalc/lcalc_Lfunction.pyx # 69 doctests failed
    sage -t  -force_lib devel/sage/sage/libs/pari/gen.pyx # 4 doctests failed
    sage -t  -force_lib devel/sage/sage/misc/cython.py # 3 doctests failed
    sage -t  -force_lib devel/sage/sage/misc/getusage.py # 4 doctests failed
    sage -t  -force_lib devel/sage/sage/misc/inline_fortran.py # 3 doctests failed
    sage -t  -force_lib devel/sage/sage/misc/sageinspect.py # 1 doctests failed
    sage -t  -force_lib devel/sage/sage/plot/graphics.py # Time out
    sage -t  -force_lib devel/sage/sage/plot/plot.py # Time out
    sage -t  -force_lib devel/sage/sage/plot/plot3d/implicit_plot3d.py # Time out
    sage -t  -force_lib devel/sage/sage/plot/plot3d/plot3d.py # Time out
    sage -t  -force_lib devel/sage/sage/quadratic_forms/quadratic_form__ternary_Tornaria.py # 13 doctests failed
    sage -t  -force_lib devel/sage/sage/rings/arith.py # 1 doctests failed
    sage -t  -force_lib devel/sage/sage/rings/tests.py # 4 doctests failed
    sage -t  -force_lib devel/sage/sage/schemes/elliptic_curves/ell_rational_field.py # Time out
    sage -t  -force_lib devel/sage/sage/schemes/elliptic_curves/heegner.py # Time out
    sage -t  -force_lib devel/sage/sage/schemes/toric/fano_variety.py # 12 doctests failed
    sage -t  -force_lib devel/sage/sage/symbolic/expression.pyx # 1 doctests failed
    sage -t  -force_lib devel/sage/sage/tests/cmdline.py # Time out
----------------------------------------------------------------------

Some were expected:

sage/libs/lcalc/lcalc_Lfunction.pyx with fix at #13351
sage/geometry/lattice_polytope.py with fix at #13960
sage/schemes/toric/fano_variety.py is #13960 as well
sage/misc/getusage.py which is #9170
sage/graphs/genus.pyx which is in fact #9170
sage/libs/pari/gen.pyx which is also #9170
sage/rings/tests.py as well #9170
sage/quadratic_forms/quadratic_form__ternary_Tornaria.py which is hopefully the same pari stack problem as #9176 Some are trivial:
sage/functions/other.py which is numerical noise
sage/rings/arith.py as well
sage/symbolic/expression.pyx as well Some look more serious:
sage/misc/cython.py but that might just be a dynamic library extension problem
sage/misc/inline_fortran.py which does not find gfortran?!
sage/misc/sageinspect.py -> fork errors... maybe a rebase could fix that
sage/gsl/ode.pyx

And the other are timeouts (for now).

But the aim of this ticket is to build Sage, so I guess we can say we are quite finished with that :) And #13841 which aims at starting Sage is as well :)

jpflori commented 11 years ago

comment:117

FYI a rebase fixed the sage/misc/sageinspect.py failure.

jpflori commented 11 years ago

comment:118

Replying to @jpflori:

New ptest log for 5.7.beta1 at http://boxen.math.washington.edu/home/jpflori/ptest-5.7.beta1-cygwin.log

Some were expected:

sage/libs/lcalc/lcalc_Lfunction.pyx with fix at #13351

Fixed by #13351 indeed.

sage/geometry/lattice_polytope.py with fix at #13960

sage/schemes/toric/fano_variety.py is #13960 as well

Both fixed by #13960 indeed.

sage/misc/getusage.py which is #9170

sage/graphs/genus.pyx which is in fact #9170

sage/libs/pari/gen.pyx which is also #9170

sage/rings/tests.py as well #9170

All fixed by #9170 indeed.

sage/quadratic_forms/quadratic_form__ternary_Tornaria.py which is hopefully the same pari stack problem as #9176

Fixed after increasing the max cygwin heap size of python.exe using peflags ("peflags --cygwin-heap=600 local/bin/python.exe").

Some are trivial:

sage/functions/other.py which is numerical noise

sage/rings/arith.py as well

sage/symbolic/expression.pyx as well Some look more serious:

sage/misc/cython.py but that might just be a dynamic library extension problem

sage/misc/inline_fortran.py which does not find gfortran?!

sage/misc/sageinspect.py -> fork errors... maybe a rebase could fix that

sage/gsl/ode.pyx

And the other are timeouts (for now).

But the aim of this ticket is to build Sage, so I guess we can say we are quite finished with that :) And #13841 which aims at starting Sage is as well :)

jpflori commented 11 years ago

comment:119

Replying to @jpflori:

Some are trivial:

sage/functions/other.py which is numerical noise

sage/rings/arith.py as well

sage/symbolic/expression.pyx as well

See https://groups.google.com/d/topic/sage-devel/QACdziLhniU/discussion which looks related.

kcrisman commented 11 years ago

comment:120

But the aim of this ticket is to build Sage, so I guess we can say we are quite finished with that :) And #13841 which aims at starting Sage is as well :)

Awesome! But we do need something in prereq or configure or SOMEWHERE that ensures that these things still will stay the case. So we can't close it yet.

jpflori commented 11 years ago

comment:121

After increasing SAGE_TIMEOUT, the following tests passed:

sage/plot/graphics.py
sage/plot/plot3d/implicit_plot3d.py
sage/plot/plot3d/plot3d.py
sage/schemes/elliptic_curves/heegner.py

The following still fail:

sage/plot/plot.py problems with GSL like sage/gsl/ode.pyx (GSL handler blabla)
sage/schemes/elliptic_curves/ell_rational_field.py fork erros it seems when starting external programs
sage/tests/cmdline.py seemingly fork errors (calling os.fork() might not be a great idea)

And there is the one I omitted before:

sagenb/misc/misc.py # Exception from doctest framework The exception bit seems to be from an error I had with previous release as well about an alarm set to 1 second and is random. I guess Cygwin is just to slow for such a timeout. After a few tries I indeed get a bunch only one real error on the test with 'print "ignore this' because of a (Python) RuntimeError about "no available port" from the find_next_available_port function.

kcrisman commented 11 years ago

comment:122

After increasing SAGE_TIMEOUT, the following tests passed:

Maybe it's time to open a new ticket to get Cygwin to pass all tests (at least on 64-bit)?

jpflori commented 11 years ago

comment:123

Yep, I guess so. Although I'm not convinced we will ever be able to deal with the one involving horrible fork errors.

kcrisman commented 11 years ago

comment:124

Well, it was partly rhetorical! Maybe we could repurpose #13841 to have it pass most reasonable tests.

Sorry I can't help at all right now other than morally - zero time and of course crazy fork/BLODA problems. I will try to ask my computer folks for a Win 7 box to try things on when things calm down.

jdemeyer commented 11 years ago

comment:125

Replying to @kcrisman:

It built and installed fine, with pretty much the same exact error in the tests, and factored the huge number fine in the console. If all else is well, these spkgs can be included. Well, I guess one of them is :)

I have to update PARI anyway for #13054 and will remove all Cygwin-specific code from spkg-install.

kcrisman commented 11 years ago

comment:126

It built and installed fine, with pretty much the same exact error in the tests, and factored the huge number fine in the console. If all else is well, these spkgs can be included. Well, I guess one of them is :)

I have to update PARI anyway for #13054 and will remove all Cygwin-specific code from spkg-install.

JP, did you have any problems with this? I guess you can just try the spkg from #13054, which does what Jeroen says.

jpflori commented 11 years ago

comment:127

FYI, although it might be unrelated, the GSL failing tests passed after I installed a shared version of the lib and rebuilt the related Sage library files.

jpflori commented 11 years ago

comment:128

I've started building 5.7.beta2 plus some patches on a notebook running a 32 bits Windows 7 install.

jpflori commented 11 years ago

comment:129

R was upgraded at #14008 and merged in sage-5.7.beta2 and now needs math functions for long doubles, more precisely logl, which is not provided by Cygwin's libm.

Not sure if its possible to disable the use of this function in R, otherwise I guess we're back to installing Cephes on Cygwin.

jpflori commented 11 years ago

comment:130

In the devel version there is support to build without long double, see http://stat.ethz.ch/R-manual/R-devel/doc/html/NEWS.html but that does not seem to be the case in 2.15.2.

dimpase commented 11 years ago

comment:131

Replying to @jpflori:

R was upgraded at #14008 and merged in sage-5.7.beta2 and now needs math functions for long doubles, more precisely logl, which is not provided by Cygwin's libm.

Not sure if its possible to disable the use of this function in R, otherwise I guess we're back to installing Cephes on Cygwin.

Yes, that's where my Cygwin install of 5.7.beta2 got stuck, too.

as a temporary plug, we can do something like

#define logl log

in the right place. Cygwin has a weird long double size - 12 bytes, so in that sense it's not as huge loss of precision as for sane platforms.

jpflori commented 11 years ago

comment:132

I'm seeing if we cannot just backport the support for --disable-long-double. That seems difficult but who knows.

jpflori commented 11 years ago

comment:133

Replying to @dimpase:

Replying to @jpflori:

R was upgraded at #14008 and merged in sage-5.7.beta2 and now needs math functions for long doubles, more precisely logl, which is not provided by Cygwin's libm.

Not sure if its possible to disable the use of this function in R, otherwise I guess we're back to installing Cephes on Cygwin.

Yes, that's where my Cygwin install of 5.7.beta2 got stuck, too.

as a temporary plug, we can do something like
#define logl log
in the right place.

FYI this is exactly what R (not using the same mechanism) does when --disable-long-double.

Cygwin has a weird long double size - 12 bytes, so in that sense it's not as huge loss of precision as for sane platforms.

jpflori commented 11 years ago

comment:134

But it also completely disable long double, which is a little unfortunate, so I guess we'll have to provide a custom patch to just disable the use of logl.

It later lots of such functions get used we could consider falling back to --disable-long-double or using Cephes.

Any thoughts?

kcrisman commented 11 years ago

comment:135

Anything that allows us to get this Cygwin show on the road is best, assuming it is not about "core" functionality. I think it's okay to have the custom patch as long as it doesn't break R being used at all; we shouldn't be relying on R for multi precision stuff anyway. Add that patch just for Cygwin.

jpflori commented 11 years ago

comment:136

See #14078 for the R issue.

By the way all Sage and its doc built fine on my 32 bits Windows 7.

jpflori commented 11 years ago

comment:137

And result of "make ptest" is

The following tests failed:

    sage -t  -force_lib devel/sagenb-main/sagenb/misc/misc.py # Exception from doctest framework
    sage -t  -force_lib devel/sage/sage/combinat/partition_algebra.py # Time out
    sage -t  -force_lib devel/sage/sage/combinat/skew_tableau.py # Time out
    sage -t  -force_lib devel/sage/sage/combinat/crystals/kirillov_reshetikhin.py # Time out
    sage -t  -force_lib devel/sage/sage/functions/other.py # 1 doctests failed
    sage -t  -force_lib devel/sage/sage/geometry/lattice_polytope.py # Time out
    sage -t  -force_lib devel/sage/sage/groups/perm_gps/cubegroup.py # 1 doctests failed
    sage -t  -force_lib devel/sage/sage/interfaces/sage0.py # Time out
    sage -t  -force_lib devel/sage/sage/libs/gap/test_long.py # Time out
    sage -t  -force_lib devel/sage/sage/misc/cython.py # 3 doctests failed
    sage -t  -force_lib devel/sage/sage/misc/inline_fortran.py # 3 doctests failed
    sage -t  -force_lib devel/sage/sage/misc/temporary_file.py # 5 doctests failed
    sage -t  -force_lib devel/sage/sage/plot/graphics.py # Time out
    sage -t  -force_lib devel/sage/sage/plot/plot.py # Time out
    sage -t  -force_lib devel/sage/sage/plot/plot3d/implicit_plot3d.py # Time out
    sage -t  -force_lib devel/sage/sage/plot/plot3d/plot3d.py # Time out
    sage -t  -force_lib devel/sage/sage/quadratic_forms/quadratic_form__ternary_Tornaria.py # 13 doctests failed
    sage -t  -force_lib devel/sage/sage/rings/arith.py # 1 doctests failed
    sage -t  -force_lib devel/sage/sage/sandpiles/sandpile.py # Time out
    sage -t  -force_lib devel/sage/sage/schemes/elliptic_curves/ell_number_field.py # Time out
    sage -t  -force_lib devel/sage/sage/schemes/elliptic_curves/ell_rational_field.py # Time out
    sage -t  -force_lib devel/sage/sage/schemes/elliptic_curves/heegner.py # Time out
    sage -t  -force_lib devel/sage/sage/symbolic/expression.pyx # 1 doctests failed
    sage -t  -force_lib devel/sage/sage/tests/cmdline.py # Time out
----------------------------------------------------------------------
Total time for all tests: 92015.4 seconds

jpflori commented 11 years ago

comment:138

Log is at http://boxen.math.washington.edu/home/jpflori/ptest-5.7.beta2-cygwin-w7-32.log

Apart from time outs and errors similar to the one on my 64 bits W7, we got:

cubegroup.py, strange error about string out of range,
temporary_file.py which looks like a time out as well.

jpflori commented 11 years ago

comment:139

Increasing the timeout limit, let all tests which teimed out pass except for:

heegner.py because of the limited size of the PARI stack,
cmdline.py which fails like temporary_file.py because of "timeout in test_executable()", note there were no fork errors!

jpflori commented 11 years ago

comment:140

Also note that sage/schemes/elliptic_curves/ell_rational_field.py did not fail with fork errors as on my 64bits W7.

jdemeyer commented 11 years ago

comment:141

Replying to @jpflori:

heegner.py because of the limited size of the PARI stack

It's known that this test simply needs a lot of RAM, so perhaps your machine doesn't have enough memory available. What's the error message and how much memory do you have?

jpflori commented 11 years ago

comment:142

Replying to @jdemeyer:

Replying to @jpflori:

heegner.py because of the limited size of the PARI stack

It's known that this test simply needs a lot of RAM, so perhaps your machine doesn't have enough memory available. What's the error message and how much memory do you have?

I'm aware of that, I was just reporting the results of a straight test. The problem on Cygwin is that the maximal virtual memomy available is limited to something quite low by default, I'd say about 512MB. In particular it prevents the PARI stack to grow above (a little less than) this limit, even if you have 1 billion GB of RAM available. The solution is to modify this limit by hand for the python.exe file using the peflags executable, see some comments on #9176.

jdemeyer commented 11 years ago

comment:143

OK fair enough, you obviously already thought about this :-)

sagemath / sage