Upgrade numpy to 1.5.0 and scipy to 0.8

[ac9ad401-3030-4fb0-957e-6c14f81e046a]

This ticket updates two packages, which must be updated together,

http://sage.math.washington.edu/home/kcrisman/numpy-1.5.0.spkg

http://sage.math.washington.edu/home/palmieri/SPKG/scipy-0.8.spkg

If you are applying these to any version before Sage 4.6.alpha3, then you also have to update the scipy_sandbox package too (#10092). This has been merged in 4.6.alpha3, though.

http://boxen.math.washington.edu/home/kirkby/patches/scipy_sandbox-20071020.p7.spkg

After installing Numpy, one needs to execute sage -ba, or do

touch $SAGE_LOCAL/lib/python/site-packages/Cython/Includes/numpy.pxd

or else one will get runtime warnings. (Or if someone wants less hassle, they can patch sage-4.6.alpha3.spkg before building Sage).

trac_9808_changed_doctests.patch in the attachment has to be applied, in order to get all doctests running because some of the output has changed.

For reviewers: changes.txt holds a summary of all changes with reference to the diffs, and links to other tickets

Upstream: Fixed upstream, but not in a stable release.

Component: packages: standard

Keywords: numpy, scipy

Author: Stefan Reiterer, François Bissey, John Palmieri, David Kirkby, Karl-Dieter Crisman

Reviewer: Karl-Dieter Crisman, David Kirkby, Leif Leonhardy, François Bissey

Merged: sage-4.6.1.alpha0

Issue created by migration from https://trac.sagemath.org/ticket/9808

[ac9ad401-3030-4fb0-957e-6c14f81e046a]

comment:1

I have now workable spkg files for the both packages. Scipy worked well. Numpy need some work, but it's functioning.

I host the files at megaupload, at: Numpy: http://www.megaupload.com/?d=6GCFZD65 Scipy: http://www.megaupload.com/?d=JORT40DK

I think the problem is the same as described in Trac # 7791 (https://github.com/sagemath/sage-prod/issues/7791 )

I get the following errors:

/home/maldun/sage/sage-4.5.2/local/lib/python2.6/site-packages/sage/plot/all.py:22: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility
  from complex_plot import complex_plot
/home/maldun/sage/sage-4.5.2/local/lib/python2.6/site-packages/sage/plot/all.py:22: RuntimeWarning: numpy.flatiter size changed, may indicate binary incompatibility
  from complex_plot import complex_plot
/home/maldun/sage/sage-4.5.2/local/lib/python2.6/site-packages/sage/plot/plot3d/implicit_plot3d.py:5: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility
  from implicit_surface import ImplicitSurface
/home/maldun/sage/sage-4.5.2/local/lib/python2.6/site-packages/sage/plot/plot3d/implicit_plot3d.py:5: RuntimeWarning: numpy.flatiter size changed, may indicate binary incompatibility
  from implicit_surface import ImplicitSurface
/home/maldun/sage/sage-4.5.2/local/lib/python2.6/site-packages/sage/calculus/all.py:17: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility
  from riemann import Riemann_Map
/home/maldun/sage/sage-4.5.2/local/lib/python2.6/site-packages/sage/calculus/all.py:17: RuntimeWarning: numpy.flatiter size changed, may indicate binary incompatibility
  from riemann import Riemann_Map
/home/maldun/sage/sage-4.5.2/local/lib/python2.6/site-packages/sage/calculus/all.py:19: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility
  from interpolators import polygon_spline, complex_cubic_spline
/home/maldun/sage/sage-4.5.2/local/lib/python2.6/site-packages/sage/calculus/all.py:19: RuntimeWarning: numpy.flatiter size changed, may indicate binary incompatibility
  from interpolators import polygon_spline, complex_cubic_spline
/home/maldun/sage/sage-4.5.2/local/lib/python2.6/site-packages/sage/stats/hmm/all.py:8: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility
  from hmm  import DiscreteHiddenMarkovModel
/home/maldun/sage/sage-4.5.2/local/lib/python2.6/site-packages/sage/stats/hmm/all.py:8: RuntimeWarning: numpy.flatiter size changed, may indicate binary incompatibility
  from hmm  import DiscreteHiddenMarkovModel

What's this? Has someone hardcoded the sizes of the routines?

[ac9ad401-3030-4fb0-957e-6c14f81e046a]

comment:2

numpy 1.5.0rc1: http://www.megaupload.com/?d=KK31RSZV

[mwhansen]

comment:3

I don't think that we should upgrade to 1.5.0rc1 -- we should do 1.4.1 for now and wait until 1.5 is released.

[kiwifb]

comment:4

Replying to @mwhansen:

I don't think that we should upgrade to 1.5.0rc1 -- we should do 1.4.1 for now and wait until 1.5 is released.

That has to be double checked but maldun says he needs features in 1.5. It may be that the features he wants are 1.4.x and he doesn't know. Do we know roughly when 1.5 will be released? They are at 1.5rc1 9 days after beta2 so it could very well be upon us in a very short time.

[ac9ad401-3030-4fb0-957e-6c14f81e046a]

comment:5

The warnings go away after rebuilding the source. And the doctests only throw warnings but there don't seem to be errors, and the output is correct

@mwhansen, fbissey I think numpy has the same issues since version 1.4. and I'm quite sure that resolving it in 1.5.0rc1 would, solve the problem with 1.4.1. too. (and perhaps also with 1.5. later) So I suggest the following procedure: I solve the issues with the latest. Dobule check if 1.4.1 also works, and we decide then which of the versions we keep, or for example keep 1.4.1 in standard and 1.5.0rc1 as experimental

[bac7d3ea-3f1b-4826-8464-f0b53d5e12d2]

comment:6

If 1.5.0rc1 is merged, #7166 can be closed. I don't know about 1.4.1, but in any case that is not a critical bug.

Some time back I made a Solaris-specific change to Numpy, as I wanted to get it reviewed with the least hassle - that generally means making it Solaris specific, as reviewers are easier to please if it only effects a rarer platform.

But I think the change should be implemented on OS X too. Currently there's a really nasty hack on OS X to build Numpy, that involves a script called fake_gcc, copying that to $SAGE/LOCAL/bin/gcc, then using the fake gcc to build Numpy. Finally this fake gcc file gets deleted.

The far neater option is the way I did it on Solaris. I suggest the changes I made for Solaris are implemented whenever SAGE64 is set to "yes", irrespective of whatever platform it may be. Then all this fake gcc rubbish can be removed.

If you want, I can create a patch.

[ac9ad401-3030-4fb0-957e-6c14f81e046a]

comment:7

Replying to @sagetrac-drkirkby:

If you want, I can create a patch.

would be nice! But first I have to sort some things out, I hope it will get ready soon...

[kiwifb]

comment:8

Looking at all that stuff in the spkg and comparing to Gentoo. Not very pretty. Do we really still need to use sage_fortran? On most platform we now use a native fortran compiler rather than a sage shipped one - I don't know if we still do that for OS X. In the patch to gnu.py there is:

+        # Note that sse flags and so on lead to gfortran code that segfaults, so disable arch flags
+        return opt
+

An extract of the Gentoo set up:

    export NUMPY_FCONFIG="config_fc --noopt --noarch"

Actually here is the full set up that you might find interesting:

    # See progress in http://projects.scipy.org/scipy/numpy/ticket/573
    # with the subtle difference that we don't want to break Darwin where
    # -shared is not a valid linker argument
    if [[ ${CHOST} != *-darwin* ]] ; then
        append-ldflags -shared
    fi

    # only one fortran to link with:
    # linking with cblas and lapack library will force
    # autodetecting and linking to all available fortran compilers
    use lapack || return
    [[ -z ${FC} ]] && FC=$(tc-getFC)
    # when fortran flags are set, pic is removed.
    FFLAGS="${FFLAGS} -fPIC"
    export NUMPY_FCONFIG="config_fc --noopt --noarch"

The other patches we have are relatively minor, a fix to the f2py man page, a patch for freebsd - that may be usefull, a patch for interix(!). I cannot comment on the cygwin patches but they look very small. We are are a bit more anal with site.cfg, I don't think it is useful in the context of sage - we just have more complex possible combinations of blas/lapack.

The NUMPY_FCONFIG is passed to distutils as an argument of

python setup.py install

i.e. in the end what we do boils down to "python setup.py install ${NUMPY_FCONFIG}".

Can you point me to your solaris fix Dave? I'd like to see if I can tidy all that up in something that requires less patching and is more based on FLAGS settings.

[bac7d3ea-3f1b-4826-8464-f0b53d5e12d2]

comment:9

Replying to @kiwifb:

Looking at all that stuff in the spkg and comparing to Gentoo. Not very pretty.

What a surprise.

Do we really still need to use sage_fortran? On most platform we now use a native fortran compiler rather than a sage shipped one - I don't know if we still do that for OS X.

I've never understood the need for this SAGE_FORTRAN. I've tried arguing for it to be removed and use FC instead, but I had no luck.

A Fortran compiler is still shipped on OS X, but I don't see why the variable FC can't be made to point to that, rather than have the variable SAGE_FORTRAN.

Can you point me to your solaris fix Dave? I'd like to see if I can tidy all that up in something that requires less patching and is more based on FLAGS settings.

On Solaris, and some OS X versions, if you want a 64-bit build, you must add the compiler flag -m64 with gcc. Usually, putting that in CFLAGS is enough. However, for Numpy that does not work.

Someone came up with a fix for this which was implemented only on OS X, that involved creating a wrapper script called gcc_fake which basically calls gcc with the -m64 option. You can see the script yourself in the top level directory.

Since they had done this only on OS X, it did not work on Solaris. So I came up with a solution for Solaris, but I avoided the wrapper script. Instead I set the variable to CC=gcc -m64

I'm attaching a patch, which basically uses the Solaris on any system, including OS X. I think this is the sensible way to do it, not have a wrapper script.

I've not tested the attached patch - not even on Solaris!! But I think you can see what I am trying to do. I was going to try to explain it in words, but a bit of code, even if untested, should be more sensible.

[bac7d3ea-3f1b-4826-8464-f0b53d5e12d2]

Attachment: 9808-remove-gcc_fake.patch.gz

An untested patch, which makes Numpy build the same was on OS X as it does on Solaris or other platforms where SAGE64=yes. It removes the stupid wrapper script.

[kiwifb]

comment:10

Ok - so we still use g95 on some targets. So we need to keep some patches just for these - bother.

[bac7d3ea-3f1b-4826-8464-f0b53d5e12d2]

comment:11

Replying to @kiwifb:

Ok - so we still use g95 on some targets. So we need to keep some patches just for these - bother.

I don't believe g95 is used anywhere. There are g95 binaries in the Fortran package in Sage, but William said they can be removed. There is a gfortran binary. So as far as I'm aware, all g95 stuff can be removed, but SAGE_FORTRAN can't be removed.

Dave

[kiwifb]

comment:12

Replying to @sagetrac-drkirkby:

Replying to @kiwifb:

Ok - so we still use g95 on some targets. So we need to keep some patches just for these - bother.

I don't believe g95 is used anywhere. There are g95 binaries in the Fortran package in Sage, but William said they can be removed. There is a gfortran binary. So as far as I'm aware, all g95 stuff can be removed, but SAGE_FORTRAN can't be removed.

That's good! That means we probably can give the shove to the gnu.py and init.py patches. I wouldn't be sorry to see the back of these.

There is a comment in SPKG.txt:

Special Update/Build Instructions:
  * The file $SAGE_ROOT/devel/sage/sage/ext/numpy.pxi comes from this file and must be updated if/when 
    the file src/numpy/doc/cython/numpy.pxi is updated.

I cannot find the file in question in that location. There is however a numpy.pxi under src/numpy/random/mtrand but I am not sure that's the file in question. Furthermore I don't appear to have a numpy.pxi in $SAGE_ROOT/devel/sage/sage/ext/numpy.pxi . Does anyone know if these instructions are obsolete?

[jasongrout]

comment:13

FYI: Numpy 1.5 has been officially released now.

[jasongrout]

comment:14

Replying to @kiwifb:

Replying to @sagetrac-drkirkby:

Replying to @kiwifb:

Ok - so we still use g95 on some targets. So we need to keep some patches just for these - bother.

I don't believe g95 is used anywhere. There are g95 binaries in the Fortran package in Sage, but William said they can be removed. There is a gfortran binary. So as far as I'm aware, all g95 stuff can be removed, but SAGE_FORTRAN can't be removed.

That's good! That means we probably can give the shove to the gnu.py and init.py patches. I wouldn't be sorry to see the back of these.

There is a comment in SPKG.txt:

Special Update/Build Instructions:
  * The file $SAGE_ROOT/devel/sage/sage/ext/numpy.pxi comes from this file and must be updated if/when 
    the file src/numpy/doc/cython/numpy.pxi is updated.

I cannot find the file in question in that location. There is however a numpy.pxi under src/numpy/random/mtrand but I am not sure that's the file in question. Furthermore I don't appear to have a numpy.pxi in $SAGE_ROOT/devel/sage/sage/ext/numpy.pxi . Does anyone know if these instructions are obsolete?

I think I was the one that added those instructions, and I'm pretty sure they're obsolete instructions now. I believe we took care of merging the differences between the two numpy.pxi/pxd files a while ago.

[kiwifb]

comment:15

Replying to @jasongrout:

I think I was the one that added those instructions, and I'm pretty sure they're obsolete instructions now. I believe we took care of merging the differences between the two numpy.pxi/pxd files a while ago.

Thank you for the confirmation. I have done some cleaning to numpy's spkg-install and it seems to work as intended on my machine. I guess we'll update to 1.5 and give it a spin.

[bac7d3ea-3f1b-4826-8464-f0b53d5e12d2]

comment:16

Replying to @kiwifb:

Replying to @mwhansen:

I don't think that we should upgrade to 1.5.0rc1 -- we should do 1.4.1 for now and wait until 1.5 is released.

That has to be double checked but maldun says he needs features in 1.5. It may be that the features he wants are 1.4.x and he doesn't know. Do we know roughly when 1.5 will be released? They are at 1.5rc1 9 days after beta2 so it could very well be upon us in a very short time.

In general though, we should not be upgrading to pre-release versions just because one person needs a feature that's only available in a pre-release. Everyone should not have to suffer the extra risks a pre-release gives unless there are compelling arguments for it.

I realise in this case 1.5 has since been released, so it's immaterial now.

Dave

[jasongrout]

comment:17

Replying to @sagetrac-drkirkby:

In general though, we should not be upgrading to pre-release versions just because one person needs a feature that's only available in a pre-release. Everyone should not have to suffer the extra risks a pre-release gives unless there are compelling arguments for it.

+1

[kiwifb]

comment:18

Replying to @sagetrac-drkirkby:

In general though, we should not be upgrading to pre-release versions just because one person needs a feature that's only available in a pre-release. Everyone should not have to suffer the extra risks a pre-release gives unless there are compelling arguments for it.

I realise in this case 1.5 has since been released, so it's immaterial now.

Yes, I didn't think it would be worth working on that particular version if it wasn't for the real proximity of the release (curses pari svn upgrade). I would probably have put pressure for 1.4.1 otherwise (very keen to see numpy upgraded).

[ac9ad401-3030-4fb0-957e-6c14f81e046a]

comment:19

Replying to @jasongrout:

FYI: Numpy 1.5 has been officially released now.

So I was right =) Til I'm done with 1.5.0rc1 1.5 is out...

So changed so far:

The following doctest had to be rewritten:

File "/home/maldun/sage/sage-4.5.2/devel/sage/doc/en/faq/faq-usage.rst", line 341:
    sage: stats.ttest_ind(list([1,2,3,4,5]),list([2,3,4,5,.6]))
Expected:
    doctest...DeprecationWarning...
    (0.076752955645333687, 0.94070490247380478)
Got:
    (0.076752955645333687, 0.94070490247380478)

That was easy =)

The next prob was:

File "/home/maldun/sage/sage-4.5.2/devel/sage/sage/graphs/graph.py", line 615:
    sage: Graph(A)
Exception raised:
    Traceback (most recent call last):
      File "/home/maldun/sage/sage-4.5.2/local/bin/ncadoctest.py", line 1231, in run_one_test
        self.run_one_example(test, example, filename, compileflags)
      File "/home/maldun/sage/sage-4.5.2/local/bin/sagedoctest.py", line 38, in run_one_example
        OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags)
      File "/home/maldun/sage/sage-4.5.2/local/bin/ncadoctest.py", line 1172, in run_one_example
        compileflags, 1) in test.globs
      File "<doctest __main__.example_1[22]>", line 1, in <module>
        Graph(A)###line 615:
    sage: Graph(A)
      File "/home/maldun/sage/sage-4.5.2/local/lib/python/site-packages/sage/graphs/graph.py", line 846, in __init__
        data = networkx.MultiGraph(data)
      File "/home/maldun/sage/sage-4.5.2/local/lib/python2.6/networkx/classes/graph.py", line 220, in __init__
        convert.from_whatever(data,create_using=self)
      File "/home/maldun/sage/sage-4.5.2/local/lib/python2.6/networkx/convert.py", line 157, in from_whatever
        if isinstance(thing,numpy.core.defmatrix.matrix) or \
    AttributeError: 'module' object has no attribute 'defmatrix'

that was also easy. defmatrix just changed from numpy.core to numpy.matrix to numpy.matrixlib Only changed that line in /local/lib/python2.6/networkx/classes/graph.py

Here comes now a trickier one:

       sage -t -valgrind "devel/sage/sage/rings/polynomial/polynomial_element.pyx"
Total time for all tests: 716.4 seconds
maldun@zauberbuch:~/sage/sage-4.5.2$ sage -t  -valgrind "devel/sage/sage/rings/polynomial/real_roots.pyx
> "
ERROR: File ./devel/sage/sage/rings/polynomial/real_roots.pyx
 is missing

----------------------------------------------------------------------
The following tests failed:

        ./devel/sage/sage/rings/polynomial/real_roots.pyx
 # File not found
Total time for all tests: 0.0 seconds
maldun@zauberbuch:~/sage/sage-4.5.2$ sage -t  -valgrind "devel/sage/sage/rings/polynomial/real_roots.pyx"
sage -t -valgrind "devel/sage/sage/rings/polynomial/real_roots.pyx"
**********************************************************************
File "/home/maldun/sage/sage-4.5.2/devel/sage/sage/rings/polynomial/real_roots.pyx", line 1819, in __main__.example_76
Failed example:
    oc.find_roots()###line 3064:_sage_    >>> oc.find_roots()
Expected nothing
Got:
    doctest:1: RuntimeWarning: tp_compare didn't return -1 or -2 for exception
**********************************************************************
File "/home/maldun/sage/sage-4.5.2/devel/sage/sage/rings/polynomial/real_roots.pyx", line 1840, in __main__.example_77
Failed example:
    oc.find_roots()###line 3085:_sage_    >>> oc.find_roots()
Expected nothing
Got:
    doctest:1: RuntimeWarning: tp_compare didn't return -1 or -2 for exception
**********************************************************************
File "/home/maldun/sage/sage-4.5.2/devel/sage/sage/rings/polynomial/real_roots.pyx", line 1934, in __main__.example_80
Failed example:
    oc.find_roots()###line 3157:_sage_    >>> oc.find_roots()
Expected nothing
Got:
    doctest:1: RuntimeWarning: tp_compare didn't return -1 or -2 for exception
**********************************************************************
File "/home/maldun/sage/sage-4.5.2/devel/sage/sage/rings/polynomial/real_roots.pyx", line 2320, in __main__.example_98
Failed example:
    real_roots(x**Integer(5) * (x**Integer(2) - Integer(9999))**Integer(2) - Integer(1))###line 3870:_sage_    >>> real_roots(x^5 * (x^2 - 9999)^2 - 1)
Expected:
    [((-29274496381311/9007199254740992, 419601125186091/2251799813685248), 1), ((2126658450145849453951061654415153249597/21267647932558653966460912964485513216, 4253316902721330018853696359533061621799/42535295865117307932921825928971026432), 1), ((1063329226287740282451317352558954186101/10633823966279326983230456482242756608, 531664614358685696701445201630854654353/5316911983139663491615228241121378304), 1)]
Got:
    doctest:1: RuntimeWarning: tp_compare didn't return -1 or -2 for exception
    [((-29274496381311/9007199254740992, 419601125186091/2251799813685248), 1), ((2126658450145849453951061654415153249597/21267647932558653966460912964485513216, 4253316902721330018853696359533061621799/42535295865117307932921825928971026432), 1), ((1063329226287740282451317352558954186101/10633823966279326983230456482242756608, 531664614358685696701445201630854654353/5316911983139663491615228241121378304), 1)]
**********************************************************************
4 items had failures:
   1 of  10 in __main__.example_76
   1 of   9 in __main__.example_77
   1 of  10 in __main__.example_80
   1 of  44 in __main__.example_98
***Test Failed*** 4 failures.
         [227.6 s]

----------------------------------------------------------------------
The following tests failed:

        sage -t -valgrind "devel/sage/sage/rings/polynomial/real_roots.pyx"
Total time for all tests: 227.6 seconds

This is not the only one, but the root of evil seems to be in real_roots (how ironic...) more precisly: I tracked it down, with help of valgrind, to the class ocean. Has anyone an Idea??

[ac9ad401-3030-4fb0-957e-6c14f81e046a]

comment:21

Update: Found the problem. see: http://groups.google.com/group/cython-users/browse_thread/thread/624c696293b7fe44

It seems that all versions 1.5.x hold this bug.... Tried downgrading to 1.4.1. and all corrections I have done so far worked.

I will now apply the patch from drkirkby, pack it again, test it overnight and hopefully we are done with 1.4.1, and hopefully they get it right in the next time. perhaps I should send the numpy/scipy guys the doctests I've done so far

scipy 0.8. don't seem to make any problems so far

[ac9ad401-3030-4fb0-957e-6c14f81e046a]

comment:23

an doctest I oversaw:

Type ``numpy.typecodes`` for a list of the possible
        typecodes::

            sage: import numpy
            sage: sorted(numpy.typecodes.items())
            [('All', '?bhilqpBHILQPfdgFDGSUVOMm'), ('AllFloat', 'fdgFDG'), ('AllInteger', 'bBhHiIlLqQpP'), ('Character', 'c'), ('Complex', 'FDG'), ('Datetime', 'Mm'), ('Float', 'fdg'), ('Integer', 'bhilqp'), ('UnsignedInteger', 'BHILQP')]

The output is now:

[('All', '?bhilqpBHILQPfdgFDGSUVOMm'), ('AllFloat', 'fdgFDG'), ('AllInteger', 'bBhHiIlLqQpP'), ('Character', 'c'), ('Complex', 'FDG'), ('Datetime', 'Mm'), ('Float', 'fdg'), ('Integer', 'bhilqp'), ('UnsignedInteger', 'BHILQP')]

But since it's an extension, and no downgrade it is also no prob

[kiwifb]

comment:24

Replying to @sagetrac-maldun:

an doctest I oversaw:

Type ``numpy.typecodes`` for a list of the possible
        typecodes::

            sage: import numpy
            sage: sorted(numpy.typecodes.items())
            [('All', '?bhilqpBHILQPfdgFDGSUVOMm'), ('AllFloat', 'fdgFDG'), ('AllInteger', 'bBhHiIlLqQpP'), ('Character', 'c'), ('Complex', 'FDG'), ('Datetime', 'Mm'), ('Float', 'fdg'), ('Integer', 'bhilqp'), ('UnsignedInteger', 'BHILQP')]

The output is now:

[('All', '?bhilqpBHILQPfdgFDGSUVOMm'), ('AllFloat', 'fdgFDG'), ('AllInteger', 'bBhHiIlLqQpP'), ('Character', 'c'), ('Complex', 'FDG'), ('Datetime', 'Mm'), ('Float', 'fdg'), ('Integer', 'bhilqp'), ('UnsignedInteger', 'BHILQP')]

But since it's an extension, and no downgrade it is also no prob

the doctest should be fixed as well. I have attached a new version of spkg-install for numpy if you cared to give it a spin. It completely drop the non-cygwin patches. It may need a little bit of work as I haven't looked yet at how you included Dave's fix.

[ac9ad401-3030-4fb0-957e-6c14f81e046a]

comment:25

Replying to @kiwifb:

the doctest should be fixed as well. I have attached a new version of spkg-install for numpy if you cared to give it a spin. It completely drop the non-cygwin patches. It may need a little bit of work as I haven't looked yet at how you included Dave's fix.

Your install file worked well for me. I have merged it, such that the patches are getting applied. All doctests work now. You can download the packages now at: http://code.google.com/p/spkg-upload/downloads/detail?name=numpy-1.4.1.spkg and http://code.google.com/p/spkg-upload/downloads/detail?name=scipy-0.8.spkg

but you also have to install a modified version of networkx. I have opened a ticket for this, and this also includes a package: https://github.com/sagemath/sage-prod/issues/9854

next I will add a patch which includes the changed doctests

[ac9ad401-3030-4fb0-957e-6c14f81e046a]

comment:26

One important note: after updating numpy one has to rebuild the source with sage -ba, or else you get runtime warnings.

The reason is the size change in some of the numpy functions, and then the .pyx files have to be rebuild

[ac9ad401-3030-4fb0-957e-6c14f81e046a]

modified networkx package (test version)

[ac9ad401-3030-4fb0-957e-6c14f81e046a]

Attachment: networkx-1.0.1.p0.spkg.gz

Attachment: convert.py.diff.gz

changes to networkx, which have to be applied

[kiwifb]

comment:27

Thanks for the patches. I will probably introduce these in sage-on-gentoo in short order. About networkx, you realize that sage-4.5.3 will use networkx-1.2? see ticket #9567 So have you tried to see if networkx-1.2 needs patching (my guess is no, otherwise I would know by now from gentoo).

[ac9ad401-3030-4fb0-957e-6c14f81e046a]

comment:28

Replying to @kiwifb:

Thanks for the patches. I will probably introduce these in sage-on-gentoo in short order. About networkx, you realize that sage-4.5.3 will use networkx-1.2? see ticket #9567 So have you tried to see if networkx-1.2 needs patching (my guess is no, otherwise I would know by now from gentoo).

I've seen that networkx 1.2. is out but it doesn't seem to work with my sage-4.2 with numpy 1.4.1 nor with my sage-4.1 with the old numpy so I created this spkg in order that it can be used with old versions of sage too

But it seems the line where the problems come from doesn't exist in the new version of networkx anymore. Could you give the numpy packages a spin with the new versions of numpy/scipy? It is very time consuming for me to rebuild every time the source. Thanx in advance!

[ac9ad401-3030-4fb0-957e-6c14f81e046a]

comment:29

ith the new versions of numpy/scipy? It is very time consuming for me to rebuild every time the source. Thanx in advance!

sorry I meant new versions of networkx...

[kiwifb]

comment:30

Understood. We already ship networkx-1.2 with sage-4.5.2 on sage-on-gentoo [because networkx-1.0.1 has been removed from Gentoo] so I can test your patches along with networkx. It may take 1 or 2 days for me to fit it in my schedule.

[ac9ad401-3030-4fb0-957e-6c14f81e046a]

comment:31

ok. Since I had some time this afternoon I build a version of sage-1.4.3.alpha1 (which also has networkx-1.2.) on my machine, applied the packages, the patch, rebuild the source with sage -ba and everything worked fine!

./sage -tp 4 -long devel/sage-numpy
....
sage -t -long devel/sage-numpy/doc/en/a_tour_of_sage/index.rst
         [23.1 s]
sage -t -long devel/sage-numpy/doc/en/thematic_tutorials/group_theory.rst
         [247.1 s]

----------------------------------------------------------------------
All tests passed!
Timings have been updated.
Total time for all tests: 7320.5 seconds

So networkx-1.2 works!

For Info: I use Ubuntu 10.04 on my machine. So I think it would still be good if someone else would test it

[kiwifb]

comment:33

It all works in sage-on-gentoo.

However I think there are a few points that should be taken into consideration for both numpy and scipy. Just before posting my spkg-install I edited it to remove a change that I now think is probably necessary. I had set FC to SAGE_FORTRAN... Why? Because numpy tries to autodetect a fortran compiler and will take a gnu compiler first. Unless you set FC. Which means if Dave tries to compile sage with sunstudio on a machine that has also gfortran he is in for a world of hurt. So I think we should either set FC to SAGE_FORTRAN in numpy and possibly scipy. The other option is to set it globally but that may cause problems on OSX.

Thoughts? Francois

[ac9ad401-3030-4fb0-957e-6c14f81e046a]

comment:34

Replying to @kiwifb:

It all works in sage-on-gentoo.

However I think there are a few points that should be taken into consideration for both numpy and scipy. Just before posting my spkg-install I edited it to remove a change that I now think is probably necessary. I had set FC to SAGE_FORTRAN... Why? Because numpy tries to autodetect a fortran compiler and will take a gnu compiler first. Unless you set FC. Which means if Dave tries to compile sage with sunstudio on a machine that has also gfortran he is in for a world of hurt. So I think we should either set FC to SAGE_FORTRAN in numpy and possibly scipy. The other option is to set it globally but that may cause problems on OSX.

Thoughts? Francois

A very pragmatic thougt: If it doesn"t hurt to set FC to SAGE_FORTRAN, why not? But I think it"s better not to do it globally. Because then we could cause more problems then we solve. Can you give a modified spkg-install? I can try it out over night then.

But I think it would also be good to get some feedback for Solaris and OSX for the current versions. Then we could decide to keep the current, or to take the other.

[ac9ad401-3030-4fb0-957e-6c14f81e046a]

comment:35

Good news: It seems that the problem from numpy-1.5.0 has been resolved http://projects.scipy.org/numpy/ticket/1605 I don't think it's a big deal. Shall I give it a try, or should we stick to 1.4.1 for now?

[ac9ad401-3030-4fb0-957e-6c14f81e046a]

comment:36

Replying to @sagetrac-maldun:

Good news: It seems that the problem from numpy-1.5.0 has been resolved http://projects.scipy.org/numpy/ticket/1605 I don't think it's a big deal. Shall I give it a try, or should we stick to 1.4.1 for now?

...or wait til 1.5.1 is out?

[kiwifb]

comment:37

May be wait until numpy-1.5.1. it is not a big deal right now I guess. May be we should discuss it on sage-devel. At the moment we don't really have much testing apart from the fact I unleashed the upgrade to numpy-1.4.1/scipy-0.8 on sage-on-gentoo users. I am not sure we can believably merge this in 4.5.3 unless it takes more than one week before it is released. If we target 4.5.3 I say we stay with what we have now, if we target 4.6 which should be a little further down the track we go for 1.5.x.

[kiwifb]

Attachment: spkg-install.gz

newer spkg_install setting FC

[kiwifb]

comment:38

I updated my spkg_install as you requested. The other thing about using sage_fortran, that I had forgotten to change back when I removed it, is that it includes "-fPIC". I didn't check the fortran spkg but hopefully the work done by Dave to set the correct pic flag is picked up in sage_fortran.

If we don't adopt this, FFLAG="-fPIC" (or whatever mechanism Dave came up with) should be added in my previous version.

[ac9ad401-3030-4fb0-957e-6c14f81e046a]

comment:39

Replying to @kiwifb:

May be wait until numpy-1.5.1. it is not a big deal right now I guess. May be we should discuss it on sage-devel. At the moment we don't really have much testing apart from the fact I unleashed the upgrade to numpy-1.4.1/scipy-0.8 on sage-on-gentoo users. I am not sure we can believably merge this in 4.5.3 unless it takes more than one week before it is released. If we target 4.5.3 I say we stay with what we have now, if we target 4.6 which should be a little further down the track we go for 1.5.x.

Ok I think we should do the following: Let's stick to 1.4.1 since it is necessary for scipy 0.8. and quite well tested yet, in contrary to 1.5.x and it seems to work. Especially since the update from scipy 0.7 to 0.8 holds a lot of changes, and we don't know if they build some more bugs into numpy 1.5.1... If it turns out that updating from 1.4.1 to 1.5.1 is no problem, then well... packing a new package would'nt be the prob I think.

@new spkg: Ok I will give it a try tonight!

[kcrisman]

comment:40

There's a lot of packages etc. on this ticket. Can someone provide a concise list of what would be needed to apply on a given platform? (For instance, in a very cursory glance, the networkx package being here mystifies.)

[ac9ad401-3030-4fb0-957e-6c14f81e046a]

comment:41

Replying to @kcrisman:

There's a lot of packages etc. on this ticket. Can someone provide a concise list of what would be needed to apply on a given platform? (For instance, in a very cursory glance, the networkx package being here mystifies.)

Needed are: http://code.google.com/p/spkg-upload/downloads/detail?name=numpy-1.4.1.spkg http://code.google.com/p/spkg-upload/downloads/detail?name=scipy-0.8.spkg

after installing numpy, one needs so execute sage -ba, or else one get's runtime warnings.

You also need networkx-1.2. (the other networkx is just a small hack for testing, because 1.2. didn't build correctly on my machine, but this is obsolete now, since networkx-1.2 is merged into sage 4.3.alpha1)

[ac9ad401-3030-4fb0-957e-6c14f81e046a]

Description changed:

--- 
+++ 
@@ -1 +1,11 @@
 Since I really, really need them for my work, I will try to manage it to upgrade the scipy and numpy packages to the latest versions
+
+The packages can be found found under:
+
+http://code.google.com/p/spkg-upload/downloads/detail?name=numpy-1.4.1.spkg  
+http://code.google.com/p/spkg-upload/downloads/detail?name=scipy-0.8.spkg
+
+Important notes:
+after installing numpy, one needs so execute sage -ba, or else one get's runtime warnings.
+
+You also need networkx-1.2. (the other networkx is just a small hack for testing, because 1.2. didn't build correctly on my old sage versions, but this is obsolete now, since networkx-1.2 is merged into sage 4.3.alpha1) 

[ac9ad401-3030-4fb0-957e-6c14f81e046a]

Description changed:

--- 
+++ 
@@ -6,6 +6,6 @@
 http://code.google.com/p/spkg-upload/downloads/detail?name=scipy-0.8.spkg

 Important notes:
-after installing numpy, one needs so execute sage -ba, or else one get's runtime warnings.
+after installing numpy, one needs to execute sage -ba, or else one get's runtime warnings.

 You also need networkx-1.2. (the other networkx is just a small hack for testing, because 1.2. didn't build correctly on my old sage versions, but this is obsolete now, since networkx-1.2 is merged into sage 4.3.alpha1) 

[ac9ad401-3030-4fb0-957e-6c14f81e046a]

comment:44

I gave the links and (current) build instructions into the discription, so that people can find the latest versions more quickly

[ac9ad401-3030-4fb0-957e-6c14f81e046a]

Description changed:

--- 
+++ 
@@ -9,3 +9,5 @@
 after installing numpy, one needs to execute sage -ba, or else one get's runtime warnings.

 You also need networkx-1.2. (the other networkx is just a small hack for testing, because 1.2. didn't build correctly on my old sage versions, but this is obsolete now, since networkx-1.2 is merged into sage 4.3.alpha1) 
+
+trac_9808_numpy_doctest_change.patch in the attachment has to be applied, in order to get all doctests running because some of the output has changed.

[kcrisman]

comment:46

Included direct links to files in description.

[kcrisman]

Description changed:

--- 
+++ 
@@ -5,6 +5,12 @@
 http://code.google.com/p/spkg-upload/downloads/detail?name=numpy-1.4.1.spkg  
 http://code.google.com/p/spkg-upload/downloads/detail?name=scipy-0.8.spkg

+Files are found with direct links:
+
+http://spkg-upload.googlecode.com/files/numpy-1.4.1.spkg  
+http://spkg-upload.googlecode.com/files/scipy-0.8.spkg
+
+
 Important notes:
 after installing numpy, one needs to execute sage -ba, or else one get's runtime warnings.

[ac9ad401-3030-4fb0-957e-6c14f81e046a]

Description changed:

--- 
+++ 
@@ -14,6 +14,6 @@
 Important notes:
 after installing numpy, one needs to execute sage -ba, or else one get's runtime warnings.

-You also need networkx-1.2. (the other networkx is just a small hack for testing, because 1.2. didn't build correctly on my old sage versions, but this is obsolete now, since networkx-1.2 is merged into sage 4.3.alpha1) 
+You also need networkx-1.2. (the other networkx is just a small hack for testing, because 1.2. didn't build correctly on my old sage versions, but this is obsolete now, since networkx-1.2 is merged into sage 4.5.3.alpha1) 

 trac_9808_numpy_doctest_change.patch in the attachment has to be applied, in order to get all doctests running because some of the output has changed.

[kcrisman]

comment:49

This seems to work fine on Mac OS X 10.6.4 Intel. I will try to test it tomorrow on a PowerPC machine. Note: I haven't looked at the spkg installs or anything, this is just a data point with regard to whether it works, not a review.