Closed GoogleCodeExporter closed 9 years ago
Depending on the version of gmpy, there are two different strategies for memory
management. In older versions, the only benefit for the custom allocations was
an increase of the minimum memory size which decreases the number of
reallocations. In later versions, Python's internal memory manager is used. The
PyMem_ versions are slightly faster. Which version of gmpy are you using?
In either case, changing the memory allocation functions will impact the entire
GMP library so I agree that it shouldn't be done. I was planning on a bugfix
release soon so I'll look at making the standard memory allocation functions
the default.
Original comment by casevh
on 10 Nov 2011 at 4:09
Mandriva has python-gmpy 1.14 packaged.
I believe it should work if python-gmpy did use malloc/realloc/free,
as in older versions, but should defeat whatever is fixed in
sagemath by wrapping the malloc/realloc/free with the sig_off/sig_on
interfaces.
Right now, if gmpy is installed, it does not matter is MPMATH_NOGMPY
is set, and it calls mp_set_memory_functions from the gmpy shared
object, and afaik PyMem_* memory pointers cannot be used with standard
malloc/realloc/free.
If mpmath actually use gmpy in sagemath, I think there would also be
issues with sage and gmpy mp*_t object caches.
Note that actually it somehow works for most things, save for a few
very weird corruptions/crashes, and, if not setting MALLOC_CHECK_=1
it does not even print any warnings from glibc.
Original comment by paulo.ce...@gmail.com
on 10 Nov 2011 at 4:39
gmpy 1.14 uses the PyMem_ approach. Older versions just changed the minimum
memory size and then called the libc functions. I'll change 1.15 to offer three
options for memory management: do nothing, increase minimum memory size (like
older versions), or use PyMem. I'll let you know when I have a test version
available.
Original comment by casevh
on 10 Nov 2011 at 5:18
I've posted a test build in the Download section. I marked it deprecated since
I haven't tested it on anything other 64-bit Linux. The link is:
http://code.google.com/p/gmpy/downloads/detail?name=gmpy-1.15-test.zip
Please give it a try and let me know if it solves the issues with Sage. It
fixes a couple of other bugs, too.
Original comment by casevh
on 11 Nov 2011 at 6:56
It does not print any glibc messages besides my /usr/bin/sage
script sets MALLOC_CHECK_=1, and should be due to _PyInitGMP()
not calling mp_set_memory_functions().
But something is still happening, because the mpmath and
gmpy interaction should have been to not use gmpy, but the
sage cython code to handle integers, as the sage code does
several "short circuits" directly updating mpz_t fields,
calculating number of mp_limb_t, etc before some operations,
like converting to/from pari, ntl, pylong, etc numbers.
I mean, this:
$ GMPY_DEBUG=99 MPMATH_NOGMPY=1 sage -t -force_lib
"devel/sage/sage/rings/integer.pyx" >& log.txt
and in log.txt I see
sage -t -force_lib "devel/sage/sage/rings/integer.pyx"
[?1034hinitgmpy() called...
Entering set_pympzcache
Entering set_pympqcache
gmpy_module at 0x7f6c104922b8
gmpy_module imported copy_reg OK
Pygmpy_mpz() called...
Entering Pympz_new
Pympz_new is creating a new object
Initing new not in zcache
anynum2Pympz(0x228c9b0)->0x7f6c112f2530
Pygmpy_mpz: created mpz = 0
Pympz_dealloc: 0x7f6c112f2530
Pygmpy_mpq() called...
[...]
Not placing in full zcache(100/100)
Pympz_dealloc: 0x7f6c0f7417b0
Not placing in full zcache(100/100)
Pympz_dealloc: 0x7f6c0f741790
Not placing in full zcache(100/100)
For whitespace errors, see the file /home/pcpa/.sage//tmp/integer_12676.py
[6.6 s]
The sagemath doctest errors I see are or should be
due to using python 2.7 and a few other components
with different versions in my rpm packages but should
be worth to paste here. Updating the python-gmpy rpm
package to use gmp-1.5-test.zip as source (setting
GMPY_DEBUG=1 besides also setting MPMATH_NOGMPY is
enough to have more than 1k lines output).
x86_64:
$ sage -t -force_lib "devel/sage/sage/rings/integer.pyx"
sage -t -force_lib "devel/sage/sage/rings/integer.pyx"
**********************************************************************
File "/usr/share/sage/devel/sage/sage/rings/integer.pyx", line 4653:
sage: numpy.array(2**40).dtype
Expected:
dtype('int64')
Got:
dtype('object')
**********************************************************************
File "/usr/share/sage/devel/sage/sage/rings/integer.pyx", line 3046:
sage: n = -920390823904823094890238490238484; n.__hash__()
Expected:
6874330978542788722
Got:
-2623069716
**********************************************************************
File "/usr/share/sage/devel/sage/sage/rings/integer.pyx", line 3061:
sage: hash(n)
Expected:
-9223372036854767616
Got:
8192
**********************************************************************
File "/usr/share/sage/devel/sage/sage/rings/integer.pyx", line 3064:
sage: hash(n) == hash(int(n))
Expected:
True
Got:
False
**********************************************************************
2 items had failures:
1 of 9 in __main__.example_113
3 of 13 in __main__.example_63
***Test Failed*** 4 failures.
For whitespace errors, see the file /home/pcpa/.sage//tmp/integer_12790.py
[6.6 s]
----------------------------------------------------------------------
The following tests failed:
sage -t -force_lib "devel/sage/sage/rings/integer.pyx"
Total time for all tests: 6.6 seconds
i586 (same computer, "linux32 chroot ..."):
$ sage -t -force_lib "devel/sage/sage/rings/integer.pyx"
sage -t -force_lib "devel/sage/sage/rings/integer.pyx"
**********************************************************************
File "/usr/share/sage/devel/sage/sage/rings/integer.pyx", line 4653:
sage: numpy.array(2**40).dtype
Expected:
dtype('int64')
Got:
dtype('object')
**********************************************************************
1 items had failures:
1 of 9 in __main__.example_113
***Test Failed*** 1 failures.
For whitespace errors, see the file /home/pcpa/.sage//tmp/integer_12824.py
[13.5 s]
----------------------------------------------------------------------
The following tests failed:
sage -t -force_lib "devel/sage/sage/rings/integer.pyx"
Total time for all tests: 13.5 seconds
I do not know what else may be loading gmpy if
it is installed...
Original comment by paulo.ce...@gmail.com
on 11 Nov 2011 at 1:56
A couple of ideas. In gmpy.c, change the line
} options = { 0, 0, 5, 100, 128, 0 };
to
} options = { 0, 0, 5, 0, 0, 0 };
That will enable internal caching of deleted objects. It shouldn't be an issue,
but it might.
The hash() error message imply something is working with 32-bits and and
something else is working in 64-bit mode. Could there be an old version of gmpy
used by something else?
Try editing the mpmath "import gmpy" code to something like "import gmpyXXX" to
see if that stops the importing of gmpy.
Original comment by casevh
on 11 Nov 2011 at 3:22
After looking at grep -r 'gmpy' /usr/lib{,64}/python2.7
output, I got it to not load gmpy with these system wide
changes:
--- /usr/lib/python2.7/site-packages/sympy/core/numbers.py~ 2011-11-08
02:28:15.000000000 -0200
+++ /usr/lib/python2.7/site-packages/sympy/core/numbers.py 2011-11-11
13:42:23.352591670 -0200
@@ -1962,7 +1962,7 @@ except ImportError:
pass
try:
- import gmpy
+ import gmpyXX
def sympify_mpz(x):
return Integer(long(x))
--- /usr/lib/python2.7/site-packages/sympy/polys/domains/groundtypes.py~
2011-11-08 02:28:15.000000000 -0200
+++ /usr/lib/python2.7/site-packages/sympy/polys/domains/groundtypes.py
2011-11-11 13:44:15.954119582 -0200
@@ -2,16 +2,16 @@
from sympy.external import import_module
-HAS_GMPY = True
+HAS_GMPY = False
# Versions of gmpy prior to 1.03 do not work correctly with int(largempz)
# For example, int(gmpy.mpz(2**256)) would raise OverflowError.
# See issue 1881.
-gmpy = import_module('gmpy', min_module_version='1.03',
- module_version_attr='version', module_version_attr_call_args=())
+#gmpy = import_module('gmpy', min_module_version='1.03',
+# module_version_attr='version', module_version_attr_call_args=())
-HAS_GMPY = bool(gmpy)
+#HAS_GMPY = bool(gmpy)
from __builtin__ import (
int as PythonIntegerType,
so that running:
$ GMPY_DEBUG=99 MPMATH_NOGMPY=1 sage -t -force_lib
"devel/sage/sage/rings/integer.pyx"
does not print any debug from gmpy.
Note that sympy in sagemath is sympy-0.6.4 while Mandriva have
$ rpm -q python-sympy
python-sympy-0.7.1-2-mdv2012.0.noarch
The sympy in the sagemath spkg does not use either gmpy or
mpmath, so, these issues are probably not known to upstream.
Offtopic: I recently imported a gentoo patch to force sympy to
use system mpmath instead of an internal copy, after keeping
rediff'ing for quite some time a patch to force sagemath to
use sympy's mpmath.
Ontopic: Using the same approach I used for mpmath for some
time where I added a "Conflicts: python-mpmath", now I added
a "Conflicts: python-gmpy" to the sagemath rpm spec file. I
wish it would not be required...
The hash errors still happen when not loading gmpy and should
be a bug in sagemath with python 2.7 (upstream sage uses
python 2.6) -> http://trac.sagemath.org/sage_trac/ticket/11986
For extra information, with the patches above, it not setting
MPMATH_NOGMPY=1, and still toying with install/unsintall of
the rpm I generated with gmpu-1.5-test.zip:
$ GMPY_DEBUG=99 sage -t -force_lib "devel/sage/sage/rings/integer.pyx"
sage -t -force_lib "devel/sage/sage/rings/integer.pyx"
initgmpy() called...
Entering set_pympzcache
Entering set_pympqcache
gmpy_module at 0x7f53ae42f638
gmpy_module imported copy_reg OK
Pygmpy_mpz() called...
Entering Pympz_new
Pympz_new is creating a new object
Initing new not in zcache
anynum2Pympz(0x22b49b0)->0x7f53b53e3db0
Pygmpy_mpz: created mpz = 0
Pympz_dealloc: 0x7f53b53e3db0
Pygmpy_mpq() called...
Entering Pympq_new
Pympq_new is creating a new object
Initing new not in qcache
Initing new not in qcache, done
anynum2Pympq(0x22b49b0)->0x7f53ae435b70
Pygmpy_mpq: created mpq = 0
Pympq_dealloc: 0x7f53ae435b70
Pygmpy_mpf() called...
prec 64 size 0 toclear -2 carry 0
anynum2Pympf(0x22b49b0,0)->0x7f53ae435ba0 (53)
Pygmpy_mpf: created mpf = 0.e0 bits=53 (0)
Pympf_dealloc: 0x7f53ae435ba0
gmpy_module enable pickle OK
**********************************************************************
File "/usr/share/sage/devel/sage/sage/rings/integer.pyx", line 4653:
sage: numpy.array(2**40).dtype
Expected:
dtype('int64')
Got:
dtype('object')
**********************************************************************
File "/usr/share/sage/devel/sage/sage/rings/integer.pyx", line 3046:
sage: n = -920390823904823094890238490238484; n.__hash__()
Expected:
6874330978542788722
Got:
-2623069716
**********************************************************************
File "/usr/share/sage/devel/sage/sage/rings/integer.pyx", line 3061:
sage: hash(n)
Expected:
-9223372036854767616
Got:
8192
**********************************************************************
File "/usr/share/sage/devel/sage/sage/rings/integer.pyx", line 3064:
sage: hash(n) == hash(int(n))
Expected:
True
Got:
False
**********************************************************************
2 items had failures:
1 of 9 in __main__.example_113
3 of 13 in __main__.example_63
***Test Failed*** 4 failures.
For whitespace errors, see the file /home/pcpa/.sage//tmp/integer_17350.py
[6.7 s]
----------------------------------------------------------------------
The following tests failed:
sage -t -force_lib "devel/sage/sage/rings/integer.pyx"
Total time for all tests: 6.7 seconds
Original comment by paulo.ce...@gmail.com
on 11 Nov 2011 at 4:03
The hash problem still bothers me. I don't think there was a change in Python's
hash() from 2.6 to 2.7. There is a major change in the values calculated by
hash() in Python 3.2. Does the hash issue go away in a 32-bit build of Sage?
I don't have Sage running on any of my systems at the moment. I'll see if I can
get it installed and produce the issue. Is this a Mandriva specific issue?
Original comment by casevh
on 11 Nov 2011 at 6:10
I've released 1.15. I was not able to find any differences in the has function
with Python 2.7. I believe Sage has some functions that mutate the memory
contents of an integer. Since gmpy caches the hash value, that may be related
to errors in the hash values.
Original comment by casevh
on 13 Jan 2012 at 6:58
Sorry for the long delay in the response.
I confirm, as expected, it no longer causes memory corruption due to both
sagemath, and later gmpy conflicting usage of mp_set_memory_functions, and
sympy uses gmpy without problems.
The hashing issue is not related to gmpy, and the doctest failures are
corrected when applying the patch at
http://trac.sagemath.org/sage_trac/ticket/11986
I will submit a newer sagemath package to Mandriva buildsystem, that now
conflicts only with "python-gmpy < 1.15" (gpmy 1.15 already packaged) as well
as a few other corrections, including the integer hash patch above.
Original comment by paulo.ce...@gmail.com
on 18 Jan 2012 at 9:17
Original issue reported on code.google.com by
paulo.ce...@gmail.com
on 10 Nov 2011 at 2:46