pynormaliz fails to build on 32bit system

[edd8e884-f507-429a-b577-5d554626c0fe]

See the attached log.

Upstream tarballs:

• https://github.com/Normaliz/Normaliz/releases/download/v3.2.1/normaliz-3.2.1.tar.gz

• https://pypi.python.org/packages/5e/8e/2e4f68fb395ea834be0bdd3adf3c1787e320cbb6007e0c16ff6529480ed9/PyNormaliz-1.5.tar.gz to be renamed PyNormaliz-1.5.tar.gz

CC: @mkoeppe @videlec @w-bruns

Component: packages: optional

Keywords: days86, sdl

Author: Thierry Monteil

Branch/Commit: f64520f

Reviewer: Matthias Koeppe

Issue created by migration from https://trac.sagemath.org/ticket/22684

[edd8e884-f507-429a-b577-5d554626c0fe]

comment:45

This time i go until 2671113, i do not get a crash, but a bad_alloc catched by the interface, here are the last lines:

2671112
2671113
---------------------------------------------------------------------------
interface_error                           Traceback (most recent call last)
<ipython-input-3-19b874c06a0f> in <module>()
----> 1 len(P.integral_points())

/opt/sagemath/sage-source/local/lib/python2.7/site-packages/sage/geometry/polyhedron/backend_normaliz.py in integral_points(self, threshold)
    576         cone = self._normaliz_cone
    577         assert cone
--> 578         for g in PyNormaliz.NmzResult(cone, "ModuleGenerators"):
    579             assert g[-1] == 1
    580             points.append(vector(ZZ, g[:-1]))

interface_error: std::bad_alloc

If i re-run the sage: len(P.integral_points()) command again, from the same Sage command line, it only goes to 82:

79
80
81
82
---------------------------------------------------------------------------
interface_error                           Traceback (most recent call last)
<ipython-input-4-19b874c06a0f> in <module>()
----> 1 len(P.integral_points())

/opt/sagemath/sage-source/local/lib/python2.7/site-packages/sage/geometry/polyhedron/backend_normaliz.py in integral_points(self, threshold)
    576         cone = self._normaliz_cone
    577         assert cone
--> 578         for g in PyNormaliz.NmzResult(cone, "ModuleGenerators"):
    579             assert g[-1] == 1
    580             points.append(vector(ZZ, g[:-1]))

interface_error: std::bad_alloc

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 5858acb to 28aaa7d

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

28aaa7d

#22684 : add tests suggested at comment 44.

[d6056c42-5b65-4eea-a816-4019a4d0b1e1]

comment:47

Let us try to go one step further.

The std::bad_alloc exception shown by crash.log occurs when the vector w in line 778 of matrix.cpp is to be allocated:

vector<Integer> w(nc,0);

The only reason why this could fail because of a Normaliz bug is a corrupted value of nc (the number of columns of the matrix *this). Therefore I suggest to insert

cout << nc << endl;

before this line. When our counter starts running, its printed value must always be followed by 4 (in the critical example). If there is a 4 immediately before the exception is thrown, then the std::bad_alloc is a system problem.

[edd8e884-f507-429a-b577-5d554626c0fe]

comment:48

Here is how it ends:

sage: P = Polyhedron(vertices=((0, 0), (1789345,37121))) + 1/1000*polytopes.hypercube(2)
....: P = Polyhedron(vertices=P.vertices_list(), backend='normaliz', verbose=True)
....: len(P.integral_points())
....: 
# Calling PyNormaliz.NmzCone(['vertices', [[-1, -1, 1000], [-1, 1, 1000], [1, -1, 1000], [1789345001, 37121001, 1000], [1789345001, 37120999, 1000], [1789344999, 37121001, 1000]], 'cone', [], 'subspace', []])
3
3
3
3
3
3
3

[snip]

2698103
4
2698104
4
2698105
4
2698106
4
2698107
4
2698108
4
2698109
4
2698110
4
---------------------------------------------------------------------------
interface_error                           Traceback (most recent call last)
<ipython-input-1-2df9fc6cdad6> in <module>()
      1 P = Polyhedron(vertices=((Integer(0), Integer(0)), (Integer(1789345),Integer(37121)))) + Integer(1)/Integer(1000)*polytopes.hypercube(Integer(2))
      2 P = Polyhedron(vertices=P.vertices_list(), backend='normaliz', verbose=True)
----> 3 len(P.integral_points())

/opt/sagemath/sage-source/local/lib/python2.7/site-packages/sage/geometry/polyhedron/backend_normaliz.pyc in integral_points(self, threshold)
    576         cone = self._normaliz_cone
    577         assert cone
--> 578         for g in PyNormaliz.NmzResult(cone, "ModuleGenerators"):
    579             assert g[-1] == 1
    580             points.append(vector(ZZ, g[:-1]))

interface_error: std::bad_alloc

So indeed, the last thing before the exception is a 4.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 28aaa7d to 2d37127

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

fbce6fe	Merge branch 'u/tmonteil/pynormaliz_fails_to_build_on_32bit_system' of trac.sagemath.org:sage into HEAD
2d37127	#22684 : add tests suggested at comment 47.

[edd8e884-f507-429a-b577-5d554626c0fe]

comment:50

If i understand your last comment, it is a system problem. I upgraded my kernel and libc (Debian jessie), rebooted, and rebuilt normaliz but the problem persists. Is there something to do ? Does this mean that the problem comes from my computer and not from the normaliz code ?

[d6056c42-5b65-4eea-a816-4019a4d0b1e1]

comment:51

I really don't think that the problem comes from the Normaliz code. The last think Normaliz tries to do is to correctly allocate a vector with 4 components, and the system responds by a std::bad_alloc. I am not a system expert. The only explanation I can think of is that you run out of memory.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

797a967

Merge branch 'develop' into HEAD

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 2d37127 to 797a967

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 797a967 to 240ce84

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

240ce84

#22684 : lower memory requirement of a normaliz doctest.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 240ce84 to 2dd1ca5

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

2dd1ca5

22684 : lower memory requirement of a normaliz doctest.

[edd8e884-f507-429a-b577-5d554626c0fe]

comment:55

Replying to @w-bruns:

I really don't think that the problem comes from the Normaliz code. The last think Normaliz tries to do is to correctly allocate a vector with 4 components, and the system responds by a std::bad_alloc. I am not a system expert. The only explanation I can think of is that you run out of memory.

You are right, it seems that the weirdness of the error came from a ulimit (3GB for Sage) i set to protect the memory of my computer. If i remove it, then the process goes until my computer freezes out of memory.

Hence, i reduced the size of the culprit polytope by a factor 10 in one direction (hope it is still illustrative, please tell me).

Also, i kept the patch suggested at comment:27.

[mkoeppe]

comment:56

The checksum for pynormaliz is wrong.

[mkoeppe]

comment:57

Replying to @sagetrac-tmonteil:

i reduced the size of the culprit polytope by a factor 10 in one direction (hope it is still illustrative, please tell me).

Yes, I agree with this change.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

f64520f

#22684 : update PyNormaliz checksums.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 2dd1ca5 to f64520f

[edd8e884-f507-429a-b577-5d554626c0fe]

comment:59

Weird indeed. I checked: the current tarball contains:

PyNormaliz-1.5/
PyNormaliz-1.5/README
PyNormaliz-1.5/PyNormaliz.py
PyNormaliz-1.5/COPYING
PyNormaliz-1.5/GPLv2
PyNormaliz-1.5/NormalizModule.cpp
PyNormaliz-1.5/PKG-INFO
PyNormaliz-1.5/setup.py

while the previous contains:

PyNormaliz-1.5/
PyNormaliz-1.5/.gitignore
PyNormaliz-1.5/.travis-install.sh
PyNormaliz-1.5/.travis-test.sh
PyNormaliz-1.5/.travis.yml
PyNormaliz-1.5/COPYING
PyNormaliz-1.5/GPLv2
PyNormaliz-1.5/MANIFEST.in
PyNormaliz-1.5/Makefile
PyNormaliz-1.5/NormalizModule.cpp
PyNormaliz-1.5/PyNormaliz.py
PyNormaliz-1.5/README
PyNormaliz-1.5/Readme.md
PyNormaliz-1.5/examples/
PyNormaliz-1.5/examples/PyNormaliz_Tutorial.ipynb
PyNormaliz-1.5/examples/first.py
PyNormaliz-1.5/examples/first_long.py
PyNormaliz-1.5/examples/simple.py
PyNormaliz-1.5/setup.py

The common files are the same. I updated the checksums but perhaps something went wrong there.

[d6056c42-5b65-4eea-a816-4019a4d0b1e1]

comment:60

In the next version 3.3.0 the algorithm that computes the lattice points in this example will be significantly improved. In particular it will use less memory by discarding superfluous vectors as early as possible (and not as late as possible).

[mkoeppe]

Reviewer: Matthias Koeppe

[vbraun]

Changed branch from u/tmonteil/pynormaliz_fails_to_build_on_32bit_system to f64520f

[edd8e884-f507-429a-b577-5d554626c0fe]

Changed keywords from days86 to days86, sdl