Segfault when computing automorphisms

[videlec]

When computing automorphisms I constantly get a segfault. The reason might be due to the fact that the travis script installs nauty.h/libnauty.a before building Normaliz (which I did not do).

1. Are nauty.h and libnauty.la requirements to build Normaliz?
2. Should I expect segfault when I do not install nauty?

See https://trac.sagemath.org/ticket/27952

[sebasguts]

Hmm, this seems to be more like a bug in Normaliz than a PyNormaliz-Bug. I would guess there should be some exception thrown if you compile without Nauty and request an automorphism group to be computed. I will investigate.

[videlec]

If build with nauty is optional could we have a macro NAUTYNORMALIZ in Normaliz. That could be checked for in the configure.ac of PyNormaliz. There should also be a NmzHasNauty Python function.

[w-bruns]

The computation of automorphisms is not yet an official part of 3.7.3. You can build it without nauty. For example, see the Travis builds.

I am nit sure what the tests of PyNormaliz are ding. In Normaliz itself the existence of nauty is controlled by NMZ_NAUTY.

[sebasguts]

PyNormaliz just assumes the existence of the Nauty functionality. I will add the ifdefs and the function as @videlec suggested above.

[fingolfin]

I don't think any #ifdef's were added. Still, with the current master branch versions of Normaliz and PyNormaliz, I don't get a crash if I use a Normaliz w/o Nauty, though I do get some weird output I didn't expect (I expected an error):

>>> from PyNormaliz import Cone, NmzCone, NmzResult
>>> V = [[-1, -1, 1], [-1, 0, 1], [-1, 1, 1], [0, -1, 1], [0, 0, 1], [0, 1, 1], [1, -1, 1], [1, 0, 1], [1, 1, 1]]
>>> cube2 = NmzCone(vertices=V)
>>> NmzResult(cube2,"Automorphisms")
[0, [[], []], [[], []], [[], [[2], [3], [1], [0]]]]

The correct result apparently is [8, [[[], []], []], [[[0, 2, 1, 3], [1, 0, 3, 2]], [[0, 1, 2, 3]]], [[[1, 0, 3, 2], [0, 2, 1, 3]], [[0, 1, 2, 3]]]].

@w-bruns I think we are seeing the bad result because members Automs and getAutomorphismGroup are always present in class Cone, but if nauty is not available, the former is simply not modified (other than whatever the constructor does), and the latter effectively does nothing. And then tracking down things further, I arrive in AutomorphismGroup<Integer>::compute_inner which contains an #ifdef NMZ_NAUTY which simply "comments" out part of the code... but shouldn't this function rather signal an error (e.g. throw an exception) if nauty support was not compiled in?

[w-bruns]

Indeed, if the automorphism group is not computed (ornot computable) for whatever reason, libnormaliz should throw an exception. Will be zhecked.

[w-bruns]

We should try with an up-to-date version of libnormaliz. Looking at the code, I can only say that libnormaliz will throw an exception of one asks for the automorphism group and none has been computed.

[w-bruns]

This is a Normaliz problem, still present in 3.8.3. 8.3. Normaliz did not check the presence of nauty when "computing" automorphism groups, but returned an invalid result. Taken care of in current master.