IntegerLattice class

[malb]

This ticket implements sage.lattices.integer_lattice.IntegerLattice which represents a discrete subgroup of ZZⁿ.

This class is inspired by #15955 but except for the voronoi cell implementation, it is implemented anew from scratch.

This ticket also includes an updated interface to fpLLL which uses Cython's C++ features, uses the fpLLL 4.0 API and interfaces not only with LLL but also with fpLLL's BKZ and shortest vector enumeration.

Component: linear algebra

Author: Martin Albrecht

Branch: 6957074

Reviewer: Daniel Krenn, Travis Scrimshaw

Issue created by migration from https://trac.sagemath.org/ticket/15976

[malb]

Commit: f9b4bdd

[malb]

New commits:

6bad3f6	first draft of IntegerLattice class
f9b4bdd	modern interface to fpLLL

[malb]

Branch: u/malb/15976_integer_lattice

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from f9b4bdd to 9a76b85

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

9a76b85

documentation changes

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

ab6ec77

actually use fpLLL in IntegerLattice.shortest_vector

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 9a76b85 to ab6ec77

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

a13b4aa

added more of a class hierarchy

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from ab6ec77 to a13b4aa

[simon-king-jena]

comment:6

Hi Martin!

You asked me to have a look at your code from the categorical point of view.

It really depends what you want to model.

Currently, you start on top of SageObject, which is about as basic as it can be. A SageObject hasn't even support for different categories. So, if for you a lattice is not more than an object, then it's fine.

A bit less basic is CategoryObject, which is for objects in arbitrary categories. In particular, it would be a suitable starting point for objects that do not have elements (i.e., whose category is not a sub-category of the category of sets). So, if you want that your lattice is a semigroup, but you do not want to implement elements of the semigroup, then it's fine---almost. AFAIK, it is mathematically possible to define the notion of a semigroup just in terms of morphisms and direct products, and it is not needed to refer to elements in the definition. However, Sage's category framework expects semigroups to be sets, and thus the TestSuite would complain that element creation fails.

If your lattices are supposed to be groups (in particular, sets), then the lattice should inherit from Parent, and you should implement elements as well, inheriting from Element (or rather ModuleElement, since you want to add elements). I guess it would be fine to import some implementation of integer vectors (I am sure those things exist somewhere in Sage), and assign it as a class attribute IntegerLattice.Element.

Next, if you decide to inherit from Parent and assign .Element, you should do self._init_category_(Groups()) (or a better category, if there is any) during initialisation of your lattice. This should be enough to enable element creation and let the test suites (TestSuite(YourLattice).run()) pass. If it isn't---well, then we'll take care of it later.

It could make sense to have a look at sage.combinat.free_module.CombinatorialFreeModule and sage.combinat.free_module.CombinatorialFreeModuleElement. I have not much experience with it, but I know that people use it to implement all kind of things, including algebras. And it does inherit from Parent.

On a different note: Would it make sense to use UniqueRepresentation or at least CachedRepresentation for lattices? It seems strange to me to have a function Lattice that does nothing more than hand the arguments to the IntegerLattice constructor and return the result.

Provided that you want some kind of caching for your lattices, you could do

class Lattice(CachedRepresentation, Parent):
    "This is a stub, non-integral lattices will be implemented later"
    Element = some.suitable.VectorClass
    @staticmethod
    def __classcall_private__(cls, basis, lll_reduce):
        # do some preprocessing of arguments, throw
        # an error if the basis is not integral
        return IntegerLattice(preprocessed_basis, lll_reduce)
    def __cmp__(...):
        # We don't inherit from UniqueRepresentation, hence, we admit the
        # possibility that different bases result in the same lattice.
        # Perhaps move this to IntegerLattice, unless the same algorithm
        # would work for general lattices.

class IntegerLattice(Lattice):
    ...

[malb]

comment:7

Replying to @simon-king-jena:

Hi Simon,

thanks a lot for your comments!

If your lattices are supposed to be groups (in particular, sets), then the lattice should inherit from Parent, and you should implement elements as well, inheriting from Element (or rather ModuleElement, since you want to add elements). I guess it would be fine to import some implementation of integer vectors (I am sure those things exist somewhere in Sage), and assign it as a class attribute IntegerLattice.Element.

Next, if you decide to inherit from Parent and assign .Element, you should do self._init_category_(Groups()) (or a better category, if there is any) during initialisation of your lattice. This should be enough to enable element creation and let the test suites (TestSuite(YourLattice).run()) pass. If it isn't---well, then we'll take care of it later.

I tried to go down this route: I inherited from Parent and set self._init_category_(CommutativeAdditiveGroups()). The TestSuite(L).run() now first complaints that there is no _an_element_() which is easily fixable. However, the next complaint is that elements created by some_element() do not live in the lattice but in Zⁿ. I am not sure I want my lattice vectors to live in the lattice, but if I wanted to do that, I'd have to define my lattice as a proper submodule of Zⁿ? I'd view the lattice somewhat analogously to an ideal in a multivariate ring, would we want our ideal elements to live anywhere but the ring?

On a different note: Would it make sense to use UniqueRepresentation or at least CachedRepresentation for lattices? It seems strange to me to have a function Lattice that does nothing more than hand the arguments to the IntegerLattice constructor and return the result.

I don't want to cache my representations or make lattices unique. It is useful to have two objects representing the same lattice: one with a good basis and one with a bad basis (that's the secret and public key respectively in many crypto systems). I'd expect Lattice to grow when the need arises: lattices over Q, over R, over polynomial rings and stuff.

[simon-king-jena]

comment:8

Replying to @malb:

However, the next complaint is that elements created by some_element() do not live in the lattice but in Zⁿ. I am not sure I want my lattice vectors to live in the lattice, but if I wanted to do that, I'd have to define my lattice as a proper submodule of Zⁿ?

There is something called "facade". I didn't try to fully understand it, but I think the idea is: If you have the set of prime numbers, then its elements are just natural numbers. That's to say, the parent of the elements of Primes() is ZZ, not Primes():

sage: P = Primes()
sage: p = P.an_element()
sage: p
43
sage: p.parent()
Integer Ring

This is done by telling that Primes() is a "facade set":

sage: P.category()
Category of facade infinite enumerated sets

Your situation could be similar: You have a lattice L in RR^n, and the parent of the elements of L wouldn't be L but RR^n (but please not ZZ^n, since you need the embedding!).

So, could be that using

sage: Category.join([FacadeSets(),CommutativeAdditiveGroups()])
Category of facade commutative additive groups

to initialise your lattice's category would suffice.

I'd view the lattice somewhat analogously to an ideal in a multivariate ring, would we want our ideal elements to live anywhere but the ring?

No, see above. But for historical reasons, we have

sage: P.<x,y> = QQ[]
sage: I = P*[x^2+y,y^2+x]
sage: x^2 in I
False
sage: x^2+y in I
True
sage: I.category()
Category of ring ideals in Multivariate Polynomial Ring in x, y over Rational Field

without facade. I think today it would be implemented using facades.

I don't want to cache my representations or make lattices unique. It is useful to have two objects representing the same lattice: one with a good basis and one with a bad basis (that's the secret and public key respectively in many crypto systems).

That's why I suggested CachedRepresentation rather than UniqueRepresentation. If you use CachedRepresentation than starting twice with the same basis yields identical lattices, but starting with different bases will yield non-identical lattices that may still be equal. Only UniqueRepresentation would imply a "comparison by identity".

[simon-king-jena]

comment:9

Replying to @simon-king-jena:

Your situation could be similar: You have a lattice L in RR^n, and the parent of the elements of L wouldn't be L but RR^n (but please not ZZ^n, since you need the embedding!).

Ahm, or actually it should be ZZ^n, since you have integer lattices. But this ZZ^n is a subset of RR^n and should not be confused with the fact that the lattice itself is isomorphic to ZZ^n.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

2c8199c

lattices inherit from Parent, IntegerLattice is a Facade for ZZ^n

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from a13b4aa to 2c8199c

[malb]

comment:11

Thanks Simon,

Facade seems to be exactly what I needed!

On the CachedRepresentation I think even that is too much. One could, for example, construct the same lattice twice and then run lattice reduction on one instance in order to get a good basis. The design is somewhat unusual as it changes the basis during the life-time of an object.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

96e771e

added HKZ reduction

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 2c8199c to 96e771e

[simon-king-jena]

comment:13

Replying to @malb:

On the CachedRepresentation I think even that is too much. One could, for example, construct the same lattice twice and then run lattice reduction on one instance in order to get a good basis. The design is somewhat unusual as it changes the basis during the life-time of an object.

OK, it sounds like caching would be bad in such situation.

[malb]

comment:14

Okay, excellent. So - as far as I am concerned - our work here is done :)

[darijgr]

comment:15

Are you guys sure that special characters (such as · and δ) don't break the PDF documentation? Just asking.

What exactly does the jacobi method do? Sage does have a Cholesky decomposition method, but that's different (yours is rational, which is why I'm curious). Any references or explanations? (They should also go into the docstring -- also, a doctest on something other than a scalar matrix would be nice...)

Finally, I'd suggest not using such a short and generic name for a class like Lattice which is not likely to be implemented in foreseeable future (shouldn't the inexactness of the reals prevent such class from being implementable?).

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 96e771e to 43d76e8

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

43d76e8

added lattices to reference manual and fixed pdf building errors

[malb]

comment:17

Replying to @darijgr:

Are you guys sure that special characters (such as · and δ) don't break the PDF documentation? Just asking.

It didn't, but now it does. Thanks!

What exactly does the jacobi method do? Sage does have a Cholesky decomposition method, but that's different (yours is rational, which is why I'm curious). Any references or explanations? (They should also go into the docstring -- also, a doctest on something other than a scalar matrix would be nice...)

I cannot speak for that part of the code as it was taken on-board from the 2012 GSoC project. It seemed like the only non-trivial contribution and I didn't want to delete it. I never touched it. If no-one is comfortable with it, we can drop it though. I asked Daniel Krenn to comment (he was the mentor on that GSoC project)

Finally, I'd suggest not using such a short and generic name for a class like Lattice which is not likely to be implemented in foreseeable future (shouldn't the inexactness of the reals prevent such class from being implementable?).

I don't see the problem with the class itself, that's what namespaces are for, right? Lattice as the basic class for lattices (as discrete subgroups of RRⁿ) seems about right. However, I do take your point with Lattice is in the global namespace (i.e. the constructor). Any recommendation?

---

New commits:

43d76e8

added lattices to reference manual and fixed pdf building errors

---

New commits:

43d76e8

added lattices to reference manual and fixed pdf building errors

[darijgr]

comment:18

Oh, if it's not in the global namespace, then it's fine. Otherwise, what about RealLattice?

[darijgr]

Changed commit from 43d76e8 to fe8836f

[darijgr]

Changed branch from u/malb/15976_integer_lattice to public/ticket/15976

[darijgr]

comment:19

ignore this one

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from fe8836f to d5c404c

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

7f5210d	Merge branch 'u/malb/15976_integer_lattice' of git://trac.sagemath.org/sage into ila
d5c404c	jacobi method documented (and now without cruft from other branch)

[darijgr]

comment:21

OK, so the method does do something like Cholesky decomposition, except that it never takes square roots and has some diagonal factors. (Don't get me wrong: this makes the method more useful, not wrong.) I've added documentation. Warning: branch change.

I fear this is all I can do for this ticket, though...

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

f5e8e16

Lattice -> RealLattice

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from d5c404c to f5e8e16

[malb]

comment:23

I renamed Lattice to RealLattice

[dimpase]

comment:24

Replying to @malb:

I renamed Lattice to RealLattice

bad choice, IMHO. People will want to have lattices in things like C^n

[malb]

comment:25

Suggestions? Lattice seems to be too general for many, RealLattice too specific for others. ComplexLattice doesn't really seem to make much sense because one normally thinks of a lattice as a discrete subgroup of Rⁿ (at least in my neck of the woods).

[dimpase]

comment:26

How about VectorSpaceLattice ? (well, not all fields might be supported, but still).

[malb]

comment:27

Wouldn't people confuse that with vector lattices? Or is that not a common notion? cf. http://www.stat.yale.edu/~pollard/Books/Asymptopia/old-Vectorlattice.pdf

I do have to admit that I also find it a bit awkward but that's clearly a question of taste.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from f5e8e16 to e0f6e3c

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

e0f6e3c

fixed copyright notice

[dimpase]

comment:29

Replying to @malb:

Wouldn't people confuse that with vector lattices? Or is that not a common notion? cf. http://www.stat.yale.edu/~pollard/Books/Asymptopia/old-Vectorlattice.pdf

I do have to admit that I also find it a bit awkward but that's clearly a question of taste.

hmm, OK, how about AdditiveTorsionFreeAbelianGroup ? This would avoid that other lattices all together...

[malb]

comment:30

now that's just mean … for now VectorSpaceLattice seems to be a winner.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

f1686df

removed sage.lattices and mode sage.lattices.integer_lattice to sage.modules.free_module_integer

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from e0f6e3c to f1686df

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

37b3baa	fixed documentation
34734a1	guard shortest vector computation with sig_on/sig_off

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from f1686df to 34734a1

[malb]

comment:33

The patch is a lot less invasive now, anyone up for reviewing it?

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 34734a1 to eb352db

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

9872997	Merge branch 'develop' of trac.sagemath.org:sage into integer_lattices
eb352db	Merge branch 'develop' of trac.sagemath.org:sage into integer_lattices

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

44d20d5

fixed incomplete merge