sagemath / sage

Main repository of SageMath
https://www.sagemath.org
Other
1.37k stars 466 forks source link

Finite field lattices via Conway polynomials #8335

Closed roed314 closed 11 years ago

roed314 commented 14 years ago

Implements coercion within lattices of finite fields lying above the same prime when implemented with Conway polynomials.

sage: k = GF(9, conway=True, prefix='z')
sage: l = GF(27, conway=True, prefix='z')
sage: x = k.gen() + l.gen(); x
z6^5 + 2*z6^4 + 2*z6^3 + z6^2 + 2*z6 + 1
sage: x.parent()
Finite Field in z6 of size 3^6

When using the conway and prefix parameters, one does not need to specify an explicit variable name; if no variable name is given, it is constructed from the prefix and the degree (as in the above code snippet).

In the future, the functionality of this ticket will be incorporated into that for algebraic closures of finite fields. It will then be possible to construct compatible systems of finite fields outside the range of the Conway polynomial database using the pseudo-Conway polynomials from #14958: polynomials that satisfy all of the algebraic constraints on Conway polynomials without the lexicographic constraint that imposes uniqueness.

Apply:

Depends on #14958 Depends on #12142

CC: @defeo @rbeezer @sagetrac-hds @simon-king-jena @zimmermann6 @xcaruso @pjbruin @sagetrac-mraum @fredstro @sagetrac-JCooley @loefflerd @sagetrac-dfesti

Component: algebra

Keywords: days49 sd51

Author: David Roe, Jean-Pierre Flori, Peter Bruin

Reviewer: Jean-Pierre Flori, Luca De Feo

Merged: sage-5.13.beta1

Issue created by migration from https://trac.sagemath.org/ticket/8335

jpflori commented 11 years ago
comment:80

Attachment: trac_8335-fixes-5.11.b3.patch.gz

I have some failing tests on top of 5.11.b3 but they seem completely unrelated (something with get_test_shell() and I did not test a vanilla install, so they were surely there without these patches).

pjbruin commented 11 years ago
comment:81

Continuing the discussion from #13214; in the context of my remark

there should probably be two categories into which a finite field can be put:

  • the category of all finite fields. In this category, between any two objects there are either several morphisms or none at all, but no canonical one.
  • the category of finite subfields of a given algebraic closure of Fp. In this category there is at most one morphism beteen any two objects, namely the inclusion qua subfields of the given algebraic closure.

Jean-Pierre Flori wrote (referring to the second option)

8335 provides such an imlementation, though it not really practical for large fields and there is no proper categorical framework as you suggest.

This framework could be implemented in an independent ticket (and should if we want to be able to merge some tickets in a finite amount of time).

Certainly; both this ticket and #13214 are already large enough. The question is whether (a draft of) a categorical framework (i.e. algebraic closure of Fp) should be made first, or whether the new code from this ticket should be inserted into the current framework (which implements the first of the two categories) and be moved to a new framework as soon as we have it.

I would personally prefer the first option to keep things better packaged; this patch seems to make (pseudo-)Conway polynomials pop up in many different places, and moving them all to one place would require another intrusive Trac ticket later.

jpflori commented 11 years ago
comment:82

Replying to @pjbruin:

I would personally prefer the first option to keep things better packaged; this patch seems to make (pseudo-)Conway polynomials pop up in many different places, and moving them all to one place would require another intrusive Trac ticket later.

As far as functionalities are concerned, remember that Sage currently does not support

K = GF(p^n)

So pseudo Conway polynomials never appear where they did not use to. If you issue the command line which is currently supported:

K.<a> = GF(p^n)

you will get the exact same behavior as before, unless he specifies modulus="conway" and wants an extension of too large cardinality; maybe that should be changed back.

Nevertheless I agree that a user coming from Magma where

K := GF(p, n);

works might be confused... Though the user might although expect embeddings of finite fields to work out of the box.

But as I just realized I guess your concern is about the dissemination of code. From what I see, apart from code in finite_field_base.py, changes to specific finite field implementations mostly consists in replacing the part about Conway polynomials and tweak it to work with pseudo-Conway ones as well. Nonetheless it's true that properly moving all of that later will be intrusive.

But what about plain Conway polynomials? Shouldn't that be moved as well? Or do we consider they are standard enough to belong in the FiniteFields() category? But if they do it would be a waste not to use automagically the fact that they provide simple embeddings into each other, wouldn't it? though it would make the separation between the plain finite fields and subfields of a given algebraic closure blurrier.

(As you can guess, I'd prefer to get this merged first especially because I hate seeing functional code bitrotting for years on trac, but I get your point :))

pjbruin commented 11 years ago
comment:83

Replying to @jpflori:

As far as functionalities are concerned, remember that Sage currently does not support

K = GF(p^n)

So pseudo Conway polynomials never appear where they did not use to. If you issue the command line which is currently supported:

K.<a> = GF(p^n)

you will get the exact same behavior as before, unless he specifies modulus="conway" and wants an extension of too large cardinality; maybe that should be changed back.

Deciding what exactly GF(p^n) should mean (if it should mean anything at all) is an important design decision, and it is not at all obvious that Sage should make the same choice as Magma. Probably it is better not to make this decision in this ticket, which already adds a lot of code. Besides, letting an important change of behaviour depend whether the user specify a variable name or not sounds like a risky idea.

But what about plain Conway polynomials? Shouldn't that be moved as well? Or do we consider they are standard enough to belong in the FiniteFields() category?

They certainly belong to the general finite fields code, but not in any specific implementation, in my opinion. In fact, the goal of the (by now 2) patches that I'm about to post is as follows:

For backward compatibility (unpickling), the existing implementations will continue to accept string values for the parameter modulus, but new ones (such as the new PARI interface, see #12142) won't have to. The idea is that the specific implementations should "concentrate on doing their job", and things related to magic values of modulus should be in only one place.

But if they do it would be a waste not to use automagically the fact that they provide simple embeddings into each other, wouldn't it?

As I see it, that should depend on whether you are in the category of all finite fields or in the category of subfields of an algebraic closure of Fp.

(As you can guess, I'd prefer to get this merged first especially because I hate seeing functional code bitrotting for years on trac, but I get your point :))

Of course I understand that you want to see this finally appear in Sage, and I agree that it is a shame that Sage could have had something like this for years and still doesn't. I guess it will be easier if this big ticket is split into smaller pieces. It tries to do many rather different things at once: implement pseudo-Conway polynomials, adapt the construction of finite fields to use these, implement automatic coercion between the resulting fields, give a meaning to GF(p^n), and in the process add some new methods to polynomial elements. This makes it harder than necessary to understand and to review.

defeo commented 11 years ago
comment:84

Replying to @pjbruin:

This discussion looks like the dear old dichotomy between quick feature integration and long specification design.

Having some kind of support for lattices of finite fields has been a long standing request. I agree with pbruin that a better interface between generic finite fields and their actual implementation would be beneficial. But this ticket is ready for review, while pbruin's is not. Would it be that hard to adapt pbruin's or any other interface if this ticket is merged? I'm willing to give positive review to this ticket, if it stands some more testing, and it doesn't mess too much with #12142.

I'm not convinced that the interface can be decided independently of the actual algorithms. Magma's interface is engineered around the fact that constructing fields is fast, but constructing the embeddings is slow (hence the Embed function, which must explicitly be called by the user). If Sage ends up having a different construction (e.g., De Smit-Lenstra lattices... although I've looked into it, and I don't think it is viable in general), I think the interface could be different.

There are many solutions to the compatibly embedded finite fields problem, no one being ideal. I'm more in favor of seeing them emerge in parallel, being developed in different tickets under different namespaces and APIs, rather than fixing the API now, and than realizing that it needs to be amended. Once a construction clearly stands out, then we can thrash it upon the user as the default GF(p<sup>n</sup>) construction (ok, this ticket is already thrashing, but it has the merit of being the first one!).

pjbruin commented 11 years ago
comment:85

Replying to @defeo:

Replying to @pjbruin:

This discussion looks like the dear old dichotomy between quick feature integration and long specification design.

Not quite; I am not at all advocating long specification design, and quick integration of new features (which I am all for) is in fact easier if they are smaller and don't intrude in places where they don't have to.

Having some kind of support for lattices of finite fields has been a long standing request. I agree with pbruin that a better interface between generic finite fields and their actual implementation would be beneficial. But this ticket is ready for review, while pbruin's is not.

The part that is relevant for this ticket is now ready for review: see #14832 and #14833.

Would it be that hard to adapt pbruin's or any other interface if this ticket is merged? I'm willing to give positive review to this ticket, if it stands some more testing, and it doesn't mess too much with #12142.

I am actually in favour of quickly solving the main things that this ticket does (implementing pseudo-Conway polynomials and coercion between different finite fields). I just think it shouldn't add more code to the finite fields implementations (Givaro, PARI etc.), and should not (or at least not yet) fix a meaning for GF(p^n).

jpflori commented 11 years ago
comment:86

Ok, so I'll be the nice guy and try to rebase this ticket on top of your tickets.

jpflori commented 11 years ago

Work Issues: rebase

pjbruin commented 11 years ago

Changed keywords from days49 to days49 sd51

pjbruin commented 11 years ago
comment:87

Replying to @jpflori:

Ok, so I'll be the nice guy and try to rebase this ticket on top of your tickets.

Wonderful; these (#12142 and dependencies, maybe also #14888) should now be fairly stable.

jpflori commented 11 years ago
comment:88

I've begun rebasing, cleaning and splitting in a better way the patches here.

I have one question: in several finite field constructors based on a given implementation (let's say FiniteField_givaro), you can still pass the "modulus" parameter as a string and the routine corresponding to the given type of modulus is called. IMHO this kind of defeats what Peter tried to do in #14832 and #14833 (though it predates these patcehs of course). Any objection to change this behavior and instead more or less call the new irreducible_element function with the appropriate "algorithm" parameter?

jpflori commented 11 years ago
comment:89

(This will potentially introduce a slow down in some constructors, e.g. in finite_field_ntl_gf2e, but this can't really be helped.)

roed314 commented 11 years ago
comment:90

Replying to @jpflori:

I've begun rebasing, cleaning and splitting in a better way the patches here.

I have one question: in several finite field constructors based on a given implementation (let's say FiniteField_givaro), you can still pass the "modulus" parameter as a string and the routine corresponding to the given type of modulus is called. IMHO this kind of defeats what Peter tried to do in #14832 and #14833 (though it predates these patcehs of course). Any objection to change this behavior and instead more or less call the new irreducible_element function with the appropriate "algorithm" parameter?

No, I have no objection to a more unified way of generating the modulus.

jpflori commented 11 years ago
comment:91

Ok so let's go this way.

As I'll be cut from the internet next week while the workshop in Leiden is taking place and I'm not sure what I'll be able to achieve today, here are some hints on what I plan to do. Of course feel free to do something completely different and merge the ticket next week!

jpflori commented 11 years ago

Description changed:

--- 
+++ 
@@ -21,4 +21,4 @@

 1. [attachment: trac_8335-pseudo_conway-5.10.b3.patch](https://github.com/sagemath/sage-prod/files/10648223/trac_8335-pseudo_conway-5.10.b3.patch.gz)
 2. [attachment: trac_8335-finite_field_coerce-5.8.b0.patch](https://github.com/sagemath/sage-prod/files/10648220/trac_8335-finite_field_coerce-5.8.b0.patch.gz)
-3. [attachment: trac_8335-fixes-5.11.b1.patch](https://github.com/sagemath/sage-prod/files/10648225/trac_8335-fixes-5.11.b1.patch.gz)
+3. [attachment: trac_8335-fixes-5.11.b3.patch](https://github.com/sagemath/sage-prod/files/10648226/trac_8335-fixes-5.11.b3.patch.gz)
jpflori commented 11 years ago
comment:92

I found no time to work on this today, so I'll just post the very rough and incomplete patch I produced yesterday evening, not sure it is worth anything, but just in case you can apply it after applying attachment: trac_8335-pseudo_conway-5.10.b3.patch (note that hg will rant and you'll get three rejection files in sage/rings/finite_rings/, just push the new patch on top of that). Note it does not include anything from attachment: trac_8335-fixes-5.11.b3.patch yet.

jpflori commented 11 years ago

Attachment: trac_8335-pseudo_conway-fixes.patch.gz

pjbruin commented 11 years ago
comment:93

OK, thanks. To begin with we'll start by trying to understand better how these patches work.

pjbruin commented 11 years ago

Changed dependencies from #13894 to #13894, #14833

pjbruin commented 11 years ago
comment:95

There are a few small problems with applying and testing this set of patches; I am now combining them into one patch that can be applied on top of 5.11.beta3 + (dependencies of this ticket) and that we can use as a starting point during Sage Days 51.

pjbruin commented 11 years ago
comment:96

I'm going to split the following parts off into separate tickets, which will be dependencies of this one:

These will hopefully be relatively easy to review. We can then concentrate on implementing coercion between finite fields in this ticket.

pjbruin commented 11 years ago

Description changed:

--- 
+++ 
@@ -11,14 +11,4 @@

 This feature is implemented for fields outside the range of the Conway polynomial database by the implementation of a function for finding pseudo-Conway polynomials: polynomials that satisfy all of the algebraic constraints on Conway polynomials without the lexicographic constraint that imposes uniqueness.

-Finite fields no longer require an explicit variable name (though they still accept one).  If a variable name is given, then outside the range of the Conway polynomial database a random or sparse polynomial is used for speed reasons; if no variable name is given then either a Conway polynomial or pseudo-Conway polynomial is used.
-
-Also adds methods `any_root` and `squarefree_decomposition` to polynomials over finite fields.
-
-Depends on #8218, #8332, #7880, #7883, #8333, #8334.
-
-__APPLY__
-
-1. [attachment: trac_8335-pseudo_conway-5.10.b3.patch](https://github.com/sagemath/sage-prod/files/10648223/trac_8335-pseudo_conway-5.10.b3.patch.gz)
-2. [attachment: trac_8335-finite_field_coerce-5.8.b0.patch](https://github.com/sagemath/sage-prod/files/10648220/trac_8335-finite_field_coerce-5.8.b0.patch.gz)
-3. [attachment: trac_8335-fixes-5.11.b3.patch](https://github.com/sagemath/sage-prod/files/10648226/trac_8335-fixes-5.11.b3.patch.gz)
+For the time being, finite fields still require an explicit variable name (so the above code snippet won't work exactly as shown); in the future, a variant is planned that no longer requires a variable name.
pjbruin commented 11 years ago

Changed dependencies from #13894, #14833 to #14958

pjbruin commented 11 years ago
comment:98

Once #14957 and #14958 are stable enough, the next step will be to update attachment: trac_8335-finite_field_coerce-5.8.b0.patch and attachment: trac_8335-fixes-5.11.b3.patch​, according to Jean-Pierre's plan from #8335 comment:91.

pjbruin commented 11 years ago
comment:99

The patch attachment: trac_8335_sd51.patch contains the changes that remain after splitting off #14957 and #14958, and has been rebased on 5.11.beta3 + (dependencies of this ticket).

Patchbot: apply trac_8335_sd51.patch
pjbruin commented 11 years ago

to work on during Sage Days 51

pjbruin commented 11 years ago

Attachment: trac_8335_sd51.patch.gz

unified, rebased and cleaned up

pjbruin commented 11 years ago

Changed work issues from rebase to wait for #14958

pjbruin commented 11 years ago

Description changed:

--- 
+++ 
@@ -1,8 +1,8 @@
 Implements coercion within lattices of finite fields lying above the same prime when implemented with (pseudo-)Conway polynomials.

-sage: k = GF(9) -sage: l = GF(27) +sage: k = GF(9, conway=True, prefix='z') +sage: l = GF(27, conway=True, prefix='z') sage: x = k.gen() + l.gen(); x z6^5 + 2z6^4 + 2z6^3 + z6^2 + 2*z6 + 1 sage: x.parent() @@ -11,4 +11,9 @@

This feature is implemented for fields outside the range of the Conway polynomial database by the implementation of a function for finding pseudo-Conway polynomials: polynomials that satisfy all of the algebraic constraints on Conway polynomials without the lexicographic constraint that imposes uniqueness.

-For the time being, finite fields still require an explicit variable name (so the above code snippet won't work exactly as shown); in the future, a variant is planned that no longer requires a variable name. +When using the conway and prefix parameters, one does not need to specify an explicit variable name; if no variable name is given, it is constructed from the prefix and the degree (as in the above code snippet). + +In the future, the functionality of this ticket will be incorporated into that for algebraic closures of finite fields. + +Apply: attachment: trac_8335-finite_field_coerce-5.11.b3.patch +

pjbruin commented 11 years ago
comment:100

Attachment: trac_8335-finite_field_coerce-5.11.b3.patch.gz

Besides everything else, the latest patch moves _coerce_map_from_() from the various finite field implementations to the FiniteField base class; it is now implementation-independent. For this reason, this ticket now depends on #12142. Various other changes have been made.

The syntax for constructing finite fields using Conway polynomials that admit automatic coercion is now

sage: F.<a> = FiniteField(5^3, conway=True, prefix='z')

This is not too pretty, but it is meant as a temporary solution until we have algebraic closures of finite fields.

Older patches on this ticket contained various changes that were in older attachments and that do not seem immediately relevant to this ticket. I deleted those changes that seemed superfluous and kept those that I thought could be necessary after all.

This ticket should be reviewed once #14958 is done.

For patchbot:

apply trac_8335-finite_field_coerce-5.11.b3.patch
pjbruin commented 11 years ago

Changed dependencies from #14958 to #14958, #12142

jpflori commented 11 years ago

Changed work issues from wait for #14958 to none

jpflori commented 11 years ago

Changed author from David Roe, Jean-Pierre Flori to David Roe, Jean-Pierre Flori, Peter Bruin

jpflori commented 11 years ago

Rebased on top of #14888.

jpflori commented 11 years ago

Description changed:

--- 
+++ 
@@ -15,5 +15,5 @@

 In the future, the functionality of this ticket will be incorporated into that for algebraic closures of finite fields.

-Apply: [attachment: trac_8335-finite_field_coerce-5.11.b3.patch](https://github.com/sagemath/sage-prod/files/10648229/trac_8335-finite_field_coerce-5.11.b3.patch.gz)
+Apply: [attachment: trac_8335-finite_field_coerce-5.11.b3-14888.patch](https://github.com/sagemath/sage-prod/files/10648230/trac_8335-finite_field_coerce-5.11.b3-14888.patch.gz)
jpflori commented 11 years ago
comment:102

Attachment: trac_8335-finite_field_coerce-5.11.b3-14888.patch.gz

jpflori commented 11 years ago

Work Issues: rebase on top of #14958

jpflori commented 11 years ago
comment:103

Need to be rebased because of the name changes in #14958.

jpflori commented 11 years ago
comment:104

My bad. I thought you still cached the lattice in the finite field but you don't. (Un)fortunately when the lattice is built it is only weak-cached in conway_polynomials.py (so that it can be gc'ed when no finite field uses it anymore).

So either we have to store the lattice in the finite field to keep it alive and be able to use it when building extensions/pushouts and so on, or strongly cache it in conway_polynomials.py (I don't like that solution, in fact I hate caching things too much; note that currently everything is strongly cached anyway because of comment:69, #14711).

jpflori commented 11 years ago

Changed work issues from rebase on top of #14958 to rebase on top of #14958, store ref to PCL in finite field (or implement another way of weak caching)

jpflori commented 11 years ago

Attachment: trac_8335-rebase_14958.patch.gz

Name changes in #14958.

jpflori commented 11 years ago

Description changed:

--- 
+++ 
@@ -15,5 +15,7 @@

 In the future, the functionality of this ticket will be incorporated into that for algebraic closures of finite fields.

-Apply: [attachment: trac_8335-finite_field_coerce-5.11.b3-14888.patch](https://github.com/sagemath/sage-prod/files/10648230/trac_8335-finite_field_coerce-5.11.b3-14888.patch.gz)
+Apply:
+* [attachment: trac_8335-finite_field_coerce-5.11.b3-14888.patch](https://github.com/sagemath/sage-prod/files/10648230/trac_8335-finite_field_coerce-5.11.b3-14888.patch.gz)
+* [attachment: trac_8335-rebase_14958.patch](https://github.com/sagemath/sage-prod/files/10648231/trac_8335-rebase_14958.patch.gz)
jpflori commented 11 years ago

Changed work issues from rebase on top of #14958, store ref to PCL in finite field (or implement another way of weak caching) to store ref to PCL in finite field (or implement another way of weak caching)

jpflori commented 11 years ago
comment:106

In fact, the more I think about it, the less I like the way things are stored in the pseudo-Conway polynomial code.

We should make some big changes when implementing the AlgebraicClosure thing... Has anyone open a ticket for that?

pjbruin commented 11 years ago
comment:107

Replying to @jpflori:

We should make some big changes when implementing the AlgebraicClosure thing... Has anyone open a ticket for that?

Not yet, as far as I have been able to find out; I can do it now.

pjbruin commented 11 years ago
comment:108

Replying to @jpflori:

In fact, the more I think about it, the less I like the way things are stored in the pseudo-Conway polynomial code.

I dislike it too. It is problematic to store pseudo-Conway lattices globally in a Sage session (at least until they are garbage-collected) given that they are not uniquely defined.

It appears that the user could do the following:

I was also worried that storing the pseudo-Conway lattice in the finite field would mean that we would forever have to keep suitable pickling/unpickling code around to deal with this. Actually, this is not necessary, since the pseudo-Conway lattice can be reconstructed from the defining polynomial. However, this does not seem to solve the above problem. Using strong references does not solve it either. In both cases, the user may pickle k0, restart Sage, create k1, and finally unpickle k0; then again k0 and k1 will be different in general.

All of this is basically a manifestation of the fact that "the" field of pn elements is only defined up to non-canonical isomorphism.

jpflori commented 11 years ago
comment:109

I suggest we store a strong to the (top of the) lattice in the FF for the moment, as used to be done, and merge this ticket.

And let's think of a better design for #14990. Having AlgebraicClosure object will surely greatly simplify this.

pjbruin commented 11 years ago
comment:110

Replying to @jpflori:

I suggest we store a strong to the (top of the) lattice in the FF for the moment

We could do that to make garbage collection less likely, but it won't really solve the problem, see comment:108.

And let's think of a better design for #14990. Having AlgebraicClosure object will surely greatly simplify this.

Yes, the more I think about it, the more convinced I am that algebraic closures are the only real solution to the problem of compatible embeddings.

Jenny Cooley suggested the following idea, which I think is a good compromise: implement this ticket only for standard (not pseudo-) Conway polynomials. Hopefully this would suffice for most practical purposes, and since they are uniquely determined, we wouldn't have to come up with half-baked solutions to the caching problem. In #14990 (which I am working on now), pseudo-Conway polynomials can then finally be put to use, and they will be cached in the algebraic closure, where they really belong.

jpflori commented 11 years ago
comment:111

I'm ok with that.

I think I could accept anything as long as we close this ticket :)

pjbruin commented 11 years ago

use only (non-pseudo-)Conway polynomials