Improved use of category framework for IntegerModRing

[simon-king-jena]

On sage-devel, some discussion on IntegerModRing and its relation to categories took place. I would summarize the discussion and its consequences as follows:

• It should be possible for the user to indicate that the modulus of an IntegerModRing is prime, so that the ring is a field, without Sage performing a primality test. However (not part of the discussion) GF(n) does a primality test (actually it factors n---is this really needed?). So, the user can not simply use GF(n) if n is a huge number that the user knows to be a prime number.
• It was suggested that the user can instead do IntegerModRing(n, category=Fields()). This feature was asked for in #8562 and implemented in #9138.
• .is_field() should do a primality test, unless it is already known that the ring is a field. It has been suggested in the discussion on sage-devel to let .is_field() change the category of the ring. At the time of the discussion, this did not seem possible, but in #11900 this feature has been added.

By #11900, refinement of category happens when it is tested whether the IntegerModRing is a field by R in Fields(). However, there is some inconsistency:

sage: S = IntegerModRing(5)
sage: S.category()
Join of Category of commutative rings and Category of subquotients of monoids and Category of quotients of semigroups and Category of finite enumerated sets
sage: S in Fields()
True
sage: S.category()
Join of Category of fields and Category of subquotients of monoids and Category of quotients of semigroups and Category of finite enumerated sets
sage: R = IntegerModRing(5, Fields())
sage: R.category()
Join of Category of fields and Category of subquotients of monoids and Category of quotients of semigroups

I think we would want that R and S are in the same category after the category refinement of S.

So, the first aim of this ticket is to make the categories consistent.

Secondly, comparison of IntegerModRing includes comparison of types. Originally, this has been introduced since one wanted GF(p) and IntegerModRing(p) evaluate differently. However, changing the category changes the type, and thus changes the ==-equivalence class. I think this must not happen. Hence, the second aim of this ticket is to make == independent of the category, while still letting GF(p) and IntegerModRing(p, category=Fields()) evaluate unequal.

Thirdly, in the discussion on sage-devel, it was suggested to refine the category of R when R.is_field() returns true. Currently, refinement only happens when R in Fields() returns true. So, the third aim of this ticket to do the refinement as suggested.

Cc to John Cremona, since it was he who brought the "category refinement in is_field()" topic up...

CC: @JohnCremona @nthiery @jpflori @novoselt

Component: categories

Keywords: sd53 category integermodring

Author: Simon King

Branch/Commit: 048aec7

Reviewer: Jean-Pierre Flori

Issue created by migration from https://trac.sagemath.org/ticket/15229

[simon-king-jena]

comment:1

I just notice: We have IntegerModRing.is_field(self, proof=True), but the proof argument gets ignored! I think it should instead be passed to is_prime(), which also accepts a proof keyword. Hence, regardless whether proof=None, False or True, there will always the default arithmetic proof flag be used.

John, do you think it would make sense to set proof=None by default, and pass it to is_prime()? In this way, the "default arithmetic proof flag" would still be used by default, but it would be possible to override it.

[simon-king-jena]

comment:2

If one did

R = IntegerModRing(5, category=Fields())

then R.is_field() would still do a primality test.

Question: Shouldn't R.is_field() trust the user and first check whether R.category() is a subclass of Fields() before doing an expensive test?

The following seems reasonable to me:

def is_field(self, proof=None):
    if not proof:
        if self.category().is_subcategory(Fields()):
            return True
    is_prime = self.order().is_prime(proof=proof)
    if is_prime:
        self._refine_category_(Fields())
    return is_prime

[JohnCremona]

comment:3

THis looks a very sensible soution to me.

[simon-king-jena]

comment:4

Thank you, John. Concerning GF, I stand corrected: It only does a full factorisation after it has already been found that the modulus is a prime power.

[nbruin]

comment:5

Replying to @simon-king-jena:

Thank you, John. Concerning GF, I stand corrected: It only does a full factorisation after it has already been found that the modulus is a prime power.

That's probably OK because the factorization routine in question is quick in recognizing perfect powers and primes. Calling a generic factorization algorithm to find out what prime power we're dealing with is bad, of course.

[nbruin]

comment:6

As I point out on [#15223 comment:18], if we do (in a fresh sage session)

sage: R=IntegerModRing(19)
sage: C1=R.category()
sage: R in Fields()
sage: C2=R.category()

we have C1 is not C2. But if we run that same code again, we do find that C1 is C2. That's the normal result from R being a globally unique object and the conclusion one must draw is that R.category() is not an important attribute. Otherwise, the call R in Fields() would significantly modify global state, which means that a simple sanity check of input in a library routine somewhere could really foul up computations elsewhere!

So why would one care to specify the category upon construction if the thing doesn't matter? Worse, presently specifying category upon construction can cause multiple copies to exist. This implies that the category DOES matter, and hence should not be changed on global copies.

Another solution to forcing category refinement then would be

R.is_field(proof=True) #primality proof
R.is_field(proof=False) #only probable prime power check
R.is_field(proof=None) #no check at all

where the spelling of None is up for discussion. Perhaps it should be "I_know_I_can_screw_up_this_sage_session_with_this_never_mind_the_consequences_of_pickling_this".

So I think we can't have both automatic category refinement on global objects (indicating categories are not important for equality/identity) and category specification upon construction (which raises the expectation that they are and presently actually makes that the case).

The consequence of automatic category refinement is that you can't actually control the category that you get back even if you specify it: you might get back an answer in a finer category if that refinement happened to have been made before. Quoting from #15223 again:

sage: A1=IntegerModRing(19,category=Rings())
sage: A1.category()
Join of Category of commutative rings and Category of subquotients of monoids and Category of quotients of semigroups
sage: A1 in Fields()
True
sage: A2=IntegerModRing(19,category=Rings())
sage: A2.category()
Join of Category of subquotients of monoids and Category of quotients of semigroups and Category of fields

[nbruin]

comment:7

Concerning making non-identical parents equal, e.g., if we were to have

sage: R1=IntegerModRing(19)
sage: R2=IntegerModRing(19,distinguishing_flag)
sage: R1 == R2, R1 is R2
(True, False)

Then

sage: P1=PolynomialRing(R1,'x')
sage: P2=PolynomialRing(R2,'x')

wreaks havoc: Since the construction parameters of P1 and P2 are equal, we'll have P1 is P2, so the P2.base_ring() will actually be R1, not R2. This can lead to horrible surprises. I think we had some really convincing examples of that at some point, but I can't find the reference right now.

[simon-king-jena]

comment:8

Replying to @nbruin:

So why would one care to specify the category upon construction if the thing doesn't matter? Worse, presently specifying category upon construction can cause multiple copies to exist. This implies that the category DOES matter, and hence should not be changed on global copies.

That's related with one of the question I raised in #15223:

Would we like that there is precisely one instance of an integer mod ring for a given modulus, that is automatically updated if the user later provides more information?

Isn't GAP using a similar approach? It creates one object, and then its behaviour may change, when one learns more about it.

Hence, in this scenario, we would like that IntegerModRing (which is a factory) does not use the category argument for caching, but still passes it to the quotient ring.

I guess this is again a question to the number theorists (hence, John...): Would we like the following behaviour (not implemented yet)?

sage: R = IntegerModRing(5)
sage: R.category().is_subcategory(Fields())
False
sage: S = IntegerModRing(5, category=Fields())
sage: S is R    # currently False!
True
sage: R.category().is_subcategory(Fields())  # would currently only hold for S.category()
True

We would have a unique instance of IntegerModRing(5), and this unique instance would be refined, the more we learn about it, respectively the more the user tells us about it.

Another solution to forcing category refinement then would be

R.is_field(proof=True) #primality proof
R.is_field(proof=False) #only probable prime power check
R.is_field(proof=None) #no check at all

I wouldn't like that proof=False did more tests than proof=None.

[simon-king-jena]

comment:9

Replying to @nbruin:

Concerning making non-identical parents equal, e.g., if we were to have

sage: R1=IntegerModRing(19)
sage: R2=IntegerModRing(19,distinguishing_flag)
sage: R1 == R2, R1 is R2
(True, False)

Then ... this can lead to horrible surprises.

You are right. This would be no good.

[simon-king-jena]

comment:10

Replying to @nbruin:

So I think we can't have both automatic category refinement on global objects (indicating categories are not important for equality/identity) and category specification upon construction (which raises the expectation that they are and presently actually makes that the case).

We can! Namely, if the additional argument is ignored for the cache, but is still used to refine the existing object.

The consequence of automatic category refinement is that you can't actually control the category that you get back even if you specify it: ...

If we decide to ignore the additional argument for the cache and only use it to refine the unique global object, then this is a desired consequence. But phrased like "you can't actually make Sage forget what it found out about the category of the quotient ring".

[simon-king-jena]

comment:11

Replying to @simon-king-jena:

We would have a unique instance of IntegerModRing(5), and this unique instance would be refined, the more we learn about it, respectively the more the user tells us about it.

Perhaps it would make sense then to introduce a method Parent._unset_category(self), that would change the class of self to self.__class__.__base__ (I tested: This would actually be possible) and set the cdef ._category attribute to None?

Using such method, it would be possible to revert the changes, if the user was wrong in assuming that the modulus was prime. For example:

sage: R = IntegerModRing(15, category=Fields()) # 15 is odd, and hence it is prime
sage: R in Fields()
True
sage: R.is_field()
True
sage: R.is_field(proof=True)    # Ooops, there are odd numbers that aren't prime
False
sage: R._unset_category()
sage: R._init_category_(<put appropriate category here>)

And then, R would be in the same state as with originally defining R = IntegerModRing(15).

Perhaps erasing of wrong assumptions on primality can automatically be done in R.is_field(proof=True)?

[simon-king-jena]

Branch: u/SimonKing/ticket/15229

[simon-king-jena]

comment:13

I have pushed a preliminary version of the branch.

It addresses the three points from the ticket description. But, as Nils has reminded me, we should not use the additional argument for the cache key (because of polynomial rings over integer mod rings). This, and perhaps an automatic update of a globally unique version of ZZ/nZZ, may be in the next commit...

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:1d2e2bc]

Document the "proof" argument of is_field.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Commit: 1d2e2bc

[nbruin]

comment:15

Replying to @simon-king-jena:

Sure, having a way to do R.is_field(proof="believe_me") upon construction is fine. But is specifying category=Fields() the most economic/obvious interface for doing so? What other meaningful values are there? Wouldn't it be better to do

IntegerModRing(19,is_field=True)

instead? If the only meaningful value is Fields(), it seems to me that allowing the caller to specify any category there is just handing him extra rope for hanging only.

Concerning GAP's approach: I'm not familiar, but allowing the behaviour of global objects to significantly change sounds like a bad idea. It makes it really hard to argue about any piece of code if you have to take into account the whole history of the system all the time.

As far as I understand, presently, category refinement is simply used as a "cache" of computed data, but that means one shouldn't attach much further value to it either. Note how easily it can change:

sage: R=IntegerModRing(19)
sage: R.category()
Join of Category of commutative rings and Category of subquotients of monoids and Category of quotients of semigroups and Category of finite enumerated sets
sage: P=R['x']
sage: (P.0^2-1).roots()
[(18, 1), (1, 1)]
sage: R.category()
Join of Category of subquotients of monoids and Category of quotients of semigroups and Category of finite enumerated sets and Category of fields

[nbruin]

Changed commit from 1d2e2bc to none

[nbruin]

Changed branch from u/SimonKing/ticket/15229 to none

[nbruin]

comment:16

Branch u/SimonKing/ticket/15229 deleted Commit 1d2e2bcf623690bdaf81b63b5e200b1ea0feb4bb deleted

Ouch, that doesn't look good. I don't know why trac did that. I definitely didn't selected those things to happen.

[simon-king-jena]

Branch: u/SimonKing/ticket/15229

[simon-king-jena]

comment:17

My "category eraser" works. With it, one can do the following:

        Let us create a parent in the category of rings::

            sage: class MyParent(Parent):
            ....:     def __init__(self):
            ....:         Parent.__init__(self, category=Rings())
            ....:
            sage: P = MyParent()
            sage: P.category()
            Category of rings

        Of course, its category is initialised::

            sage: P._is_category_initialized()
            True

        We may now refine the category to the category to the
        category of fields. Note that this changes the class::

            sage: C = type(P)
            sage: C == MyParent
            False
            sage: P._refine_category_(Fields())
            sage: P.category()
            Category of fields
            sage: C == type(P)
            False

        Now we notice that the category refinement was a mistake.
        Hence, we undo category initialisation totally::

            sage: P._unset_category()
            sage: P._is_category_initialized()
            False
            sage: type(P) == MyParent
            True

        And finally, we initialise the parent again in the original
        category, i.e., the category of rings, finding that the
        class of the parent is brought back to what it was after
        the original category initialisation::

            sage: P._init_category_(Rings())
            sage: type(P) == C
            True

Question (e.g. to Nicolas):

Do we want to have such a category eraser? Then, refining the category of a quotient ring would not hurt so much, because it could easily be reverted when one finds that the refinement was a mistake.

[simon-king-jena]

Commit: 1d2e2bc

[simon-king-jena]

comment:18

Replying to @nbruin:

Branch u/SimonKing/ticket/15229 deleted Commit 1d2e2bcf623690bdaf81b63b5e200b1ea0feb4bb deleted

Ouch, that doesn't look good. I don't know why trac did that. I definitely didn't selected those things to happen.

Dunno. Anyway, trac chose to revert the deletion.

Hmm. It is ironical that trac reverted a mistake exactly when I suggested a new method to revert a mistaken category refinement... :-D

[nbruin]

comment:19

I don't know about playing around with categories too much that way. Could we have something like the following:

sage: R = SomeRing()
sage: F = FieldOfFractions(R)
ValueError: R is not an integral domain
sage: R._refine_category_(IntegralDomains())
sage: F = FieldOfFractions(R)
sage: R._unset_category()
sage: R._init_category_(Rings())
sage: R in IntegralDomains()
False
sage: F
Field of fractions of R

That would seem bad to me. Of course, as a debugging tool it may be OK, but I don't thing unrefining a category should be an advertised/supported feature. There may be objects around that rely on the category of the object as it was (and it may well be too expensive to ensure the category gets re-refined any time it relies on it)

[simon-king-jena]

comment:20

Replying to @nbruin:

I don't know about playing around with categories too much that way. Could we have something like the following:

Yes, of course.

All these _refine_category_ and _unset_category methods start with an underscore for a reason: You should only use them if you know what you are doing. And similarly, with the category=Fields() option of IntegerModRing, you should be ready to bear the consequences of any misuse.

That would seem bad to me. Of course, as a debugging tool it may be OK, but I don't thing unrefining a category should be an advertised/supported feature.

You can see it in this way: If you are in the fourth line of your example, then you are in a total mess. Unsetting the category might help you to clean some mess after the mess was created, but it wouldn't be appropriate to expect that it can clean everything.

However, being able to clean some mess is still better than nothing.

[simon-king-jena]

comment:21

And, by the way, we have:

sage: R = IntegerModRing(15, category=Fields())
sage: Frac(R)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-2-d5df03326cb5> in <module>()
----> 1 Frac(R)

/home/king/Sage/git/sage/local/lib/python2.7/site-packages/sage/rings/fraction_field.pyc in FractionField(R, names)
    126         raise TypeError, "R must be a ring"
    127     if not R.is_integral_domain():
--> 128         raise TypeError, "R must be an integral domain."
    129     return R.fraction_field()
    130 

TypeError: R must be an integral domain.

So, this might actually be an argument of using the R.is_bla() methods, rather than relying on R in CategoryOfBla().

[nbruin]

comment:22

Replying to @simon-king-jena:

So, this might actually be an argument of using the R.is_bla() methods, rather than relying on R in CategoryOfBla().

It might, but in my view an invalid one: I think we established that R in ... was faster (once initialized) because we end up co-opting python's highly efficient mro implementation. So I think it's more a argument to ensure the category stays sane.

Coming back to the category= keyword. The documentation would be something along the lines of

   category = C : Ensure that the object returned lies in a refinement
       of C. Note that the returned object may lie in a strict
       refinement. Additionally, note that, due to the fact that the
       object returned is globally unique, the object returned 
       may be identical to the result of independent constructions.
       These would naturally now also have a refined category.

By the time the doc looks like that, do you still think that's the best API to allow specifying that a quotient ring is a field upon construction?

Do you have a rationale to expose something that complex in the construction interface if the same can be accomplished using _refine_category_ afterwards?

[simon-king-jena]

comment:23

Replying to @nbruin:

Do you have a rationale to expose something that complex in the construction interface if the same can be accomplished using _refine_category_ afterwards?

Good point. If one has a potentially dangerous functionality, then it makes sense to hide it from the occasional user by making it underscore---and it may be a bad idea to make this "forbidden magic" available in the constructor.

To my rehabilitation: I introduced the category keyword because it was formulated as "todo" in the docs, and it was formulated as "todo" because of the mentioned discussion on sage-devel.

The question to the number theorists is: Is the category keyword actually used somewhere? If it is, then one might replace it by _refine_category_, so that the category keyword can be removed.

And the question to all of us is:

Testing R in Fields() may rely on a probabilistic primality test. Thus, it might be that R in Fields() and R.is_field() give the wrong answer and hence the category is automatically refined incorrectly. Do we want that R.is_field(proof=True) reverts the category refinement, if it finds that R is in fact not a field?

[simon-king-jena]

comment:24

A quick "grep" told me that the category argument is in fact not used. But I think it would make sense to make a poll on sage-devel or sage-algebra or sage-nt before removing it.

[nbruin]

comment:25

Replying to @simon-king-jena:

Testing R in Fields() may rely on a probabilistic primality test. Thus, it might be that R in Fields() and R.is_field() give the wrong answer and hence the category is automatically refined incorrectly. Do we want that R.is_field(proof=True) reverts the category refinement, if it finds that R is in fact not a field?

If we use something like Baillie-PSW it shouldn't just revert the category, it should also file for $620 to be donated to the Sage foundation. Probabilistic primality tests are really good.

[simon-king-jena]

comment:26

I guess at some point we should post a question/poll on either sage-devel or sage-nt. But before doing so, we should discuss here what exactly to ask. Here, I try to give arguments for several approaches to solve our problem.

It was requested in the old sage-devel discussion that the user should be able to indicate that a certain modulus is prime, so that the integer mod ring is mathematically a field. The question is mainly: How should be indicated that it is a field, and what consequences should providing this information imply?

How?

It could be by an optional argument, but what exactly should be provided?

• "category=...": Most parent's constructors now have an optional category argument. So, it seems consistent with other parts of Sage to offer the same argument to integer mod rings.

• "is_field=False/True": All what we requested was a way to indicate whether the ring is a field or not. So, we just need a bool, and there is no need to allow an arbitrary fully fledged category in the constructor. On the other hand, there might be future applications which make use of categories beyond "is field/is not field".

Alternatively, one could indicate the "field-ness" only after initialisation. Hence:

• There should be no optional argument to IntegerModRing. Instead, the user has the possibility to do R._refine_category_(Fields()) after initialisation, perhaps wrapped into a method .use_as_field().

Consequences

Mathematically, there is only one quotient ring ZZ/nZZ. But what shall happen computationally?

One could argue that it might make sense to computationally have two versions of the same ring, namely one version that knows it is a field, and another one that does not know it. Or perhaps we find constructions which equip certain quotient rings with other fancy structures (say, Hopf algebra or so). Then it might make sense to have a third version of the same ring, knowing that it is a Hopf algebra.

Or one could argue that the ring either is a field or it is not, and it either has a Hopf algebra structure or it has not. From this point of view, there should be a unique instance of this ring, and additional structures (field, Hopf algebras) are all accumulated in this single instance.

So, it is not clear whether an optional argument provided to the IntegerModRing factory should be ignored in the cache or not.

A potential argument against accumulating information in a globally unique instance is: Globally unique instances should be immutable. But what exactly does "immutable" mean? It means: It will not change its hash and its equivalence class with respect to cmp (because this is what is used to construct a set or dictionary).

Thus, in any case, we have a bug to fix, because _refine_category_ changes the class of the instance, and the class is currently having an impact in __cmp__.

Safety

As you have pointed out, providing the wrong category on purpose may result in a serious mess. You argued that this means we must not allow full freedom in choosing the category. But I think we should acknowledge that Python has a rather permissive policy: We must not be over-protective.

So, it might be good to have an "eraser" such as _unset_category(), but of course such an eraser will impossibly be able to fix all bad consequences of choosing a wrong category.

Proposal

These are my personal preferences; in some cases my preference is actually not clear...

• Keep the optional argument "category", to be consistent with the rest of Sage.
• Use the category to speed-up is_field(), unless "proof=True` is requested.
• Keep refining the category if R.is_field() or R in Fields() succeeds with some primality check.
• As you said, a wrong primality test would be a great discovery. And bad consequences of a wrong primality test are quite likely to be not easily fixable. Hence, I think it would not make sense to automatically fix the category, if R.is_field(proof=True) finds that R is not a field.
• Fix: Since the category can be refined, but the object must be immutable, __cmp__ must be invariant under a change of category. This would also be consistent with the current bidirectional coercion maps between integer mod rings.
• I am undecided whether there should be a globally unique quotient ring for a given modulus, accumulating information, or one should keep having different instances evaluating equal.

[simon-king-jena]

Changed keywords from none to sd53 category integermodring

[simon-king-jena]

comment:27

For what it's worth, I have created #15234, introducing the category eraser.

I chose to open a different ticket, because I think it makes sense to have the eraser indepently of the problem discussed here, and since it is not clear from the discussion whether we actually want to apply the eraser here.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 1d2e2bc to 75576e2

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:75576e2]

Warn if an integer mod ring is mistakenly found in Fields()

[simon-king-jena]

comment:29

Replying to @simon-king-jena:

Proposal

These are my personal preferences; in some cases my preference is actually not clear...

• Keep the optional argument "category", to be consistent with the rest of Sage.
• Use the category to speed-up is_field(), unless "proof=True` is requested.
• Keep refining the category if R.is_field() or R in Fields() succeeds with some primality check.
• As you said, a wrong primality test would be a great discovery. And bad consequences of a wrong primality test are quite likely to be not easily fixable. Hence, I think it would not make sense to automatically fix the category, if R.is_field(proof=True) finds that R is not a field.
• Fix: Since the category can be refined, but the object must be immutable, __cmp__ must be invariant under a change of category. This would also be consistent with the current bidirectional coercion maps between integer mod rings.

All the above is achieved with the current commit and its ancestors. A warning is printed, that also asks to inform the developers, in the case that a probabilistic primality test has failed.

• I am undecided whether there should be a globally unique quotient ring for a given modulus, accumulating information, or one should keep having different instances evaluating equal.

What do y'all think?

[simon-king-jena]

comment:30

Replying to @simon-king-jena:

• I am undecided whether there should be a globally unique quotient ring for a given modulus, accumulating information, or one should keep having different instances evaluating equal.

What do y'all think?

I made a poll here at SD53: Would you think that there should be a unique instance of ZZ/nZZ that accumulates information, or should one have one instance that was told that it is a field (perhaps erroneously) and another instance that does not know about it?

It seems that all but two people said that one should have a unique instance, whose category is updated when one learns more on it or when the user provides more information on it.

I am not totally sure, but perhaps Jean-Pierre can chime in: What has been the argument of having distinct instances?

I also talked with Andrey, and he said (in my own summary): If the user explicitly provides certain wrong data on a ring (such as IntegerModRing(15, category=Fields())) and then later wants to construct the same ring without the wrong data, then it would not make sense to return a new instance of the ring. Namely, after IntegerModRing(15, category=Fields()) is done, the whole Sage session is compromised, and it would not help to have a second non-broken ring. Starting a new Sage session is the only reasonable thing to do in such situation.

So, I will now attempt to modify the IntegerModRing factory such that the additional category argument will be ignored in the cache, but used to refine the category of what is found in the cache. In other words, I try to make it so that there is a unique instance of ZZ/nZZ.

[nbruin]

comment:31

Replying to @simon-king-jena:

It was requested in the old sage-devel discussion that the user should be able to indicate that a certain modulus is prime, so that the integer mod ring is mathematically a field. The question is mainly: How should be indicated that it is a field, and what consequences should providing this information imply?

• "is_field=False/True": All what we requested was a way to indicate whether the ring is known to be a field or not.

(note correction in bold). For reasons I explain below, I'm not so sure using category=... is actually consistent with other uses in sage. Most category settings/changes should and ''do' affect equality and should result in distinct objects. This is not the case for the "is_field" case.

Alternatively, one could indicate the "field-ness" only after initialisation.

I suspect many people would consider that bothersome. It does better convey it is a mutation of something that (possibly) already exists, though, making it clearer that it may modify global state.

One could argue that it might make sense to computationally have two versions of the same ring, namely one version that knows it is a field, and another one that does not know it. Or perhaps we find constructions which equip certain quotient rings with other fancy structures (say, Hopf algebra or so).

Different situation! There's extra data (or perhaps you forget some data) so, for one thing, the homomorphisms on the object in that category are different.

Or one could argue that the ring either is a field or it is not, and it either has a Hopf algebra structure or it has not.

Being a field is indeed a property of a ring. To make a ring into a Hopf algebra you have to specify extra information. Even if there may be only one way to specify that extra information, this usually affects the category.

Thus, in any case, we have a bug to fix, because _refine_category_ changes the class of the instance, and the class is currently having an impact in __cmp__.

+1

So, it might be good to have an "eraser" such as _unset_category(), but of course such an eraser will impossibly be able to fix all bad consequences of choosing a wrong category.

I don't mind if it's there and it might be useful for debugging, but it won't actually help recover. It's just too hard to see what the ramifications of even a temporary lapse in consistency in the category settings does.

Proposal

These are my personal preferences; in some cases my preference is actually not clear...

• Keep the optional argument "category", to be consistent with the rest of Sage.

Is it consistent? How does category get used in other cases? Do we have uses analogous to IntegersMod(17,category=AbelianGroups())? In that cases the specified category IS important.

The relation of Fields, IntegralDomains etc. to Rings is a bit special: The concept of homomorphism doesn't change, which is why the category refinement can happen automatically. So these are very limited category changes that are indeed "insignificant". Most other category changes are significant (because they arise from applying some forgetful functor or fixing some extra (arbitrary?) data.

If other uses of category are or a different type, we should definitely not try to be consistent with it for special is_field case.

Of course, we do want to be consistent with other ring quotient constructors, where testing primality or maximality of the ideal we mod out by might be even more involved.

• Use the category to speed-up is_field(), unless "proof=True` is requested.

+1

• Keep refining the category if R.is_field() or R in Fields() succeeds with some primality check.

+1

• As you said, a wrong primality test would be a great discovery. And bad consequences of a wrong primality test are quite likely to be not easily fixable. Hence, I think it would not make sense to automatically fix the category, if R.is_field(proof=True) finds that R is not a field.

Given that at this point we've found that the global state of sage is really messed up, we should probably raise an error. The state is whatever it is at that point ...

• Fix: Since the category can be refined, but the object must be immutable, __cmp__ must be invariant under a change of category. This would also be consistent with the current bidirectional coercion maps between integer mod rings.

+1 with caveat: What are the possible values of category we want to consider here?

• I am undecided whether there should be a globally unique quotient ring for a given modulus, accumulating information, or one should keep having different instances evaluating equal.

We cannot do the latter if we want to be able to construct UniqueRepresentation parents with coefficients in these non-unique-but-equal bases: constructing a polynomial ring over one might return a polynomial ring over the other, which violates all kinds of assumptions in the coercion framework.

See this hypothetical example (yay, I found the reference).

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:cdc6806]

Replace optional "category" argument by "is_field"

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 75576e2 to cdc6806

[simon-king-jena]

comment:33

In the new commit I just pushed, I drop the "category" argument and replace it by "is_field". Note that old pickles using the category argument should still be unpicklable, but perhaps someone can test.

Moreover, I made ZZ/nZZ globally unique. Rationale: Being a field is a property that a ring either has or has not. Hence, it makes no sense to have different instances, one knowing that it is a field and the other not knowing. The unique instance is updated when one learns more on it, one way (primality test) or the other (user info).

Replying to @nbruin:

Replying to @simon-king-jena: For reasons I explain below, I'm not so sure using category=... is actually consistent with other uses in sage. Most category settings/changes should and ''do' affect equality and should result in distinct objects. This is not the case for the "is_field" case.

You say somewhere below that my example of a Hopf algebra structure is not good, because one could not say that this structure is a property of the ring, since it is an additional structure. Indeed this convinces me, and in this case, one should create a sub-class of IntegerModRing_generic.

Note that IntegerModRing is just a factory. Hence, while IntegerModRing_generic.__init__ should of course have a "category" argument (to be consistent with all base classes!), the factory does not need "category".

Alternatively, one could indicate the "field-ness" only after initialisation.

I suspect many people would consider that bothersome. It does better convey it is a mutation of something that (possibly) already exists, though, making it clearer that it may modify global state.

In the current branch, doing IntegerModRing(p, is_field=True) will put the result into the category of fields, either taking an existing non-field version of the ring from the cache, or creating a new instance.

Thus, in any case, we have a bug to fix, because _refine_category_ changes the class of the instance, and the class is currently having an impact in __cmp__.

+1

For the record, that's in some previous commit already.

So, it might be good to have an "eraser" such as _unset_category(), but of course such an eraser will impossibly be able to fix all bad consequences of choosing a wrong category.

I don't mind if it's there and it might be useful for debugging, but it won't actually help recover. It's just too hard to see what the ramifications of even a temporary lapse in consistency in the category settings does.

See #15234

The relation of Fields, IntegralDomains etc. to Rings is a bit special: The concept of homomorphism doesn't change, which is why the category refinement can happen automatically. So these are very limited category changes that are indeed "insignificant". Most other category changes are significant (because they arise from applying some forgetful functor or fixing some extra (arbitrary?) data.

If other uses of category are or a different type, we should definitely not try to be consistent with it for special is_field case.

Meanwhile, I made it "is_field"...

• Use the category to speed-up is_field(), unless "proof=True` is requested.

+1

done

• Keep refining the category if R.is_field() or R in Fields() succeeds with some primality check.

+1

done

• As you said, a wrong primality test would be a great discovery. And bad consequences of a wrong primality test are quite likely to be not easily fixable. Hence, I think it would not make sense to automatically fix the category, if R.is_field(proof=True) finds that R is not a field.

Given that at this point we've found that the global state of sage is really messed up, we should probably raise an error. The state is whatever it is at that point ...

OK. Currently it just prints a warning, but raising a proper error would of course be an option. You think I should do it in the next commit?

• Fix: Since the category can be refined, but the object must be immutable, __cmp__ must be invariant under a change of category. This would also be consistent with the current bidirectional coercion maps between integer mod rings.

+1 with caveat: What are the possible values of category we want to consider here?

Well, meanwhile there are only two possible values. Something like "join of Category of commutative rings, finite enumerated sets and (sub?)quotients of rings", and the same thing with "fields" instead of commutative rings.

The current __cmp__ removes the category information totally, which would of course be a mistake for sub-classes that are more than just "field" or "not field" (as in my Hopf algebra example). Hence, __cmp__ should be overridden on the subclasses.

• I am undecided whether there should be a globally unique quotient ring for a given modulus, accumulating information, or one should keep having different instances evaluating equal.

We cannot do the latter if we want to be able to construct UniqueRepresentation parents with coefficients in these non-unique-but-equal bases: constructing a polynomial ring over one might return a polynomial ring over the other, which violates all kinds of assumptions in the coercion framework.

OK. I prefer uniqueness anyway. With the current commit, we have uniqueness.

[simon-king-jena]

comment:34

Pickling Problem/Question:

Suppose that R is an integer mod ring, originally created not in the category of fields. Suppose that later we find out that it is in fact a field. Suppose that even later we save R to a file, and load this file into a new Sage session.

Do we want that the reloaded version of R is immediately initialised as a field?

With the current commit, the following would happen.

sage: R = IntegerModRing(7)
sage: R.category().is_subcategory(Fields())
False
sage: R.is_field()
True
sage: R.category().is_subcategory(Fields())
True
sage: save(R, 'tmp')

Quit sage and start a new session

sage: R = load('tmp.sobj')
sage: R.category().is_subcategory(Fields())
False
sage: R.is_field.cache
{}

So, the "field" information, that might have been obtained by a quite expensive primality test, has been completely lost.

Do we want that pickles remember the field-ness?

[novoselt]

comment:35

I think pickles should remember everything, unless there are insurmountable technical difficulties to it! Especially if this ring was told by the user that it is a field to avoid doing any tests.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

[changeset:6ada7e0]

Raise an error if a non-prime integer mod ring is found in Fields()

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from cdc6806 to 6ada7e0

[simon-king-jena]

comment:37

Pickling is not addressed yet in the current commit, but now there is a BIG FAT error if a primality test fails on something that was supposed to be a field.

[simon-king-jena]

comment:38

It would be possible to make pickles store all cached information about the integer mod ring. Namely, we just need to append self.__dict__ to the current return value of self.__reduce__(). Do we want this?

[novoselt]

comment:39

I think reasons against pickling something are:

• it takes too much space
• it takes too much time to pickle
• it takes too much time to unpickle

In most cases none of these applies, I suspect, so I don't think what is the problem with pickling everything by default. Having a convenient way to manually turn off pickling in the above cases would be nice, of course, as well as having a generic framework to deal automatically with circular references from cache and uniqueness (which is currently a problem with toric varieties).

[simon-king-jena]

comment:40

The pickling thing is not so easy.

First of all, __setstate__ is inherited from ParentWithGens, that expects the dictionary to contain certain data that make no sense to pickle. It could mean that we have to override __setstate__.

Secondly, what happens if we have created a ring R in a sage session, and R.is_field already has cached values. If we then load a ring with the same modulus from a file, then (by uniqueness) R is returned---but R.__dict__ is updated with what we have found on the disc: The cached values would still be available to R.is_field, but the values would not be part of R.__dict__ any longer. Hence, if we then save R again to a new file, the new pickle would not contain the cached values.

Would it be worth it? It doesn't seem so.

But one thing would certainly be possible: If the category is updated to Fields(), either by providing the optional argument to the factory or by what is now happening inside R.is_field(), then the data used for pickling could be updated. Hence, when we now load R from a file, it would immediately be in Fields(), even though the cached values of is_field() will be lost.

Would this be an acceptable compromise?

[novoselt]

comment:41

Certainly fine with me! Someyear, hopefully, ParentWithGens will be gone and perhaps things will work smoother, but for now it can count as "insurmountable technical difficulties". I don't insist on pickling anything, just think that in general it should work.