Implement indexed monoids

[tscrim]

Implements free (abelian) monoids whose generators are indexed by an arbitrary set. This also moves common code from CombinatorialFreeModule into a new class sage.misc.indexed_generators.IndexedGenerators. With this we can now create (noncommutative) polynomials whose generators are given by a combinatorial object.

Also implements a very crude and basic version for groups.

Depends on #15309 Depends on #15169 Depends on #16349

CC: @sagetrac-sage-combinat @nthiery @sagetrac-mshimo @simon-king-jena

Component: algebra

Keywords: days54

Author: Travis Scrimshaw

Branch: 69ec7b2

Reviewer: Nicolas M. Thiéry

Issue created by migration from https://trac.sagemath.org/ticket/15289

[tscrim]

comment:36

Fixed everything mentioned above except I've run into a major pickling issue with UniqueFactory (which is why I've cc-ed Simon).

There is some precedence for creating a generator from an index:

sage: C = CombinatorialFree
Module(ZZ, ['a','b','c'])
sage: C('a')
B['a']

So I'd like to leave that in for now. However I made things like F(1) only return F[1].

I changed the input for Monoids().free(...) but I left FreeMonoid(...) as I had it before because I don't want to deal with deprecations there in this ticket (this is #16330). I'll do the cleanup of the factory functions in the followup #16329.

I'm not doing anything to FreeMonoid or FreeAbelianMonoid other than redirecting input, so I'm not going to make these changes here:

• Now or later: __repr__ -> _repr_, __call__ -> _element_constructor_. The custom __contains__ might not be needed.
• Now or later: initialize the category

We could probably also fit that in #16329 or another followup.

Simon, so I've changed the UniqueFactory object FreeMonoid to a new factory function since it returns a class which I've designed as a UniqueRepresentation. However unpickling old FreeMonoid objects fails because of the unpickling mechanism of UniqueFactory redirects to generic_factory_unpickle() and has FreeMonoid from the global namespace as the first object. It is expecting it to be a UniqueFactory instance, so it errors out here. Furthermore because of the structure of the pickle, I'm unable to use register_unpickle_override or see anyway to redirect that specific pickle without changing the behavior of generic_factory_unpickle().

I'm thinking the best thing to do is to change generic_factory_unpickle() to accept a general object as input and if the first input is no longer a UniqueFactory, then call it with the arguments given as part of the pickle. Another option I see is to override UniqueFactory.__call__, but I really don't want to do that. The last option I see, and I absolutely don't like this, is to make my new factory function into a different name temporarily (or deprecate FreeMonoid construction altogether). Any ideas?

I'm also thinking we should change this behavior so we don't keep running into this problem.

[nthiery]

comment:37

Replying to @tscrim:

Fixed everything mentioned above except I've run into a major pickling issue with UniqueFactory (which is why I've cc-ed Simon).

Cool, thanks!

There is some precedence for creating a generator from an index:

sage: C = CombinatorialFree
Module(ZZ, ['a','b','c'])
sage: C('a')
B['a']

So I'd like to leave that in for now.

Precisely: we have made that mistake once, and been bitten by it. So I'd rather not redo it!

I changed the input for Monoids().free(...) but I left FreeMonoid(...) as I had it before because I don't want to deal with deprecations there in this ticket (this is #16330). I'll do the cleanup of the factory functions in the followup #16329.

Fair enough.

I'm not doing anything to FreeMonoid or FreeAbelianMonoid other than redirecting input, so I'm not going to make these changes here:

• Now or later: __repr__ -> _repr_, __call__ -> _element_constructor_. The custom __contains__ might not be needed.
• Now or later: initialize the category

We could probably also fit that in #16329 or another followup.

Ok.

Simon, so I've changed the UniqueFactory object FreeMonoid to a new factory function since it returns a class which I've designed as a UniqueRepresentation. However unpickling old FreeMonoid objects fails because of the unpickling mechanism of UniqueFactory redirects to generic_factory_unpickle() and has FreeMonoid from the global namespace as the first object. It is expecting it to be a UniqueFactory instance, so it errors out here. Furthermore because of the structure of the pickle, I'm unable to use register_unpickle_override or see anyway to redirect that specific pickle without changing the behavior of generic_factory_unpickle().

I'm thinking the best thing to do is to change generic_factory_unpickle() to accept a general object as input and if the first input is no longer a UniqueFactory, then call it with the arguments given as part of the pickle. Another option I see is to override UniqueFactory.__call__, but I really don't want to do that. The last option I see, and I absolutely don't like this, is to make my new factory function into a different name temporarily (or deprecate FreeMonoid construction altogether). Any ideas?

I'm also thinking we should change this behavior so we don't keep running into this problem.

Maybe, at least as a temporary solution, the FreeMonoid object from the global namespace could remain a trivial factory function delegating to a class, also called FreeMonoid, but in its own module?

Cheers, Nicolas

[tscrim]

comment:38

The FreeMonoid in the global namespace must be the UniqueFactory instance for the old pickles to work, irregardless of which module or the name of the class. That's the big problem. Or are you suggesting to just leave FreeMonoid alone (or perhaps give it some warnings and redirections when given other types of input)?

There is one other thing I thought of today, to implement a custom unpickle override for UniqueFactory. I'm thinking this might be the best way to work around this (and could easily create this on a separate ticket). Thoughts?

[nthiery]

comment:39

Replying to @tscrim:

The FreeMonoid in the global namespace must be the UniqueFactory instance for the old pickles to work,

Yes. But this UniqueFactory instance can probably be a fake one and just act as a delegator to whichever new class now implements FreeMonoid.

There is one other thing I thought of today, to implement a custom unpickle override for UniqueFactory. I'm thinking this might be the best way to work around this (and could easily create this on a separate ticket). Thoughts?

I am not sure I see what you mean, but I can live with that if someone else answers :-)

[tscrim]

comment:40

Replying to @nthiery:

Replying to @tscrim:

The FreeMonoid in the global namespace must be the UniqueFactory instance for the old pickles to work,

Yes. But this UniqueFactory instance can probably be a fake one and just act as a delegator to whichever new class now implements FreeMonoid.

So do you mean doing something like this:

class DummyFactory(UniqueFactory):
    def __call__(self, *args, **kwds):
        return factory_function(*args, **kwds)
    def get_object(self, the, ..., args):
        return OriginalFactoryInstance.get_object(the, ..., args)

and DummyFactory would become the object in the global namespace?

[nthiery]

comment:41

Yeah. Not sure it's a good idea though. Do as you feel is right.

[simon-king-jena]

Changed dependencies from #15309 #15169 to #15309, #15169

[simon-king-jena]

comment:43

In order to checkout this ticket, I had to fix the list of dependencies (which is supposed to be comma-separated).

[simon-king-jena]

comment:44

Replying to @tscrim:

Simon, so I've changed the UniqueFactory object FreeMonoid to a new factory function since it returns a class which I've designed as a UniqueRepresentation. However unpickling old FreeMonoid objects fails because of the unpickling mechanism of UniqueFactory redirects to generic_factory_unpickle() and has FreeMonoid from the global namespace as the first object.

First thing I wonder: Why is that statically typed? Just to shave off a little calling overhead?

It is expecting it to be a UniqueFactory instance, so it errors out here. Furthermore because of the structure of the pickle, I'm unable to use register_unpickle_override or see anyway to redirect that specific pickle without changing the behavior of generic_factory_unpickle().

I don't really understand how register_unpickle_override is working. It seems that it can only work if unpickle_global is involved---and it is not clear to me if this is involved everywhere or only if there is no __reduce__ method around.

I'm thinking the best thing to do is to change generic_factory_unpickle() to accept a general object as input and if the first input is no longer a UniqueFactory, then call it with the arguments given as part of the pickle.

Makes sense to me. And I even think the following idiom would not be slower in terms of calling overhead than the current implementation:

def generic_factory_unpickle(factory, *args):
    cdef UniqueFactory F
    try:
        F = factory
    except TypeError:
        return factory(*args[1:])
    return F.get_object(*args)

Note that I wrote factory(*args[1:]), since the first argument is the version information. So, either we need to shave it off here, or the new class (or function) replacing the old factory is required to take care of this.

I'm also thinking we should change this behavior so we don't keep running into this problem.

I think with my suggestion above, unpickling of using factory would work as quickly as it used to, and it would relatively easily allow to replace the factory by something else (provided that this "something else" takes care of the argument mangling and the version information that is involved in the factory pickle).

[simon-king-jena]

comment:45

PS: The documentation breaks. After repeated attempts of make doc-clean and make, I keep getting

OSError: [misc     ] /home/king/Sage/git/sage/src/doc/en/reference/misc/sage/misc/indexed_generators.rst:11: WARNING: autodoc can't import/find module 'sage.misc.indexed_generators', it reported error: "No module named indexed_generators", please check your spelling and sys.path

[tscrim]

Changed dependencies from #15309, #15169 to #15309, #15169, #16349

[tscrim]

comment:46

This is now #16349. Try sage -sync-build (there's probably the old file in the lib build folder which is getting picked up).

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

d570dda	Merge branch 'develop' into public/monoids/15289-indexed
321a9e4	Unpickling when replacing an old UniqueFactory by something new
187d1aa	Fix arguments passed to the constructor that replaces an old UniqueFactory
83c79ef	Merge branch 'u/SimonKing/ticket/16349' of trac.sagemath.org:sage into public/pickling/unique_factories-16349
2df09ed	Implemented an unpickle override for UniqueFactory.
c7646c1	Fixed doctest caused by me forgetting the registration is global.
d0c9eb3	Merge branch 'public/pickling/unique_factories-16349' into public/monoids/15289-indexed
bf16135	Fixed pickling issues and now using index_set as 1st arg to Free*Monoid().

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 1ff1a04 to bf16135

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from bf16135 to 164fb4b

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

164fb4b

Fixed documentation.

[tscrim]

comment:50

Sorry that actually is my fault, I forgot to move the file over in the doc index.rst files.

[rwst]

comment:51

    AttributeError: 'FreeAbelianMonoid_class' object has no attribute 'monoid_generators'
**********************************************************************
6 items had failures:
   3 of   7 in sage.monoids.indexed_free_monoid.IndexedFreeMonoidElement.__init__
   1 of   8 in sage.monoids.indexed_free_monoid.IndexedMonoid.__classcall__
   1 of   9 in sage.monoids.indexed_free_monoid.IndexedMonoid.__init__
   1 of   5 in sage.monoids.indexed_free_monoid.IndexedMonoid._an_element_
   1 of   5 in sage.monoids.indexed_free_monoid.IndexedMonoid.monoid_generators
   3 of  11 in sage.monoids.indexed_free_monoid.IndexedMonoidElement.__init__
    [213 tests, 10 failures, 0.29 s]
----------------------------------------------------------------------
sage -t src/sage/monoids/indexed_free_monoid.py  # 10 doctests failed

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

ecdc00f

Fixed doctests because of different processing of input.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 164fb4b to ecdc00f

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

909196e

Merge branch 'develop' into public/monoids/15289-indexed

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from ecdc00f to 909196e

[nthiery]

comment:55

Hi Travis,

Sorry for being slow once again. Unless I missed it, you haven't commented on my suggestions for:

• Using dict_addition whenever meaningful
• Implementing Groups().Commutative().free() rather than Groups().free(commutative=True) now that #10963 is in (I can do that if you want)

• Putting index_set as first argument to the free methods to make the following work properly:

     sage: C.free([1,2,3])
     Free monoid indexed by None

  By the way, with no argument there should either be some reasonable default, or an error:

     sage: C.free()
     Free monoid indexed by None

• Construction of the generators by F(i) (see [comment:37]).
• [comment:32]

Cheers, Nicolas

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

1db7c1d	Merge branch 'develop' into public/monoids/15289-indexed
73a4e48	Implemented other changes/improvements Nicolas suggested.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 909196e to 73a4e48

[tscrim]

comment:57

Hey Nicolas,

Thanks for getting back to this ticket. I know how hard it can be to find time.

Replying to @nthiery:

• Using dict_addition whenever meaningful

Done.

• Implementing Groups().Commutative().free() rather than Groups().free(commutative=True) now that #10963 is in (I can do that if you want)

Done (now that #10963 is in; big yay!). I also cleaned up some of the logic and made the *.free() always return the indexed monoid/group except for Groups.free(n, names) since IMO (perhaps not-so-humble) the indexed version is better since it's a proper parent and more general. I can revert this if desired.

• Putting index_set as first argument to the free methods to make the following work properly:

     sage: C.free([1,2,3])
     Free monoid indexed by None

I had missed this somehow for monoids (but I had done it for groups).

By the way, with no argument there should either be some reasonable default, or an error:

       sage: C.free()
       Free monoid indexed by None

Now it raises an error.

• Construction of the generators by F(i) (see [comment:37]).

I've made it a forbidden construction.

• [comment:32]

I think we should just use CombinatorialFreeModule over ZZ for this, or do you think it would be too heavy-handed? However I don't think it would be difficult to do (following your comment).

Best,

Travis

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

b9f6f18	Merge branch 'develop' into public/monoids/15289-indexed
d4fe805	Fixed documentation links.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 73a4e48 to d4fe805

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from d4fe805 to 56cd05d

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

56cd05d

Some fixes discussed.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

f1f5547

Fixed category issue.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 56cd05d to f1f5547

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from f1f5547 to f6a73fd

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

f6a73fd

Fixed failing doctests in categories due to new intermediate categories.

[nthiery]

comment:62

I checked the recent commits, and we have been discussing side by side for the last ones today. doc and pdf doc compiles. Positive review assuming all long tests pass (running them now).

Thanks Travis!

Cheers, Nicolas

[nthiery]

Reviewer: Nicolas M. Thiéry

[tscrim]

comment:64

Thank you for doing the review Nicolas.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. New commits:

69ec7b2

Fixed c3_controlled.pyx doctests.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from f6a73fd to 69ec7b2

[vbraun]

Changed branch from public/monoids/15289-indexed to 69ec7b2

[kcrisman]

Changed commit from 69ec7b2 to none

[kcrisman]

comment:68

Hi guys, please see http://ask.sagemath.org/question/25763/incorrect-parsing-of-docstring-of-sagestructureindexed_generatorsindexedgenerators/ where it seems that putting "\left" becomes something quite different in html. I assume this is not desired; see #17745.