Metaclass framework

[simon-king-jena]

Currently, SageMath hardcodes the metaclasses DynamicMetaclass, DynamicClasscallMetaclass, DynamicInheritComparisonMetaclass, DynamicInheritComparisonClasscallMetaclass, NestedClassMetaclass, InheritComparisonMetaclass, and InheritComparisonClasscallMetaclass.

Apparently, they are all based on few single-purpose metaclasses (DynamicMetaclass, InheritComparisonMetaclass, NestedClassMetaclass and ClasscallMetaclass), and the hardcoded combinations exist because Python doesn't allow much freedom when providing different metaclasses in the bases of a class definition. And apparently several combinations are missing.

The purpose of this ticket is to allow for an automatic creation of combined metaclasses, so that only the single-purpose metaclasses need to be implemented, and all other metaclasses will be created dynamically.

The approach is by introducing a meta-metaclass SageMetaclass --- i.e., any metaclass in Sage is instance of SageMetaclass.

Component: refactoring

Keywords: metaclass dynamic

Branch/Commit: u/SimonKing/metaclass_framework @ 4d493e0

Issue created by migration from https://trac.sagemath.org/ticket/21681

[simon-king-jena]

Branch: u/SimonKing/metaclass_framework

[jdemeyer]

Commit: 1744c65

[jdemeyer]

comment:2

The branch is just sage-7.4.rc0 without any extra commits... was this intentional?

As explained on the sage-devel post, I am curious to see if you can make this work on Python 3.

[simon-king-jena]

comment:3

Replying to @jdemeyer:

The branch is just sage-7.4.rc0 without any extra commits... was this intentional?

Yes.

As explained on the sage-devel post, I am curious to see if you can make this work on Python 3.

I am confident that it would work on Python 3, of course with the modified syntax

class Foo(metaclass=Bar):
    ...

instead of

class Foo:
   __metaclass__ = Bar

[jdemeyer]

comment:4

Replying to @simon-king-jena:

I am confident that it would work on Python 3

I am very interested to see how you would do this!

[simon-king-jena]

comment:5

Replying to @jdemeyer:

I am very interested to see how you would do this!

The idea is: "metaclass used in Sage" should be synonymous to "instance (not sub-class!) of SageMetaclass" (hence, SageMetaclass is a metametaclass and is itself a subclass of type).

SageMetaclass defines a __call__ method for its instances (thus overriding type.__call__). When such instance is used as a metaclass when defining a class, then the metaclass' __call__ method inspects the bases of the to-be-created class and dynamically creates (if necessary) a common sub-class of the metaclasses of the bases. Eventually, the class is created as an instance of the common (dynamic) metaclass. And python will be happy, because the metaclass of the new class is a sub-class of the metaclasses of the given bases.

Both in Python 2 and in Python 3, the metaclass is called during creation of a class (which is where all magic happens). That's why I think it should work on Python 3 as well.

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 1744c65 to 4d493e0

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

4d493e0

Metaclass framework: Proof of concept

[jdemeyer]

comment:7

One comment: since this isn't really Sage-specific, could you use a different name instead of SageMetaclass, something which does not refer to Sage?

[simon-king-jena]

comment:8

I have pushed a proof of concept. The following works:

sage: from sage.structure.metaclass.metaclass import MyNestedClass, MyUniqueRepresentation
sage: class MyCombinedExample(MyNestedClass, MyUniqueRepresentation):
....:     pass
....: 
sage: C = MyCombinedExample(4,5)
sage: C is MyCombinedExample(4,5)
True
sage: C is MyCombinedExample(5,5)
False
sage: C.Test() is C.Test()
True
sage: loads(dumps(C)) is C
True

The metaclasses of the base classes MyNestedClass and MyUniqueRepresentation in the above class definition are incompatible:

sage: type(MyUniqueRepresentation).__mro__
(<class 'sage.structure.metaclass.metaclass.ClasscallMetaclass'>,
 <type 'type'>,
 <type 'object'>)
sage: type(MyNestedClass).__mro__
(<class 'sage.structure.metaclass.metaclass.NestedClassMetaclass'>,
 <type 'type'>,
 <type 'object'>)

Nonetheless, it just works. In fact, the resulting class is instance of a metaclass that is created on the fly:

sage: type(C)
<class '__main__.MyCombinedExample'>
sage: type(type(C))
<class '__main__.ClasscallNestedClassMetaclass'>
sage: type(type(type(C)))
<class 'sage.structure.metaclass.metaclass.DynamicSageMetaclass'>

Slightly strange for me is this: The metaclass that actually appears in the definition of ClasscallMetaclass and NestedClassMetaclass is not the class SageMetaclass but the function sage_metaclass --- from the patch:

from sage.misc.nested_class import nested_pickle
class NestedClassMetaclass:
    __metaclass__ = sage_metaclass
    def __init__(cls, name, bases, namespace):
        nested_pickle(cls)

If I replace sage_metaclass by SageMetaclass then the example stops working, which is something I don't fully understand. Unless Python 3 disallows to use a function as __metaclass__, there is hope that it would work with Python 3 as well.

[simon-king-jena]

comment:9

Replying to @jdemeyer:

One comment: since this isn't really Sage-specific, could you use a different name instead of SageMetaclass, something which does not refer to Sage?

Sure, so far it is just a proof of concept.

Would you agree with MetaMetaclass and (for the function that actually appears in the metaclass definitions) meta_metaclass? Or maybe just Metaclass and metaclass.

[jdemeyer]

comment:10

Replying to @simon-king-jena:

Would you agree with MetaMetaclass and (for the function that actually appears in the metaclass definitions) meta_metaclass? Or maybe just Metaclass and metaclass.

What about AutoMetaclass to refer to the automatic creation of combined metaclasses?

[simon-king-jena]

comment:11

Replying to @jdemeyer:

Replying to @simon-king-jena:

Would you agree with MetaMetaclass and (for the function that actually appears in the metaclass definitions) meta_metaclass? Or maybe just Metaclass and metaclass.

What about AutoMetaclass to refer to the automatic creation of combined metaclasses?

There are two things in the code: SageMetaclass, which is for hard-coded (single purpose, atomic) metaclasses, and DynamicSageMetaclass for those that are dynamically created.

What about the following scheme:

• BaseMetaclass (instead of SageMetaclass): Its instances are metaclasses that are basic in the sense that one can combine them.
• DynamicMetaclass or CombinedMetaclass (instead of DynamicSageMetaclass): Its instances are metaclasses that are dynamically/automatically created from a combination of base metaclasses.
• auto_metaclass: The function that I currently call sage_metaclass. It returns an instance of BaseMetaclass or CombinedMetaclass, depending on the context.
• The base metaclasses should be named <Feature>ClassMetaclass, which corresponds to the existing naming scheme in Sage and means that an instance of <Feature>ClassMetaclass is a class that provides a single feature (such as: It has a classcall, or it allows nesting).
• The combined metaclasses should be named <Feature1><Feature2>...<Feature_n>ClassMetaclass (which in the current proof of concept is slightly different). So, it would be ClasscallDynamicNestedClassMetaclass for a metaclass whose instances are dynamic classes that have a classcall and allow nesting.

I think auto_metaclass would be a good name for the function that appears in the metaclass definition, i.e.

class NestedClassMetaclass:
    __metaclass__ = auto_metaclass
    def __init__(cls, name, bases, namespace):
        nested_pickle(cls)

[jdemeyer]

comment:12

Did you already test on Python 3? Because personally, I am still thinking that you are wasting your time to implement something which won't work anyway. But of course, I am gladly proven wrong.

[simon-king-jena]

Attempt to do the same tricks in Python3

[simon-king-jena]

comment:13

Attachment: metatest3.py.gz

The attachment: metatest3.py is an attempt to port stuff to Python3 --- which sadly fails. In Python3, I get

>>> exec(open("/home/king/Sage/work/metaclass/metatest3.py").read())
>>> class MyCombinedExample(MyNestedClass, MyUniqueRepresentation):
...     class SomeClass:
...         pass
...     class SomeCachedClass(MyUniqueRepresentation):
...         def __init__(self, x):
...             self.x = x
... 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases

whereas the analogous example in Sage works.

[simon-king-jena]

comment:14

I am now trying whether using !ABCMeta (from Python's abc module) can help. It would hopefully fool Python into believing that the different metaclasses are subclasses of each other and thus would make it stop complaining about metaclass conflicts.

[simon-king-jena]

comment:15

From some tests, I get the impression that the abc module of Python 3 is broken. I am getting errors like

>>> SageMetaclass.__subclasscheck__(NestedClassMetaclass, type(1))
being called <class '__main__.NestedClassMetaclass'> <class 'int'>
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.5/abc.py", line 225, in __subclasscheck__
    for scls in cls.__subclasses__():
TypeError: descriptor '__subclasses__' of 'type' object needs an argument

hence, the error does not occur in a call in my code but in a call at line 225 of the abc module.

[simon-king-jena]

comment:16

I made progress regarding Python 3, but at the end of the day you may be right: It seems that it won't work.

The question is why it won't. The error is "TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases". However, using Python's abc-module, I arranged things so that issubclass(ClasscallMetaclass,NestedClassMetaclass) returns true.

Hence, it seems that Python does in fact not use the logic of issubclass when dealing with a metaclass. The remaining questions are: What does Python do instead, and how can it be hooked?

[jdemeyer]

comment:17

Replying to @simon-king-jena:

However, using Python's abc-module, I arranged things so that issubclass(ClasscallMetaclass,NestedClassMetaclass) returns true.

Just a minor comment: you don't need the abc module for that (especially if it's giving you problems). You could just define __instancecheck__ and/or __subclasscheck__ directly.

[jdemeyer]

comment:18

Replying to @simon-king-jena:

Hence, it seems that Python does in fact not use the logic of issubclass when dealing with a metaclass.

Right, it uses PyType_IsSubtype which doesn't use __subclasscheck__.

The remaining questions are: What does Python do instead, and how can it be hooked?

It checks only the MRO. Basically, it implements issubclass(x, y) as y in x.__mro__.

[simon-king-jena]

comment:19

Replying to @jdemeyer:

Right, it uses PyType_IsSubtype which doesn't use __subclasscheck__.

Ouch. That's efficient, but bad for my purpose.

I just came up with a totally different approach.

In a nutshell: There should only be a single metaclass (hence, there will be no metaclass conflict), but there are also several functions affecting the class creation, and the metaclass will choose which of the functions are to be applied, based on attributes of the class and its bases.

More elaborated:

One could have just a single metaclass, say, class UniversalMetaclass(type), and a couple of functions, say, nested_class_metaclass and classcall_metaclass. These functions would return instances of UniversalMetaclass. So, roughly one would have

class SomeNestedClass(metaclass=nested_class_metaclass):
    ...

class UniqueRepresentation(metaclass=classcall_metaclass):
    ...
sage: type(SomeNestedClass) == type(UniqueRepresentation) == UniversalMetaclass
True

Now, when one does

class CombinedClass(UniqueRepresentation, SomeNestedClass):
    ...

then first of all Python 3 would not complain, since the metaclasses of both bases are identical. What we want is that the features used in the creation of UniqueRepresentation (namely addition of a classcall method) and of NestedClass (namely invocation of nested_pickle upon creation of the class) are combined.

So, the question is: Given UniqueRepresentation and SomeNestedClass, how can we access the functions classcall_metaclass and nested_class_metaclass? In Python 2, this would be UniqueRepresentation.__metaclass__, but that's gone in Python 3.

But why couldn't we simply have our own attribute, say, cls.__features__, returning a tuple (or frozen set) of functions that were invoked during creation of the class cls?

In the creation of CombinedClass, Python 3 would conclude that UniversalMetaclass is responsible for the class creation (as it is the common metaclass of the bases).

We could make it so that UniversalMetaclass applies each function in X.__features__ during creation of CombinedClass, for each base class X of CombinedClass, but of course without repetitions. And finally, it would store the tuple of functions in CombinedClass.__features__.

I will try this.

[simon-king-jena]

comment:20

I did some research on the differences of Python 2 and Python 3 regarding metaclasses. In Python 2, the metaclass seems to come into play quite late in the class creation. But in Python 3, the first step is to determine the metaclass (which means that incompatible metaclasses of the bases result in an immediate error). Then, metaclass.__prepare__ is called to initialise the namespace of the to-be-created class. Then, the body of the class definition is executed, followed by calling the metaclass to actually create the class. And eventually, class decorators are called.

So, we have in Python 3:

>>> def c_decorator(cls):
...     print("decorating the class")
...     return cls
... 
>>> class Meta(type):
...     @classmethod
...     def __prepare__(mcls, name, bases):
...         print("preparing the namespace")
...         return dict()
...     def __new__(cls, name, bases, namespace):
...         print("new class")
...         return type(name, bases, namespace)
... 
>>> @c_decorator
... class C(metaclass=Meta):
...     pass
... 
preparing the namespace
new class
decorating the class

Both __prepare__ and class decorators are new in Python 3. Perhaps they can be used?

Again, it would be needed to have a UniversalMetaclass, since there must be no conflicts with the metaclasses of the bases. Then, UniversalMetaclass.__prepare__ can add some information to the class' namespace, taking into account the bases. Based on this information, UniversalMetaclass.__new__ can do special things during the class creation.

Class decorators seem less relevant to me. They are called after creation of the class, but they wouldn't be called again when sub-classing. So, that post-production should better be done inside of UniversalMetaclass.__new__.

[jdemeyer]

comment:21

Should we close this?

[simon-king-jena]

comment:22

Replying to @jdemeyer:

Should we close this?

I stopped working on it. However, given the last few comments, "won't fix" would be the wrong resolution, because I do think that the topic should be reconsidered after switching to Python3.