Open simon-king-jena opened 7 years ago
Branch: u/SimonKing/metaclass_framework
The branch is just sage-7.4.rc0
without any extra commits... was this intentional?
As explained on the sage-devel
post, I am curious to see if you can make this work on Python 3.
Replying to @jdemeyer:
The branch is just
sage-7.4.rc0
without any extra commits... was this intentional?
Yes.
As explained on the
sage-devel
post, I am curious to see if you can make this work on Python 3.
I am confident that it would work on Python 3, of course with the modified syntax
class Foo(metaclass=Bar):
...
instead of
class Foo:
__metaclass__ = Bar
Replying to @simon-king-jena:
I am confident that it would work on Python 3
I am very interested to see how you would do this!
Replying to @jdemeyer:
I am very interested to see how you would do this!
The idea is: "metaclass used in Sage" should be synonymous to "instance (not sub-class!) of SageMetaclass" (hence, SageMetaclass is a metametaclass and is itself a subclass of type).
SageMetaclass defines a __call__
method for its instances (thus overriding type.__call__
). When such instance is used as a metaclass when defining a class, then the metaclass' __call__
method inspects the bases of the to-be-created class and dynamically creates (if necessary) a common sub-class of the metaclasses of the bases. Eventually, the class is created as an instance of the common (dynamic) metaclass. And python will be happy, because the metaclass of the new class is a sub-class of the metaclasses of the given bases.
Both in Python 2 and in Python 3, the metaclass is called during creation of a class (which is where all magic happens). That's why I think it should work on Python 3 as well.
Branch pushed to git repo; I updated commit sha1. New commits:
4d493e0 | Metaclass framework: Proof of concept |
One comment: since this isn't really Sage-specific, could you use a different name instead of SageMetaclass
, something which does not refer to Sage?
I have pushed a proof of concept. The following works:
sage: from sage.structure.metaclass.metaclass import MyNestedClass, MyUniqueRepresentation
sage: class MyCombinedExample(MyNestedClass, MyUniqueRepresentation):
....: pass
....:
sage: C = MyCombinedExample(4,5)
sage: C is MyCombinedExample(4,5)
True
sage: C is MyCombinedExample(5,5)
False
sage: C.Test() is C.Test()
True
sage: loads(dumps(C)) is C
True
The metaclasses of the base classes MyNestedClass and MyUniqueRepresentation in the above class definition are incompatible:
sage: type(MyUniqueRepresentation).__mro__
(<class 'sage.structure.metaclass.metaclass.ClasscallMetaclass'>,
<type 'type'>,
<type 'object'>)
sage: type(MyNestedClass).__mro__
(<class 'sage.structure.metaclass.metaclass.NestedClassMetaclass'>,
<type 'type'>,
<type 'object'>)
Nonetheless, it just works. In fact, the resulting class is instance of a metaclass that is created on the fly:
sage: type(C)
<class '__main__.MyCombinedExample'>
sage: type(type(C))
<class '__main__.ClasscallNestedClassMetaclass'>
sage: type(type(type(C)))
<class 'sage.structure.metaclass.metaclass.DynamicSageMetaclass'>
Slightly strange for me is this: The metaclass that actually appears in the definition of ClasscallMetaclass
and NestedClassMetaclass
is not the class SageMetaclass
but the function sage_metaclass
--- from the patch:
from sage.misc.nested_class import nested_pickle
class NestedClassMetaclass:
__metaclass__ = sage_metaclass
def __init__(cls, name, bases, namespace):
nested_pickle(cls)
If I replace sage_metaclass
by SageMetaclass
then the example stops working, which is something I don't fully understand. Unless Python 3 disallows to use a function as __metaclass__
, there is hope that it would work with Python 3 as well.
Replying to @jdemeyer:
One comment: since this isn't really Sage-specific, could you use a different name instead of
SageMetaclass
, something which does not refer to Sage?
Sure, so far it is just a proof of concept.
Would you agree with MetaMetaclass
and (for the function that actually appears in the metaclass definitions) meta_metaclass
? Or maybe just Metaclass
and metaclass
.
Replying to @simon-king-jena:
Would you agree with
MetaMetaclass
and (for the function that actually appears in the metaclass definitions)meta_metaclass
? Or maybe justMetaclass
andmetaclass
.
What about AutoMetaclass
to refer to the automatic creation of combined metaclasses?
Replying to @jdemeyer:
Replying to @simon-king-jena:
Would you agree with
MetaMetaclass
and (for the function that actually appears in the metaclass definitions)meta_metaclass
? Or maybe justMetaclass
andmetaclass
.What about
AutoMetaclass
to refer to the automatic creation of combined metaclasses?
There are two things in the code: SageMetaclass
, which is for hard-coded (single purpose, atomic) metaclasses, and DynamicSageMetaclass
for those that are dynamically created.
What about the following scheme:
BaseMetaclass
(instead of SageMetaclass
): Its instances are metaclasses that are basic in the sense that one can combine them.DynamicMetaclass
or CombinedMetaclass
(instead of DynamicSageMetaclass
): Its instances are metaclasses that are dynamically/automatically created from a combination of base metaclasses.auto_metaclass
: The function that I currently call sage_metaclass
. It returns an instance of BaseMetaclass
or CombinedMetaclass
, depending on the context.<Feature>ClassMetaclass
, which corresponds to the existing naming scheme in Sage and means that an instance of <Feature>ClassMetaclass
is a class that provides a single feature (such as: It has a classcall, or it allows nesting).<Feature1><Feature2>...<Feature_n>ClassMetaclass
(which in the current proof of concept is slightly different). So, it would be ClasscallDynamicNestedClassMetaclass
for a metaclass whose instances are dynamic classes that have a classcall and allow nesting.I think auto_metaclass
would be a good name for the function that appears in the metaclass definition, i.e.
class NestedClassMetaclass:
__metaclass__ = auto_metaclass
def __init__(cls, name, bases, namespace):
nested_pickle(cls)
Did you already test on Python 3? Because personally, I am still thinking that you are wasting your time to implement something which won't work anyway. But of course, I am gladly proven wrong.
Attempt to do the same tricks in Python3
Attachment: metatest3.py.gz
The attachment: metatest3.py is an attempt to port stuff to Python3 --- which sadly fails. In Python3, I get
>>> exec(open("/home/king/Sage/work/metaclass/metatest3.py").read())
>>> class MyCombinedExample(MyNestedClass, MyUniqueRepresentation):
... class SomeClass:
... pass
... class SomeCachedClass(MyUniqueRepresentation):
... def __init__(self, x):
... self.x = x
...
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases
whereas the analogous example in Sage works.
I am now trying whether using !ABCMeta (from Python's abc module) can help. It would hopefully fool Python into believing that the different metaclasses are subclasses of each other and thus would make it stop complaining about metaclass conflicts.
From some tests, I get the impression that the abc module of Python 3 is broken. I am getting errors like
>>> SageMetaclass.__subclasscheck__(NestedClassMetaclass, type(1))
being called <class '__main__.NestedClassMetaclass'> <class 'int'>
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.5/abc.py", line 225, in __subclasscheck__
for scls in cls.__subclasses__():
TypeError: descriptor '__subclasses__' of 'type' object needs an argument
hence, the error does not occur in a call in my code but in a call at line 225 of the abc module.
I made progress regarding Python 3, but at the end of the day you may be right: It seems that it won't work.
The question is why it won't. The error is "TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases". However, using Python's abc-module, I arranged things so that issubclass(ClasscallMetaclass,NestedClassMetaclass)
returns true.
Hence, it seems that Python does in fact not use the logic of issubclass
when dealing with a metaclass. The remaining questions are: What does Python do instead, and how can it be hooked?
Replying to @simon-king-jena:
However, using Python's abc-module, I arranged things so that
issubclass(ClasscallMetaclass,NestedClassMetaclass)
returns true.
Just a minor comment: you don't need the abc
module for that (especially if it's giving you problems). You could just define __instancecheck__
and/or __subclasscheck__
directly.
Replying to @simon-king-jena:
Hence, it seems that Python does in fact not use the logic of
issubclass
when dealing with a metaclass.
Right, it uses PyType_IsSubtype which doesn't use __subclasscheck__
.
The remaining questions are: What does Python do instead, and how can it be hooked?
It checks only the MRO. Basically, it implements issubclass(x, y)
as y in x.__mro__
.
Replying to @jdemeyer:
Right, it uses PyType_IsSubtype which doesn't use
__subclasscheck__
.
Ouch. That's efficient, but bad for my purpose.
I just came up with a totally different approach.
In a nutshell: There should only be a single metaclass (hence, there will be no metaclass conflict), but there are also several functions affecting the class creation, and the metaclass will choose which of the functions are to be applied, based on attributes of the class and its bases.
More elaborated:
One could have just a single metaclass, say, class UniversalMetaclass(type)
, and a couple of functions, say, nested_class_metaclass
and classcall_metaclass
. These functions would return instances of UniversalMetaclass
. So, roughly one would have
class SomeNestedClass(metaclass=nested_class_metaclass):
...
class UniqueRepresentation(metaclass=classcall_metaclass):
...
sage: type(SomeNestedClass) == type(UniqueRepresentation) == UniversalMetaclass
True
Now, when one does
class CombinedClass(UniqueRepresentation, SomeNestedClass):
...
then first of all Python 3 would not complain, since the metaclasses of both bases are identical. What we want is that the features used in the creation of UniqueRepresentation
(namely addition of a classcall method) and of NestedClass
(namely invocation of nested_pickle
upon creation of the class) are combined.
So, the question is: Given UniqueRepresentation
and SomeNestedClass
, how can we access the functions classcall_metaclass
and nested_class_metaclass
? In Python 2, this would be UniqueRepresentation.__metaclass__
, but that's gone in Python 3.
But why couldn't we simply have our own attribute, say, cls.__features__
, returning a tuple (or frozen set) of functions that were invoked during creation of the class cls
?
In the creation of CombinedClass
, Python 3 would conclude that UniversalMetaclass
is responsible for the class creation (as it is the common metaclass of the bases).
We could make it so that UniversalMetaclass
applies each function in X.__features__
during creation of CombinedClass
, for each base class X
of CombinedClass
, but of course without repetitions. And finally, it would store the tuple of functions in CombinedClass.__features__
.
I will try this.
I did some research on the differences of Python 2 and Python 3 regarding metaclasses. In Python 2, the metaclass seems to come into play quite late in the class creation. But in Python 3, the first step is to determine the metaclass (which means that incompatible metaclasses of the bases result in an immediate error). Then, metaclass.__prepare__
is called to initialise the namespace of the to-be-created class. Then, the body of the class definition is executed, followed by calling the metaclass to actually create the class. And eventually, class decorators are called.
So, we have in Python 3:
>>> def c_decorator(cls):
... print("decorating the class")
... return cls
...
>>> class Meta(type):
... @classmethod
... def __prepare__(mcls, name, bases):
... print("preparing the namespace")
... return dict()
... def __new__(cls, name, bases, namespace):
... print("new class")
... return type(name, bases, namespace)
...
>>> @c_decorator
... class C(metaclass=Meta):
... pass
...
preparing the namespace
new class
decorating the class
Both __prepare__
and class decorators are new in Python 3. Perhaps they can be used?
Again, it would be needed to have a UniversalMetaclass
, since there must be no conflicts with the metaclasses of the bases. Then, UniversalMetaclass.__prepare__
can add some information to the class' namespace, taking into account the bases. Based on this information, UniversalMetaclass.__new__
can do special things during the class creation.
Class decorators seem less relevant to me. They are called after creation of the class, but they wouldn't be called again when sub-classing. So, that post-production should better be done inside of UniversalMetaclass.__new__
.
Should we close this?
Replying to @jdemeyer:
Should we close this?
I stopped working on it. However, given the last few comments, "won't fix" would be the wrong resolution, because I do think that the topic should be reconsidered after switching to Python3.
Currently, SageMath hardcodes the metaclasses DynamicMetaclass, DynamicClasscallMetaclass, DynamicInheritComparisonMetaclass, DynamicInheritComparisonClasscallMetaclass, NestedClassMetaclass, InheritComparisonMetaclass, and InheritComparisonClasscallMetaclass.
Apparently, they are all based on few single-purpose metaclasses (DynamicMetaclass, InheritComparisonMetaclass, NestedClassMetaclass and ClasscallMetaclass), and the hardcoded combinations exist because Python doesn't allow much freedom when providing different metaclasses in the bases of a class definition. And apparently several combinations are missing.
The purpose of this ticket is to allow for an automatic creation of combined metaclasses, so that only the single-purpose metaclasses need to be implemented, and all other metaclasses will be created dynamically.
The approach is by introducing a meta-metaclass SageMetaclass --- i.e., any metaclass in Sage is instance of SageMetaclass.
Component: refactoring
Keywords: metaclass dynamic
Branch/Commit: u/SimonKing/metaclass_framework @
4d493e0
Issue created by migration from https://trac.sagemath.org/ticket/21681