[QUESTION] Defining and using metaclasses with pybind11

[jbms]

I am aware of the pybind11::metaclass option that can be passed to pybind11::class_, but I can't find any documentation on how it is supposed to be used. There is a single test case here:

https://github.com/pybind/pybind11/blob/028812ae7eee307dca5f8f69d467af7b92cc41c8/tests/test_methods_and_attributes.cpp#L282

but it isn't clear what that case is showing, and I haven't been able to find a single other example of code that uses pybind11::metaclass.

What I'd like to accomplish is to make __class_getitem__ work on Python < 3.7, equivalent to the following pure python code:

class Parent(type):

    def __getitem__(self, key):
        return self.__class_getitem__(key)

class Child(metaclass=Parent):

    @classmethod
    def __class_getitem__(cls, key):
        return 'got key: %r' % (key,)

assert Child[1] == 'got key: 1'

Note that in Python >= 3.7, this example also works if we eliminate Parent and just use type as the metaclass of Child.

I'd like to accomplish this same thing, where both Parent and Child are defined using pybind11 rather than pure Python.

My initial attempt was:

  struct Parent {};
  struct Child {};

  py::class_<Parent> cls_parent(
      m, "Parent",
      py::handle(reinterpret_cast<PyObject*>(
          pybind11::detail::get_internals().default_metaclass));
  cls_parent.def("__getitem__", [](Parent& self, py::object key) {
    return py::cast(&self).attr("__class_getitem__")(key);
  });

  py::class_<Child> cls_child(m, "Child", py::metaclass(cls_parent));
  cls_child.def_static("__class_getitem__",
                       [](std::string key) { return "got key: " + key; });

but that crashes while creating cls_parent: due to the t_size >= b_size condition in the extra_ivars function in typeobject.c.

[jbms]

I managed to get it working using plain CPython APIs:

PyTypeObject* GetClassGetitemMetaclass() {
#if PY_VERSION_HEX < 0x030700000
  // Polyfill __class_getitem__ support for Python < 3.7
  static auto* metaclass = [] {
    PyTypeObject* base_metaclass =
        pybind11::detail::get_internals().default_metaclass;
    PyType_Slot slots[] = {
        {Py_tp_base, base_metaclass},
        {Py_mp_subscript,
         (void*)+[](PyObject* self, PyObject* arg) -> PyObject* {
           auto method = py::reinterpret_steal<py::object>(
               PyObject_GetAttrString(self, "__class_getitem__"));
           if (!method.ptr()) return nullptr;
           return PyObject_CallFunctionObjArgs(method.ptr(), arg, nullptr);
         }},
        {0},
    };
    PyType_Spec spec = {};
    spec.name = "_Metaclass";
    spec.basicsize = base_metaclass->tp_basicsize;
    spec.flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE;
    spec.slots = slots;
    PyTypeObject* metaclass = (PyTypeObject*)PyType_FromSpec(&spec);
    if (!metaclass) throw py::error_already_set();
    return metaclass;
  }();
  return metaclass;
#else  // Python version >= 3.7 supports __class_getitem__ natively.
  return nullptr;
#endif
}

Is it possible to define metaclasses with pybind11 directly, though?

In this case it ended up being pretty easy to define with the CPython API directly so I suppose there isn't really a need for that.

[EricCousineau-TRI]

FWIW I'm briefly trying out something semi-related for #2332; at the moment, I'm just trying to write it in pure Python, but may fall back to CPython API like you tried.

Can I ask how you passed the result of GetClassGetitemMetaclass() to your py::class_? Did you just cast it to py::handle?

EDIT: I think that's about what I did, but for py::object:

reinterpret_borrow<py::object>(py::reinterpret_cast<PyObject*>(py::detail::get_internals().default_metaclass))

[jbms]

Yes, I did just cast it to PyObject*: https://github.com/google/tensorstore/blob/24053fc2492ea958109ade321bdc8456688167f6/python/tensorstore/dim_expression.cc#L350

[nicola-gigante]

Hi! I'm trying to use py::metaclass but it doesn't work for me. This question seems to ask what I need but there is no general answer.

For what I need to do, you can look at this SO question. In a few words: I'm trying to define __instancecheck__ to overload isinstance() but I cannot define a custom metaclass.

How is it supposed to be done?

[jbms]

I don't believe there is any built-in support in pybind11 for defining a metaclass --- instead you have to use the Python C API directly to define the metaclass, as I did here: https://github.com/google/tensorstore/blob/24053fc2492ea958109ade321bdc8456688167f6/python/tensorstore/dim_expression.cc#L350

[nicola-gigante]

I see, thanks. I'm not familiar at all with the C API. Can you help me understand your code and maybe adapt it to __instancecheck__? Also, I see you define a method contextually to the class definition. Is it possible to create the metaclass using the C API but then obtaining a py::handle object and defining the method using pybind11? Also see this discussion.

[ezyang]

On the metaclass, you'd have to define tp_methods to have an instancecheck definition, then it should work