NEURON v9 python syntax does not allow use of template instance name

[nrnhines]

Version 8.2 and before allows

$ python
>>> from neuron import h
>>> v = h.Vector()
>>> v
Vector[0]
>>> h.Vector[0]
Vector[0]

But with the current master

$ python
>>> from neuron import h
>>> v = h.Vector()
>>> v
Vector[0]
>>> h.Vector[0]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: type 'hoc.Vector' is not subscriptable

[nrnhines]

@ramcdougal I'm stuck on this and could use your help. From what I see in nrnpython/nrnpy_hoc.h the slots are properly filled

   {Py_mp_subscript, (void*) hocobj_slice_getitem},
   {Py_sq_item, (void*) hocobj_getitem},

but the error message generated by h.Vector[0] is coming from Python's Objects/abstract.c in that

PyObject *
PyObject_GetItem(PyObject *o, PyObject *key)
{

and I don't understand why we aren't getting into either hocobj_getitem or hocobj_slice_getitem .

Note that if I add a debugging statement in nrnpython/nrnpy_hoc.cpp

--- a/src/nrnpython/nrnpy_hoc.cpp
+++ b/src/nrnpython/nrnpy_hoc.cpp
@@ -1011,6 +1011,9 @@ static PyObject* hocobj_getattr(PyObject* subself, PyObject* pyname) {
     auto location = sym_to_type_map.find(sym);
     if (location != sym_to_type_map.end()) {
         Py_INCREF(location->second);
+printf("RRR return  %p hocobj_getitem=%p\n", location->second, hocobj_getitem);
+        PyObject* pp = (PyObject*)location->second;
+PyObject_Print(pp, stdout, 0);
         return (PyObject*) location->second;
     }

and with gdb set a breakpoint at the return, then it looks ok with

p *(*(PyTypeObject*)pp).tp_as_sequence
sq_item = 0x7ffff796a1bb <hocobj_getitem(PyObject*, Py_ssize_t)>,
p *(*(PyTypeObject*)pp).tp_as_mapping
mp_subscript = 0x7ffff796acb7 <hocobj_slice_getitem(PyObject*, PyObject*)>,

I guess my next debug strategy will be to build a -g version of python and follow what happens on return from hocobj_getattr. But if you have an idea about what is going on, let me know.

[ramcdougal]

I wonder if this broke with the meaningful types commit... Prior to that, there was no hoc.Vector object.

[nrnhines]

I believe it did. But it is not directly due to hoc.Vector per se. It happens with any hoc template, e,g hoc.Graph. I do have a -g compiled version of python3.12.3 now and I see that the first puzzle is in Objects/abstract.c PyObject_GetItem(PyObject *o, PyObject *key) where the o arg is

(gdb) p *o
$4 = {{ob_refcnt = 44, ob_refcnt_split = {44, 0}}, 
  ob_type = 0x7ffff61735e0 <PyType_Type>}

and

(gdb) p *(PyTypeObject*)o
$5 = {ob_base = {ob_base = {{ob_refcnt = 44, ob_refcnt_split = {44, 0}}, 
...
  tp_name = 0x7ffff29782f0 "hoc.Vector", tp_basicsize = 96, tp_itemsize = 0, 
...
  tp_as_number = 0x555555704a70, tp_as_sequence = 0x555555704ba8,
  tp_as_mapping = 0x555555704b90,
...

but both lines in the code

   PyMappingMethods *m = Py_TYPE(o)->tp_as_mapping; 
and
   PySequenceMethods *ms = Py_TYPE(o)->tp_as_sequence;

ended up as

(gdb) p m
$8 = (PyMappingMethods *) 0x0
(gdb) p ms
$9 = (PySequenceMethods *) 0x0

So I guess I need to look at the Py_TYPE(o) which (because it is a macro?) cannot be run from gdb

(gdb) p Py_TYPE(o)
Couldn't write extended state status: Bad address.

[nrnhines]

It's just

static inline PyTypeObject* Py_TYPE(PyObject *ob) {
    return ob->ob_type;
}

[ramcdougal]

All these types are created dynamically, I think... So maybe they just need to get a copy of the hoc.HocObject pointers

[nrnhines]

I think you are right. From the above casting with *(PyTypeObject*)o has non-null tp_as_sequence and tp_as_mapping. But from

(gdb) p *o->ob_type
$12 = {ob_base = {ob_base = {{ob_refcnt = 4294967295, ob_refcnt_split = {
          4294967295, 0}}, ob_type = 0x7ffff61735e0 <PyType_Type>}, 
    ob_size = 0}, tp_name = 0x7ffff5fa48f1 "type", tp_basicsize = 920, 
  tp_itemsize = 40, tp_dealloc = 0x7ffff5de8800 <type_dealloc>, 
  tp_vectorcall_offset = 400, tp_getattr = 0x0, tp_setattr = 0x0, 
  tp_as_async = 0x0, tp_repr = 0x7ffff5dea9d0 <type_repr>, 
  tp_as_number = 0x7ffff6173780 <type_as_number>, tp_as_sequence = 0x0, 
  tp_as_mapping = 0x0, tp_hash = 0x7ffff5edaf20 <_Py_HashPointer>, 
  tp_call = 0x7ffff5deaf20 <type_call>, tp_str = 0x7ffff5de4480 <object_str>, 
  tp_getattro = 0x7ffff5dfc9f0 <_Py_type_getattro>, 
  tp_setattro = 0x7ffff5df7910 <type_setattro>, tp_as_buffer = 0x0, 
  tp_flags = 2156420354, 
  tp_doc = 0x7ffff5fa94c0 <type_doc> "type(object) -> the object's type\ntype(name, bases, dict, **kwds) -> a new type", 
  tp_traverse = 0x7ffff5de4e80 <type_traverse>, 
  tp_clear = 0x7ffff5dea5c0 <type_clear>, 
  tp_richcompare = 0x7ffff5de5120 <object_richcompare>, 
  tp_weaklistoffset = 368, tp_iter = 0x0, tp_iternext = 0x0, 
  tp_methods = 0x7ffff6173b20 <type_methods>, 
  tp_members = 0x7ffff6173f40 <type_members>, 
  tp_getset = 0x7ffff6173d60 <type_getsets>, 
  tp_base = 0x7ffff6173440 <PyBaseObject_Type>, tp_dict = 0x0, 
  tp_descr_get = 0x0, tp_descr_set = 0x0, tp_dictoffset = 264, 
--Type <RET> for more, q to quit, c to continue without paging--

those are null. Not clear to me what object it gets added into. I guess I need to determine if o->ob_type is unique to (PyTypeObject*)o

[nrnhines]

@ramcdougal Do you think it makes sense to do a self reference, i.e. pto->ob_type = pto for all the

pto = (PyTypeObject*) PyType_FromSpecWithBases(&spec, bases);

I guess, I'll check what pto->ob_type starts out as for all the items in

        sym_to_type_map[hoc_lookup(name)] = pto;

[nrnhines]

The following results puzzle me since the pto->ob_type seems generic and should not be changed? I tested with

--- a/src/nrnpython/nrnpy_hoc.cpp
+++ b/src/nrnpython/nrnpy_hoc.cpp
@@ -3319,6 +3319,12 @@ PyObject* nrnpy_hoc() {
         exposed_py_type_names.push_back(std::string("hoc.") + name);
         spec = obj_spec_from_name(exposed_py_type_names.back().c_str());
         pto = (PyTypeObject*) PyType_FromSpecWithBases(&spec, bases);
+auto z = (PyObject*)pto;
+printf("%s %p %p\n  ", name, z, z->ob_type);
+PyObject_Print(z, stdout, 0);
+printf("  ");
+PyObject_Print((PyObject*)z->ob_type, stdout, 0);
+printf("\n");
         sym_to_type_map[hoc_lookup(name)] = pto;

and the first few lines of output are

SectionList 0x60f7b4778670 0x77aa34b735e0
  <class 'hoc.SectionList'>  <class 'type'>
SectionRef 0x60f7b4779cb0 0x77aa34b735e0
  <class 'hoc.SectionRef'>  <class 'type'>
Graph 0x60f7b477a3e0 0x77aa34b735e0
  <class 'hoc.Graph'>  <class 'type'>
HBox 0x60f7b4780330 0x77aa34b735e0
  <class 'hoc.HBox'>  <class 'type'>

[nrnhines]

I'm starting to think that the ob_type of h.Vector should be <class 'hoc.HocObject'>

[nrnhines]

Types of user define hoc templates are working as in the past. i.e.

from neuron import h

h(
    """
begintemplate A
proc init() {
  x = 2
}
endtemplate A
"""
)
a = h.A()
print(a)
print(type(a))
print(type(type(a)))
print(h.A[0])

v = h.Vector(3)
print(v)
print(type(v))
print(type(type(v)))
print(h.Vector[0])

prints the output

A[0]
<class 'hoc.HocObject'>
<class 'type'>
A[0]
Vector[0]
<class 'hoc.Vector'>
<class 'type'>
Traceback (most recent call last):
  File "/home/hines/neuron/vecinst/src/nrnpython/temp.py", line 22, in <module>
    print(h.Vector[0])
          ~~~~~~~~^^^
TypeError: type 'hoc.Vector' is not subscriptable

[nrnhines]

After a very long conversation with chatgpt 3.5, we came up with a pure python example of what I wanted, I.e.

% cat temp2.py
class CustomMeta(type):
    def __new__(cls, name, bases, dct):
        return super().__new__(cls, name, bases, dct)

    def __getitem__(cls, index):
        print("inside CustomMeta.__getitem__")
        return f"{cls}[{index}]"

class Foo(metaclass=CustomMeta):
    def __init__(self):
        print("Foo.__init__ ", self)

    def __getitem__(self, index):
        print("inside Foo.__getitem__")
        return f"{self}[{index}]"

print("type(Foo) is", type(Foo))
print("Foo[3] is ", Foo[3])
print("Foo()[5] is ", Foo()[5])

% python3 temp2.py
type(Foo) is <class '__main__.CustomMeta'>
inside CustomMeta.__getitem__
Foo[3] is  <class '__main__.Foo'>[3]
Foo.__init__  <__main__.Foo object at 0x1001bfe60>
inside Foo.__getitem__
Foo()[5] is  <__main__.Foo object at 0x1001bfe60>[5]

[nrnhines]

Here's the remainder of the conversation where I asked for a c-api version in the context of the nrnpy_hoc.cpp code. It will likely require another long conversation for me to get it working in our context.

Now, how would you get that behavior with my original fragment where I
started with
 for (auto name: py_exposed_classes) {

ChatGPT
To achieve similar behavior in your original C++ fragment, where you're creating classes using PyType_FromSpecWithBases, you can define a custom metaclass and include a __getitem__ method in that metaclass.

Here's how you can modify your original C++ code to achieve the desired behavior:

// Define the custom metaclass
static PyObject *CustomMeta_getitem(PyObject *cls, PyObject *index) {
    const char *cls_name = PyUnicode_AsUTF8(PyUnicode_FromObject(cls));
    const char *index_str = PyUnicode_AsUTF8(PyUnicode_FromObject(index));
    printf("%s[%s]\n", cls_name, index_str);
    Py_RETURN_NONE;
}

static PyMethodDef CustomMeta_methods[] = {
    {"__getitem__", CustomMeta_getitem, METH_O, "Get item"},
    {NULL, NULL, 0, NULL} /* Sentinel */
};

static PyTypeObject CustomMeta_Type = {
    PyVarObject_HEAD_INIT(NULL, 0)
    .tp_name = "custom_module.CustomMeta",
    .tp_basicsize = sizeof(PyTypeObject),
    .tp_flags = Py_TPFLAGS_DEFAULT,
    .tp_methods = CustomMeta_methods
};

// Class creation loop
for (auto name: py_exposed_classes) {
    // Create the type object for the class
    PyTypeObject* pto = (PyTypeObject*) PyType_FromSpecWithBases(&spec, bases);
    if (!pto) {
        // Handle error
    }

    // Set the metaclass to CustomMeta
    Py_INCREF(&CustomMeta_Type);
    pto->tp_base = &CustomMeta_Type;

    // Ensure type finalization
    if (PyType_Ready(pto) < 0) {
        // Handle error
    }

    // Add the class type to the module
    PyModule_AddObject(m, name, (PyObject*)pto);
}

[nrnhines]

A promising strategy was found at https://stackoverflow.com/questions/67077317/how-to-define-a-python-metaclass-from-c-extension and the answer referred to the repository https://github.com/ezyang/cpython-metaclass . I forked this started a hines/meta branch and added getitem methods to verify the solution might be sound in our situation. ie.

git clone git@github.com:nrnhines/cpython-metaclass meta
cd meta
git checkout hines/meta
pip install -e .
python test.py

The substance of test.py is

def p(s):
    print(s, eval(s))

import demo

fp = demo.fooparent()
p("fp")
p("type(fp)")
p("type(type(fp))")
p("type(type(type(fp)))")

p("fp[3]")
p("type(fp)[5]")

and the output is

$ python test.py
fp <demo.fooparent object at 0x78e1e0c942d0>
type(fp) <class 'demo.fooparent'>
type(type(fp)) <class 'demo.foometa'>
type(type(type(fp))) <class 'type'>
fooparent_getitem 3
<demo.fooparent object at 0x78e1e0c942d0>
fp[3] None
foometa_getitem 5
<class 'demo.fooparent'>
type(fp)[5] None

The question remaining is whether I can transform demo.cpp so that it uses the PyType_Slot and PyType_Spec style of class definition.

[nrnhines]

It was possible to (perhaps successfully) transform into the PyType_Spec style of class definition. Mostly by following some patterns found in nrn/external/nanobind. In particular grep -r -n nb_meta. The use and implementation of nb_type_from_metaclass is also interesting and explains that

static PyObject *nb_type_from_metaclass(PyTypeObject *meta, PyObject *mod,
                                        PyType_Spec *spec) {
#if NB_TYPE_FROM_METACLASS_IMPL == 0
    // Life is good, PyType_FromMetaclass() is available
    return PyType_FromMetaclass(meta, mod, spec, nullptr);
#else
    /* The fallback code below emulates PyType_FromMetaclass() on Python prior
       to version 3.12. It requires access to CPython-internal structures, which
       is why nanobind can only target the stable ABI on version 3.12+. */

Anyway, I added a hocclass and hocobject spec style implementation to git@github.com:nrnhines/cpython-metaclass branch hines/meta that attempts to produce identical behavior to foometa and fooparent in demo.cpp. Results are good with respect to

$ python test.py
fp <demo.fooparent object at 0x79974fe94300>
type(fp) <class 'demo.fooparent'>
type(type(fp)) <class 'demo.foometa'>
type(type(type(fp))) <class 'type'>
fooparent_getitem 3
<demo.fooparent object at 0x79974fe94300>
fp[3] None
foometa_getitem 5
<class 'demo.fooparent'>
type(fp)[5] None
ho <demo.hocobject object at 0x79974fe94310>
type(ho) <class 'demo.hocobject'>
type(type(ho)) <class 'demo.hocclass'>
type(type(type(ho))) <class 'type'>
hocobject_getitem 3
<demo.hocobject object at 0x79974fe94310>
ho[3] None
hocclass_getitem 5
<class 'demo.hocobject'>
type(ho)[5] None

Also, using gdb to print the class definitions give very similar but not identical results. For example it is not clear to me at the moment if the difference

(gdb) p fooparent_type
...
tp_flags = 5376,

and

gdb) p *(PyTypeObject*)custom_hocobject
...
tp_flags = 5632,

along with a few other differences, is relevant.