Python interop: Expose Chapel records as Python objects

[ben-albrecht]

Feature request: Expose Chapel records as python objects.

Today, my process for exposing Chapel records (and classes) is very tedious. The effort required to wrap and object scales with O(methods+fields). For some insight, my current process is shown below. Suggested improvements are welcome.

Chapel side

Start with a record to wrap:

record Object {
  var value: int;
  proc init(...) { }
  proc foo(...) { }
}

Store a list of objects created on the Chapel side:

var objects: list(Object);

Write wrappers for initializer (and deinitializer for classes), which takes an index the object being created/destroyed:

export proc ObjectInit(...) {
  // Convert C types to Chapel types
  //  e.g., create strings with new buffers from c_strings
  objects.append(new Object(...));
}

Write wrapper function for every public method, with an extra index field to specify which object:

/* Wrap method Object.foo(...) */
export proc objectFoo(idx: int, ...) {
  // Convert C types to Chapel types
  objects[idx].foo(...);
  // Convert Chapel types to C types (if we're returning something)
}

/* Wrap field Object.value reader */
export proc objectGetValue(idx: int): int {
  return objects[idx].value;
}

/* Wrap field Object.value writer */
export proc objectSetValue(idx: int, value: int): int {
  objects[idx].value = value;
}

Python side

Abstract away the index-tracking and any python <-> C type conversions on the python side:

# Chapel library
import libObject

_global_object_index  = 1

class Object(object):

  def __init__(self, **kwargs):
    global _global_object_index
    # Convert strings to byte strings
    for key in kwargs:
      if type(kwargs[key]) is str:
        kwargs[key] = str.encode(kwargs[key])

    libObject.ObjectInit(**kwargs)
    self.index = _global_object_index
    _global_object_index += 1

  def foo(self, **kwargs):
    # Convert strings to byte strings
    for key in kwargs:
      if type(kwargs[key]) is str:
        kwargs[key] = str.encode(kwargs[key])

    libObject.ObjectFoo(self.index, **kwargs)

  #
  # Currently using explicit get/set methods for accessing fields
  #

  def setValue(self, **kwargs):
    ...

  def getValue(self):
    ...

[lydia-duncan]

I don't know of a better way at the moment. This looks fairly close to what I would expect the compiler to do for you when we support exporting records

[dlongnecke-cray]

[Repost from https://github.com/Cray/chapel-private/issues/651]

Some users have expressed the desire to be able to export records and their methods when compiling a library (by way of the export keyword).

export record foo {
    var x: int = 0;

    proc init() { x = 42; writeln("Hello!"); }
    proc deinit() { writeln("Goodbye!"); }
    proc doSomething() { writeln("Something"); }
}

And be able to interact with it from Python as though it were a native Python object:

# This is code generated for Cygwin...
class foo:
    def __init__(self):
         # Somehow, heap allocate memory for the record instance on the Chapel heap.
         self.handle <*foo> = library.chpl_foo_heap_allocate()
         # Call the appropriate initializer in user code.
         library.chpl_init_foo_0(self.handle)
         # Any other setup...

    # This is called at some point by the Python GC when this object has a refcount of 0.
    def __del__(self):
         # Call the appropriate deinitializer in user code.
         library.chpl_deinit_foo(self.handle)
         # Somehow, deallocate memory on the Chapel heap.
         library.chpl_foo_heap_deallocate(self.handle <*void>)
         # Any other last minute accounting.

    # Still need to work out how getters/setters will work?
    def getField(self, field): pass
    def setField(self, field, value): pass

    def doSomething(self): pass

---

There are some design questions that need to be ironed out first, though:

1) When you export a record, what methods are exported? All of them? None of them? Initializers/deinitializers?

• Initially, I supposed that all methods be exported, but that prevents data hiding/encapsulation altogether.
• A record that is not POD is not very useful/functional without initializer/deinitializer routines.
• This might where public/private step in to save the day, but unfortunately those keywords are not functional yet for methods.

2) How should fields be accessed? Getters/setters? Should they be accessed directly in languages that support it?

• If getters/setters should be used, what should the naming convention for getter/setter symbols look like? IE, getField_x(), or in languages with dynamic typing, getField('x')?

• If getter/setters should be used, how should we deal with symbol collisions?

3) What should our memory strategy look like?

• Allocate a "list" of objects internally on the Chapel side, and have Python wrappers access their memory via that object index. This is the approach used by @ben-albrecht for HPO.
• Generate an "allocation" routine for each exported record which returns a handle to heap allocated memory for that object automatically. Wrapper objects generated for Cython/Python would call this and then use the handle.
• Use an even lower level approach and just chpl_malloc/chpl_free the memory required for the object directly using the compiler calculated size.
• Generate a "class wrapper" around each exported record and then just call the compiler generated new routine for the wrapper.

[ben-albrecht]

3. What should our memory strategy look like?

Allocate a "list" of objects internally on the Chapel side, and have Python wrappers access their memory via that object index. This is the approach used by @ben-albrecht for HPO.

@dlongnecke-cray - here is an example of using pointers as integers to represent a Chapel object, which can be passed between Chapel and Python to represent the Chapel-side objects:

https://github.com/chapel-lang/chapel/issues/14477#issuecomment-556090454

This may be a better approach than the manual index counting approach used in HPO.

[lydia-duncan]

Note that we had a user ask for the ability to export a type today in chat