Support exporting Chapel lists as Python lists

[ben-albrecht]

Currently, Chapel arrays export as numpy ndarrays. This makes sense, because ndarrays can be multidimensional like Chapel arrays (among other reasons). This is a feature request to support exporting Chapel lists to Python lists. This also makes sense, because both lists only support 1 dimension and are stored as dynamic arrays.

Below is a few examples demonstrating this proposal:

// exports a python list(str)
export proc foobar(): list(string) {
  var A: list(string);
  A.append('foo');
  A.append('bar');
  return A;
}

// exports a python list(int)
export proc foobar(): list(int) {
  var A: list(int);
  A.append(1);
  A.append(2);
  return A;
}

// and so on for other supports types...

[bradcray]

Since (as I understand it) Python lists are stored monolithically in memory, I wonder whether we should implement the similarly contiguous-in-memory based list implementation that we've talked about in Chapel and have that interoperate with Python lists as a first step. Otherwise, is the implication that we'd have to do a copy of the list to get it into contiguous memory for Python? (where I'm hoping the Python list could potentially refer to the same memory as the Chapel list if they used similar memory layouts?).

[ben-albrecht]

I wonder whether we should implement the similarly contiguous-in-memory based list implementation that we've talked about in Chapel

Is there an issue documenting this that we could reference?

Otherwise, is the implication that we'd have to do a copy of the list to get it into contiguous memory for Python?

I think so, but referring to the same memory sounds ideal to me, if possible.

[bradcray]

Is there an issue documenting this that we could reference?

Not that I'm aware of.

[dlongnecke-cray]

Correct me if I'm wrong @lydia-duncan, but I think we copy allocate a new numpy array when converting a Chapel array to a numpy array today, regardless of the element type.

[bradcray]

I think we copy when converting an array to a numpy array today, regardless of the element type.

That sounds familiar, but we've discussed wanting to get out of that business as well I think...

[lydia-duncan]

That's correct, and we do want to stop doing so. I'm okay with not addressing that for this change and handling it more wholesale, especially if we have someplace tracking the spots that should be updated to not copy (we may, but I don't remember off the top of my head)

[dlongnecke-cray]

I'm not entirely sure if it is possible to avoid copying memory when exporting a Chapel array unless we had some means of i.e. calling chpl_mem_free when the numpy.ndarray finalizer is called. It's definitely something I could make a spike for, though.

I think our best chance of supporting types like list in Python interop without deep copying across heaps is #14420, because the currently proposed strategy there would effectively allocate all memory on the Chapel heap (in single locale interop), while the Python wrapper object would be responsible for freeing the memory.