search

apache / arrow

Apache Arrow is a multi-language toolbox for accelerated data interchange and in-memory processing
https://arrow.apache.org/
Apache License 2.0
14.28k stars 3.47k forks source link

[Python] Promote usage of the Arrow PyCapsule Protocol (for the C Data Inteface) #39195

Open jorisvandenbossche opened 9 months ago

jorisvandenbossche commented 9 months ago

https://github.com/apache/arrow/pull/37797 added a Python layer for working with the C Data Interface through capsules and defined dunder methods, described at https://arrow.apache.org/docs/dev/format/CDataInterface/PyCapsuleInterface.html.

Currently, libraries are using pyarrow's _export_to_c/_import_from_c semi-private methods to get the C Data Interface structs / construct pyarrow object from the structs. With the formalized PyCapsule protocol, the idea is that this communication of the structs now happens with defined dunder methods __arrow_c_schema__ / __arrow_c_array__ / __arrow_c_stream__ to get the data as a C Data Interface struct wrapped into a capsule instead as raw integer (export). In addition, the pyarrow constructors (pa.array(..), pa.table(..), ..) will now also accept any object implementing the relevant dunder method (import).

This new Python-level interface has the benefits to be 1) more robust (not passing around pointers as integers, the capsule will ensure the data is released when an error would happen in the middle), and 2) not tied to pyarrow the library. This means that other libraries can also start accepting any Arrow-compatible data structure, instead of harcoding support for pyarrow and using pyarrow's _export_to_c.

We want to promote other libraries to start supporting this protocol as well, which can mean:

Linking some issues here: https://github.com/pola-rs/polars/issues/12530, https://github.com/data-apis/dataframe-api/issues/279, https://github.com/pandas-dev/pandas/pull/56587, https://github.com/OSGeo/gdal/pull/9043 and https://github.com/OSGeo/gdal/issues/9132

In addition, there are also some steps we can take within the Arrow project itself to further promote this:

There are also some other follow-up discussions about certain aspects of the protocol or APIs we provide in pyarrow:

pitrou commented 9 months ago

cc @wjones127

kylebarron commented 7 months ago

I made a feature request for duckdb here: https://github.com/duckdb/duckdb/discussions/10716

kylebarron commented 1 month ago

I've been working a bit to promote the protocol; here's a running tally:

implemented: (at least partially, return objects with pycapsule protocol and/or check for existence of protocol in constructors)

in-progress:

issues: