khwilson
commented
2 weeks ago I'm a long time fan of the arrow project and I was hoping to get a bit more involved in the project. I figured this might be a good first issue. Please let me know if it is not or the following is a bad idea.
Instead of something like a YAML or XML type, would a "rational" type make sense? Something like:
import pyarrow as pa
import pyarrow.types as pt
class RationalType(pa.ExtensionType):
"""
A rational number represented as a struct of an integer `numer` (the numerator)
and an integer `denom` (the denominator)
"""
def __init__(self, data_type: pa.DataType):
if not pt.is_integer(data_type):
raise TypeError(f"data_type must be an integer type not {data_type}")
super().__init__(
pa.struct(
[
("numer", data_type),
("denom", data_type),
],
),
# N.B. This name does _not_ reference `data_type` so deserialization
# will work for _any_ integer `data_type` after registration
"my_package.rational",
)
def __arrow_ext_serialize__(self) -> bytes:
# No serialized metadata necessary
return b""
@classmethod
def __arrow_ext_deserialize__(self, storage_type, serialized):
# return an instance of this subclass given the serialized
# metadata
return RationalType(storage_type[0].type)This shows off a few more of the parameters that are passed around than the current UUID example.
ianmcook
rok
sjperkins
Describe the bug, including details regarding any error messages, version, and platform.
In the Format docs and Python docs, there are several examples of user-defined extension types and sample code showing how to implement them (by subclassing). These all use a UUID extension type as the example:
Now that UUID is a canonical extension type (#41299) and will have native support in C++ and Python (#37298), we should replace these with examples based on some other user-defined extension type—ideally one that is not likely to become a canonical extension type anytime soon. Maybe an XML or YAML extension type (with UTF8 storage type)?
Component(s)
Documentation