feat(python) : Added support to create timestamp/date32/date64 Array from iterable

[aosingh]

Addresses https://github.com/apache/arrow-nanoarrow/issues/478

Currently, if you create a c_array from an iterable of timestamps an error is raised

Code:

from datetime import datetime

import nanoarrow as na
import pyarrow as pa

def gen_timestamp():
    for i in range(100):
        yield int(round(datetime.now().timestamp()))

timestamp_array = na.c_array(gen_timestamp(), na.timestamp("s"))

parray = pa.Table.from_arrays([timestamp_array], names=["timestamp"])

print(parray.to_pandas())

Error:

Traceback (most recent call last):
  File "/Users/as/nanoarrow_poc/.venv/lib/python3.12/site-packages/nanoarrow/c_array.py", line 128, in c_array
    builder = builder_cls(schema)
              ^^^^^^^^^^^^^^^^^^^
  File "/Users/as/nanoarrow_poc/.venv/lib/python3.12/site-packages/nanoarrow/c_array.py", line 457, in __init__
    raise ValueError(
ValueError: Can't build array of type timestamp from iterable

Expected output:

             timestamp
0  2024-05-31 18:43:10
1  2024-05-31 18:43:10
2  2024-05-31 18:43:10
3  2024-05-31 18:43:10
4  2024-05-31 18:43:10
..                 ...
95 2024-05-31 18:43:10
96 2024-05-31 18:43:10
97 2024-05-31 18:43:10
98 2024-05-31 18:43:10
99 2024-05-31 18:43:10

[100 rows x 1 columns]

Benefits

• Pyarrow Table and Pandas Dataframe can use the corresponding nanoarrow Array Type to cast the values when using to_pandas

[aosingh]

Thank you, I will go through the review comments and also check why some unit tests failed

[aosingh]

@paleolimbot

Could you please help with the GitHub CI workflow ? I wanted to verify the latest changes and unit tests. Thanks.

[paleolimbot]

Thanks for the additional changes! I was out-of-office today but will take a look tomorrow morning 🙂

[aosingh]

There are a few style guide check failures and 1 unit test failed. I am checking this.

[paleolimbot]

Did you know about pre-commit run --all-files? (It should take care of the style checks for you)

[aosingh]

Thanks, pre-commit run --all-files formatted the code as per style guide. I have also modified a unit testcase to use UTC timezone in timestamps.

[aosingh]

@paleolimbot

Please review the updates. I have simplified unit tests. Hopefully, the test cases are clear now. Let me know your feedback.

[aosingh]

I have added CSchemaView.storage_buffer_format and updated unit tests. I will further review it tomorrow.

[aosingh]

@paleolimbot

I have updated the PR based on the review comments.

To summarize:

• Creation of c_array depends on schema's storage_buffer_format
• The property storage_buffer_format either returns the existing buffer_format or the underlying buffer format corresponding to the logical data type
• Added a couple of test cases to create c_array of timestamps from PyBuffer

Please review and let me know your feedback

[aosingh]

@paleolimbot

Sure, no problem, thank you for your patience.

na.c_buffer([d1, d2, d3], na.timestamp("ms")) raises ValueError. The unit tests are modified to verify that.

with pytest.raises(ValueError):
     na.c_buffer([d1, d2, d3], na.timestamp("ms"))

is this not expected ?

[paleolimbot]

Sorry, I missed that! Thank you!

[aosingh]

@paleolimbot

I see some unit tests for Windows has failed after merge. Let me try to understand the reason