Exporting to numpy & cloud file systems

jorgecarleitao / arrowdantic

Arrow, pydantic style

Other

82 stars 7 forks source link

Open benjaminrwilson opened 2 years ago

benjaminrwilson commented 2 years ago

Hi @jorgecarleitao,

Thanks for putting this library together -- it looks awesome! I had a few quick questions.

What is the easiest way to convert to / from numpy using arrowdantic?
Do you have any recommendations for reading Arrow files from cloud storage (e.g., s3 or gcs) that are backed by rust with python bindings?

jorgecarleitao commented 2 years ago

Hey @benjaminrwilson , sorry for the late reply, only now noticed this issue.

Currently we only support from/to iterators, so it would need to be through that. This is not very efficient, though.
The interfaces to read files support a Python file descriptor. So, use a Python lib to get that interface for the cloud storage (e.g. boto3's get_object) and pass it to the file reader.