search

mongodb-labs / mongo-arrow

MongoDB integrations for Apache Arrow. Export MongoDB documents to numpy array, parquet files, and pandas dataframes in one line of code.
https://mongo-arrow.readthedocs.io
Apache License 2.0
92 stars 14 forks source link

Does mongo-arrow provide real zero copy in the chain mongodb->arrow->pandas? #176

Open sergun opened 1 year ago

sergun commented 1 year ago

.. or zero copy appear only between arrow->pandas but not here mongodb->arrow?

In other words are arrow data types used in mongodb?

ShaneHarvey commented 1 year ago

The MongoDB -> Arrow portion is not zero copy. This library translates (copies) MongoDB's BSON document result structure into Arrow's data format. There are no native Arrow types in MongoDB. Alternatively, the app can store Arrow data as opaque BSON binary data in MongoDB but then the app looses the ability to query the data.

sergun commented 1 year ago

The MongoDB -> Arrow portion is not zero copy. This library translates (copies) MongoDB's BSON document result structure into Arrow's data format. There are no native Arrow types in MongoDB. Alternatively, the app can store Arrow data as opaque BSON binary data in MongoDB but then the app looses the ability to query the data.

Thanks a lot! Do you know if there are plans to make something like Arrow backend in MongoDB to store all documents internally in Arrow structures as it was made in pandas 2.x and in Spark?

PS. My original question was caused by use case of batch processing of data stored in MongoDB by logic expressed in pandas. Try to find some light-weight alternative to popular solution like Spark+pandas_udf (which uses Arrow end-to-end: Spark->Arrow->pandas).

blink1073 commented 8 months ago

Hi @sergun, following up, no, there are no plans to change the storage format on MongoDB for alternate formats.