search

Bears-R-Us / arkouda

Arkouda (αρκούδα): Interactive Data Analytics at Supercomputing Scale :bear:
Other
239 stars 87 forks source link

Plasma Object Store Support #2385

Closed Ethan-DeBandi99 closed 1 month ago

Ethan-DeBandi99 commented 1 year ago

Implement support for writing and reading Arkouda objects to/from Plasma Object Store.

Ethan-DeBandi99 commented 1 year ago

As of Arrow 10.0.0, Plasma Object Store has been deprecated. It is scheduled for removal by Arrow 12.0.0. I will be looking into alternatives to achieve a similar outcome.

Ethan-DeBandi99 commented 1 year ago

@hokiegeek2 pointed me to a discussion of alternatives here. At first glance it appears our best option is Apache Flight. However, this is designed for server-client and may not perform as well as Plasma.

Another thing of note is that Plasma is not included in Apache Ray. It appears that this also supports a few features like distributed reference counting now that may be useful for Arkouda.