search

apache / arrow-julia

Official Julia implementation of Apache Arrow
https://arrow.apache.org/julia/
Other
284 stars 59 forks source link

Support tensors/sparse tensor messages #187

Open quinnj opened 3 years ago

Tokazama commented 3 years ago

Are there any plans to support ND formats here or some way of formally writing custom metadata that supports reading a vector of data into an array structure?

ericphanson commented 3 years ago

writing custom metadata that supports reading a vector of data into an array structure?

LegolasFlux does this: https://github.com/beacon-biosignals/LegolasFlux.jl/blob/aaa94fdb050e0b5333725d7d968b58b080b67df2/src/LegolasFlux.jl#L16

I’m not really sure the practical difference between something like that and the tensor bit of the arrow spec (https://github.com/apache/arrow/blob/master/format/Tensor.fbs).

Tokazama commented 3 years ago

Thanks for the example. Perhaps it's a more effective use of developer time to heavily document how to write custom metadata? I'm trying to find ways to sell some other communities on this format instead of making more new file formats (an abominable practice that plagues neuroscience).

ToucheSir commented 3 years ago

I guess strides and better Interop would be the main differences? I imagine native tensor support would be nice for RPC as well.

Tokazama commented 3 years ago

I think formalizing size and strides would be a great thing to have in place. I imagine it would be best to write most other things as small dictionaries so that it can be read universally and then provide specific methods in a package for type stable reading into domain specific data structures.

Dsantra92 commented 2 years ago

Has there been any update on this?

Dsantra92 commented 2 years ago

I would like to have a go at supporting tensors in Arrow.jl.

BambOoxX commented 1 year ago

Any status about this ?

ericphanson commented 1 year ago

I don't think anyone is working on it

Dsantra92 commented 1 year ago

I volunteered to work on it. Thanks for the reminder, I will start working once I get some uni problems sorted (1-2 weeks).