add conversion from arrow to tensor

gorgonia / tensor

package tensor provides efficient and generic n-dimensional arrays in Go that are useful for machine learning and deep learning purposes

Apache License 2.0

362 stars 49 forks source link

add conversion from arrow to tensor #75

Closed poopoothegorilla closed 4 years ago

poopoothegorilla commented 4 years ago

This PR adds the ability to convert an Arrow Array to a Dense Tensor.

Notes:

[x] Add Tests
[x] Adds Basic convesions for FromArrowArray function to Dense Tensor
[x] Adds ArrowToTensor Example
[x] Updates the Native Iterators to work with newer versions of Golang

coveralls commented 4 years ago

Coverage increased (+0.01%) to 73.178% when pulling 9752c8bed2fc304a661fab3b9eb3510b395abd02 on poopoothegorilla:jtw/arrow-to-tensor into cd60ada84b724da47b4740cba2a2d359835c1ded on gorgonia:master.

poopoothegorilla commented 4 years ago

I think a separate PR might be appropriate for adding Multidimensional support for Arrow. These are the tasks

[ ] Add ListArray Arrow Datatype to allow for multi dimensional conversions
[ ] Add Ability to create Multidimensional arrays without using the ListArray Arrow Datatype

I think it makes sensor to do a separate PR for TensorToArrow as well

cfgt commented 4 years ago

@poopoothegorilla Is this PR complete by itself? If so, I'll merge it. @chewxy is off the grid at the moment.

poopoothegorilla commented 4 years ago

@cfgt should be good now.