search

emer / etable

Data table structure in Go, now developed at https://github.com/cogentcore/core/tree/main/tensor
BSD 3-Clause "New" or "Revised" License
117 stars 7 forks source link

Feature Request: Parquet File Support #48

Open tgruben opened 2 years ago

tgruben commented 2 years ago

Any hope of getting Parquet file support anytime soon or is CSV going to remain the format of choice

rcoreilly commented 2 years ago

I'm guessing that Apache Arrow implements Parquet, and the etable.Tensor is Arrow compatible, so in principle this could be easy.. :)

tgruben commented 2 years ago

hmm, it appears etensor.Tensor doesn't fully implement arrow's tensor.Interface. Missing Retain() and Release() i think.

https://pkg.go.dev/github.com/apache/arrow/go/v10@v10.0.0-20220812184426-39a2f3beba90/arrow/tensor#Interface

gedw99 commented 1 year ago

@tgruben that link is taking ages to load. Here is the direct code link

https://github.com/apache/arrow/tree/main/go/arrow/tensor

gedw99 commented 1 year ago

@tgruben and @rcoreilly

Its might make sense to use DataFusions as the Data store backing all this. That means that S3 is the backing store.