Apache ORC Support in TensorFlow IO

oliverhu commented 3 years ago

(Creating this issue for visibility so people interested can join the discussion... )

Overview

Load Apache ORC formatted data natively into TensorFlow from file system supported by TensorFlow, e.g. HDFS, local disk, etc.

Motivation

We traditionally use Avro to store our dataset but it is becoming inefficient to use row based format for big data analytics processing. Historically we selected ORC as our columnar storage format. (not planning to argue Parquet vs ORC here ;))

Design Discussions

Apache ORC would be brought in via https://github.com/bazelbuild/rules_foreign_cc
Feature wise, I expect the APIs to be similar to Parquet or Arrow reader.

Milestones

[x] Add Apache ORC build dependency.
[x] Implement a simple ORC dataset that maps records in ORC files into Tensors.
[x] add a tutorial for ORC reader.
[ ] feature schemas support: support sparseTensor and VarLenFeature.
[ ] feature schemas support: support denseTensor FixedLenFeature only. (follow parse_example_v2.)
[ ] usability improvements
[ ] performance tuning
[ ] feature schemas support: support raggedTensor

kvignesh1420 commented 3 years ago

@oliverhu any update on this?

oliverhu commented 3 years ago

no update recently @kvignesh1420

kvignesh1420 commented 3 years ago

@oliverhu can we document the current feature in the form of a tutorial?

oliverhu commented 3 years ago

sure, will add that !

kvignesh1420 commented 3 years ago

Reference FYKI: https://github.com/tensorflow/io/tree/master/docs/tutorials

372046933 commented 2 years ago

Is HDFS supported now? Loading from HDFS path results in coredump

dataset = tfio.IODataset.from_orc("hdfs://xxx/yy/iris.orc", capacity=15).batch(1)

372046933 commented 2 years ago

Is HDFS supported now? Loading from HDFS path results in coredump
dataset = tfio.IODataset.from_orc("hdfs://xxx/yy/iris.orc", capacity=15).batch(1)

HDFS supported (with kerberos) by https://github.com/tensorflow/io/pull/1674

tensorflow / io