search

nathanmarz / cascalog

Data processing on Hadoop without the hassle.
Other
1.38k stars 178 forks source link

Problem in directly querying data in pail structure #299

Open pankug opened 8 years ago

pankug commented 8 years ago

I am having a problem in Lambda Architecture, Our data stored in HDFS is in fact based pail format using Thrift Serialization schemes and vertical partitioning.

Is there any direct way we can query our data (residing in HDFS) in batch view, so we don't have to store our output data in ElephantDB or another database and we can directly view our data and stored it in a readable format.

We are facing this problem If u can help us regarding code structure and other techniques using any set of Big Data tool and language

sritchie commented 8 years ago

Directly query, as in do random access lookups on a key inside of HDFS? Not that I know of.

Sam Ritchie (@sritchie https://twitter.com/sritchie) Machine Learning @ Stripe https://stripe.com/ samritchie.io http://www.samritchie.io/ | 703.863.8561

On Jan 13, 2016, at 11:30 PM, Pankaj Joshi notifications@github.com wrote:

I am having a problem in Lambda Architecture, Our data stored in HDFS is in fact based pail format using Thrift Serialization schemes and vertical partitioning

Is there any direct way we can query our data (residing in HDFS) in batch view, so we don't have to store our output data in ElephantDB or another database and we can directly view our data and stored it in a readable format

We are facing this problem If u can help us regarding code structure and other techniques using any set of Big Data tool and language

— Reply to this email directly or view it on GitHub https://github.com/nathanmarz/cascalog/issues/299.