search

nathanmarz / dfs-datastores

Dead-simple vertical partitioning, compression, appends, and consolidation of data on a distributed filesystem.
BSD 3-Clause "New" or "Revised" License
215 stars 82 forks source link

Allow custom file formats (aside from SequenceFileFormat) to pass args #33

Open baconz opened 11 years ago

baconz commented 11 years ago

I'm working on an AvroFileFormat, and I'd like to be able to pass it args -- this seemed like the least bad way to make it happen. Open to suggestions for other approaches.

sritchie commented 11 years ago

@sorenmacbeth, all you!

sorenmacbeth commented 11 years ago

Looks good at first glance to me. I'll give a more thorough going over soon. Thanks!

cmerrick commented 11 years ago

I ran into this same issue when trying to create a custom Avro file format that keeps track of the schema in args. +1 for this!

kul commented 11 years ago

:+1: for the well thought change. @baconz @cmerrick how do you get the arguments inside the serialize and deserialize methods in your implemented PailStructure? I am not able to figure this out.

stanfea commented 9 years ago

@baconz Did you get AvroFileFormat working?

I've made a pail structure that serializes and deserializes to from avro but it's stored in a SequenceFile and I'm getting poor performance from the pail tap this way. Thanks

baconz commented 9 years ago

@steanbacon we never did. More, or less abandoned this project.