search

IBMStreams / administration

Umbrella project for the IBMStreams organization. This project will be used for the management of the individual projects within the IBMStreams organization.
Other
19 stars 10 forks source link

Proposal: Apache Avro toolkit for SerDe #102

Closed fketelaars closed 7 years ago

fketelaars commented 7 years ago

Apache Avro is a popular format for serializing messages with variable columns and evolving structures. Proposing toolkit to serialize a tuple to Avro format and deserialize from Avro to a Streams tuple. Avro supports multiple languages, among other C++ and Java. Initially proposing to create a toolkit in Java with 2 main operators: AvroToTuple and TupleToAvro.

mikespicer commented 7 years ago

+1 I believe you have seen the AVRO operators in the storm benchmark which may or may not be useful https://github.com/IBMStreams/benchmarks/tree/master/StreamsAvroOperators

chanskw commented 7 years ago

+1

@mikespicer I have previously looked at the Avro operators before to try to pull that out. But those operators seem very specific to the email benchmark. I think we need something more generic.

mikespicer commented 7 years ago

@chanskw OK, I did not look at them closely but its not surprising that they were implemented for the specific needs of the benchmark and not made to be generic.

chanskw commented 7 years ago

@fketelaars are you the only committer to this project? Who else should be committer ?

fketelaars commented 7 years ago

Please add @jchailloux as well

chanskw commented 7 years ago

Done!

fketelaars commented 7 years ago

Thanks!