search

wix-incubator / kafka-connect-s3

A Kafka-Connect Sink for S3 with no Hadoop dependencies.
Other
57 stars 45 forks source link

binary output format #6

Closed iamnoah closed 7 years ago

iamnoah commented 8 years ago

We have a topic that contains Thrift serialize messages, so we wrote a binary format. Since it can be hard to grok binary files, this PR also includes some basic helper classes for reading the files back out of S3.

This also adds support for arbitrary output formats by implementing the Connect Converter interface. The binary format is just one implementation, as is the default newline delimited format.

mcandre commented 8 years ago

+1

iamnoah commented 8 years ago

I realize this is a large PR, but can we get some sort of feedback? Is this project still going? Should we start our own fork for something like this?

muteor commented 8 years ago

Yeah I would go with a fork, we don't have bandwidth to deal with this currently and are not actually using this internally anymore.