search

apache / druid

Apache Druid: a high performance real-time analytics database.
https://druid.apache.org/
Apache License 2.0
13.52k stars 3.71k forks source link

Implement the KPL Aggregated Record Format for Kinesis indexing service #6714

Closed dclim closed 4 years ago

dclim commented 5 years ago

The Kinesis indexing service (#6431) supports de-aggregation of multiple rows packed into a single record by the Kinesis Producer Library's aggregate method for more efficient data transfer. Currently, enabling the de-aggregate functionality requires the user to manually provide the Kinesis Client Library on the classpath, since this library has a license not compatible with Apache projects.

See: https://github.com/apache/incubator-druid/pull/6431/files#diff-88b66c67f91d2ab9d26a23abf0543bfbR312

A better solution would be for the Druid project to implement the protocol ourselves according to the specification here: https://github.com/awslabs/amazon-kinesis-producer/blob/master/aggregation-format.md

For legal reasons, this should be done by a contributor who has not read through the Amazon licensed implementation so that there is no case for copyright infringement (https://en.wikipedia.org/wiki/Clean_room_design).

justinborromeo commented 5 years ago

Currently working on this.

himanshug commented 5 years ago

https://lists.apache.org/thread.html/95aec3e271903320994fba321c58aff66b739b1f68796c253e430469@%3Cdev.druid.apache.org%3E means we can now package the AWS lib itself instead of implementing the protocol ?

justinborromeo commented 5 years ago

@himanshug yes. Closing #6917 since it's no longer necessary.

stale[bot] commented 4 years ago

This issue has been marked as stale due to 280 days of inactivity. It will be closed in 4 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the dev@druid.apache.org list. Thank you for your contributions.

stale[bot] commented 4 years ago

This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time.