Role of ReactoKinesix vs. Kinesis Client Library

luck02 commented 8 years ago

Hi, wondering what the long term plan for RX is in terms of the Kinesis Client library for .net.

I don't think anyone is overly attracted to taking the weird life cycle dependency on the java runtime, but it's the 'official' method and I'd be frustrated if I started using Reacto and it went dormant 3 weeks later.

Thanks allot,

Gary Lucas

luck02 commented 8 years ago

Sorry, I guess another way to ask the same question would be to say: What is the problem you're trying to solve vs. using the KCL (not that I'm a fan of the KCL mechanism).

theburningmonk commented 8 years ago

Hi, I started this project around the time Amazon announced Kinesis, I was really excited to try it out only to find everything the documentation referred to was Java only, and there was no KCL for any other language. Which was why I started this project and the hope was (still is!) that there will come a day when there'd be an official .Net client library would render this library unnecessary.

(on a side note, I also created a producer library that supports a number of things:

auto-splitting shards based on throttle rate
sync and async send (latency is still an issue, though it's much better than the 100ms per record when the service first came out which really necessitated the need for sending stuff in the background)
back-pressure handling
additional metrics in CloudWatch (which they seem to have caught up, mostly))

Like you, I'm not a fan of the .Net KCL as it stands right now, so as things stand I will still continue to maintain this library (we also have a bunch of production code depending on it too). But your point is totally valid, this is an open-source project with only one active maintainer, so there's definitely risk involved. In the short-term at least I don't see any reason for me to abandon this project though.

The way I see, the bulk of the work is in the processing logic anyway, so even if I was to stop working on this library it's not a big ask to move to the KCL. I actually borrowed a lot of design idea from the KCL at the time, including how it uses DynamoDB to track state of the stream processors, so from a consumer's point-of-view it should be easy to switch.

luck02 commented 8 years ago

That makes sense, thanks for the overview. Much appreciated.

I'm experimenting with ripping an existing process that's currently living on a 0MQ bus and running it entirely on Kinesis. I will investigate Darkseid as it looks like the Kinesis Producer Library is again, java only. I have a ton of background with C# and none with F# so it's making for some fun reading :)

theburningmonk / ReactoKinesix

Role of ReactoKinesix vs. Kinesis Client Library #57