awslabs / amazon-kinesis-client

Client library for Amazon Kinesis
Apache License 2.0
642 stars 465 forks source link

Updated documentation for the Multi-Lang Protocol #430

Open ahill-mlft opened 6 years ago

ahill-mlft commented 6 years ago

I'm working on a library to allow consumer applications written in Rust to run as children of the MultiLangDaemon, and I'm finding the documentation for the protocol somewhat challenging to work with.

As far as I know the best thing available on the public Internet is the package-info.java for the multilang package, but the last commit to this file was four years ago and it's missing details like the use of sequenceNumber and subSequenceNumber for checkpoint messages and the entire existence of the shutdownRequested message is undocumented. It's not always clear what parts of these messages are optional, or even what optionality looks like at the serialization level.

My current understanding of what's going on is mostly based on looking at the test suite for the Python KCL, but I took a look at the Node test suite today and that seems to be using a different format for checkpoint messages so I'd appreciate some guidance. I'll also pull together my notes and compile what I think I know about what is going on.

("Why are you linking to commits in the 1.x branch?" Because the multi-lang daemon in master seems to be deliberately disabled.)

pfifer commented 6 years ago

We're currently in the process of updating the MultiLang Daemon to the 2.0 version. We'll update the documentation for the message format at the same time.

zerth commented 6 years ago

@ahill-mlft: This may be of interest: https://github.com/AdRoll/erlmld/blob/master/src/erlmld_wrk_statem.erl

btel commented 5 years ago

I hit the same problem with the documentation. Is there any progress on this issue?