MultiLang Daemon for KCL 2.0

[pfifer]

We intend to update the MultiLang Daemon for KCL 2.0. As part of this update we are considering incorporating other enhancements like those previously requested. With this update all MultiLang clients (amazon-kinesis-client-python, amazon-kinesis-client-nodejs, amazon-kinesis-client-ruby, amazon-kinesis-client-net) are likely to be released with a new major version.

If you have a feature or change request please comment on this issue using the template below. If a request is not included in the initial release we will prioritize it accordingly for future releases. If you're responding to an existing proposal please reference the specific post.

Please add reactions to any of the features or changes proposed, this will aid in prioritizing requests.

### Short title for the requests
A short description of the requests

#### Background or information about the request
This can be longer, and go into more depth.   
This section can help us and others understand why the request is important. 

[EdgarOrtegaRamirez]

Redirect normal output to stdout and errors to stderr

Background or information about the request

Maybe I'm ignoring something, but it seems to me that all output is sent to stderr, I would like to be able to distinguish between them.

[EdgarOrtegaRamirez]

Fanout

Background or information about the request

I would like to start implementing this

[dkimdon]

DynamoDB Streams support for amazon-kinesis-client-nodejs

I want to be able to easily process DynamoDB Streams records with nodejs

Background

There is an existing pull request: https://github.com/awslabs/amazon-kinesis-client-nodejs/pull/11

In production I currently use that pull request and the version of aws-kcl it is based on to process DynamoDB streams. Where I am processing actually Kinesis streams I use a different version of aws-kcl. It would simplify my deployment if I could use the same aws-kcl version for both DynamoDB Streams and Kinesis record processing. Thanks.

[zerth]

Don't terminate all siblings if a child ungracefully exits

If a shard worker process exits due to an error, the MLD will terminate all of its siblings. This is inconvenient, and should be unnecessary since processing of different shards should be independent.

Background

The only means for a shard worker to indicate a fatal error is to exit. This can be useful for retryable errors in processing (e.g., worker getting into some bad state), with permanent/unretryable error handling being up to the application (e.g., skipping certain records or awaiting operator intervention/bugfixing). Limiting the blast radius of an erroring-out worker would prevent other unrelated shards from being affected.

[zerth]

Reduce INFO-level log spam

Too much logspam is emitted at INFO level.

Background

A number of KCL/MLD log messages are emitted at INFO level (or without a stated level), but look like diagnostic/debug messages. Examples:

INFO: Successfully set property shardSyncIntervalMillis with value 60000
INFO: MultiLangDaemon is adding the following fields to the User Agent: amazon-kinesis-client-library-java-1.9.1 amazon-kinesis-multi-lang-daemon/1.0.1 erlang erlmld.sh
INFO: Initialization attempt 1
INFO: Created new shardConsumer for : ShardInfo [shardId=shardId-000000000308, ...
com.amazonaws.services.kinesis.multilang.MultiLangProtocol validateStatusMessage ...

Some of these are relevant to an application's normal operation; others such as at least some of the above aren't.

[EdgarOrtegaRamirez]

Reduce Memory Usage

Background or information about the request

single shard ruby process, reading from and stream and storing in database without any additional validation

[adunham-stripe]

MultiLang Daemon for Go

I'd love to have a MultiLang Daemon client library for Go.

Background or information about the request

This can be useful in the following cases:

• Where the rest of a codebase is already in Go
• Where processing a message is expensive, so you want to use a faster language (and can't e.g. use numpy in Python, etc.)
• Where you care about memory usage; the KCL is pretty memory-hungry (see above), and adding a Ruby or Python VM on top of that isn't always necessary.

[dhessler]

@pfifer Is there a roadmap/timeline for the KCL 2.0 client release(s)?

[kdedon-socialware]

Configuration loaded from ENV

Allow configuration to be specified through environment variables in addition to the .properties file.

Background or information about the request

Our deployment pipeline involves building ECS tasks that we share between a lab and various production environments. Using the environment to specify configuration allows us to share containers between environments and maintain simplicity of deployment.

[casret]

I know it's a little late for the 2.0, but this is more of a 3.0ish feature request anyway.

Better RPC mechanism

The MLD should expose a websocket, or speak REST to its worker processes.

Background or information about the request

The stdin/stdout protocol is brittle, underdocumented and makes debugging unnecessarily hard. For instance in the node client, without monkey patching you can't do any kind of console logging (which includes any kind of library you are pulling in). For the 4 languages you are targeting, it should be easy to either establish a websocket with the MLD, or if you want to go the other way, require the workers to implement a REST interface via sinatra/express/similar and the MLD can hit it. You could then document it with standard tools, and you would see many (any) more third party client implementations.

[AquaNerd]

Is there a roadmap for when the .net KCL 2.0 feature will be done? I would love to use .net core to use enhanced fanout but it is my understanding you have to be using the KCL 2.0 which isn't yet available for .net.