awslabs / amazon-kinesis-client-python

Amazon Kinesis Client Library for Python
Apache License 2.0
368 stars 223 forks source link

Using Python KCL library for mutli-stream processing #172

Open DewanChaulagain opened 2 years ago

DewanChaulagain commented 2 years ago

I am using amazon-kclpy==2.0.6 to process kinesis streams. This works well for single stream processing. From the Java library, I see that multi-stream support was introduced in the java library version 2.3.0. Also worth noting that amazon-kclpy v2.0.6 uses Amazon Kinesis Client v2.3.9 as dependency.

The official docs released with the Java library suggests

With this new capability, you can update the list of streams at runtime for multi-stream processing in a scalable KCL application without redeploying the application

My use case requires one application to listen/process multiple Kinesis streams. As the stream names are static, could even provide the stream names as part of the configuration itself.

The issue:

As the python client interacts with the Java Multilangdaemon, we have the ability to provide the configuration file to the MultilangDaemon. Yet :

  1. We cannot provide multiple stream names ( a list of names) as the streamName in properties file. This field is expected to be a string.
  2. We cannot provide two properties file ( with each file set up with different streamName to the Multilangdaemon.

How can we configure the KCL Python to use multiple stream? Thank you!

harrydaniels-IW commented 1 year ago

I also need this, are there any plans to update KCL Library to support multiple streams?