search

awslabs / amazon-kinesis-client-python

Amazon Kinesis Client Library for Python
Apache License 2.0
370 stars 222 forks source link

Follow up question: how to use initialize_input parameter #86

Open jiawen-earnin opened 6 years ago

jiawen-earnin commented 6 years ago

@pfifer Thanks for your reply in this issue https://github.com/awslabs/amazon-kinesis-client-python/issues/81 . The document of AWS kinesis said that 'The KCL calls the initialize method when the record processor is instantiated, passing a specific shard ID as a parameter. This record processor processes only this shard' (https://docs.aws.amazon.com/streams/latest/dev/kinesis-record-processor-implementation-app-py.html). So if I develop my application in python, how can I scale it when there are large volume of data ?

pfifer commented 6 years ago

The KCL will pass in the shard that will be processed. The Java process that controls all of this uses threads and will start a new process for each acquired shard.