Amazon Kinesis Storm Spout
The Amazon Kinesis Storm spout helps Java developers integrate Amazon Kinesis with Storm.
Requirements
Overview
The Amazon Kinesis Storm spout fetches data records from Amazon Kinesis and emits them as tuples. The spout stores checkpoint state in ZooKeeper to track the current position in the stream.
The Amazon Kinesis Storm spout can be configured to retry failed records. By default, it retries a failed record 3 times. If a record fails and the retry limit has been reached, the spout will log an error and skip over the record. The spout buffers pending records in memory, so it can re-emit a failed record without having to re-fetch the record from Amazon Kinesis. The spout sets the checkpoint to the highest sequence number that has been ack'ed (or exhausted retry attempts).
To use the spout, you'll need to add it to your Storm topology.
- KinesisSpout: Constructs an instance of the spout, using your AWS credentials and the configuration specified in KinesisSpoutConfig (as well as com.amazonaws.ClientConfiguration, via the AWS SDK). Each task executed by the spout operates on a distinct set of Amazon Kinesis shards. Shard states are periodically committed to ZooKeeper. When the spout is deactivated, it will disconnect from ZooKeeper, but the spout will continue monitoring its local state so you can activate it again later.
- KinesisSpoutConfig: Configures the spout, including the Storm topology name, the Amazon Kinesis stream name, the endpoint for connecting to ZooKeeper, and the prefix for the ZooKeeper paths where the spout state is stored. See the samples folder for configuration examples.
- DefaultKinesisRecordScheme: This default scheme, used by the sample topology, emits a tuple of
(partitionKey, record). If you want to emit more structured data, you can provide your own implementation of IKinesisRecordScheme.
The samples folder includes a sample topology and sample bolt, using the number of Amazon Kinesis shards as the parallelism hint for the spout. For more information about Storm topologies and bolts, see the Storm documentation.
Using the Sample
- Edit the *.properties file to configure your Storm topology, Amazon Kinesis stream, and ZooKeeper details. For your AWS Credentials, we recommend using IAM roles on Amazon EC2 when possible. You can also specify your credentials using system properties, environment variables, or AwsCredentials.properties.
- Package the spout and the sample (including all dependencies but excluding Storm itself) into one JAR file.
- Deploy the package to Storm via the JAR file, e.g.,
storm jar my-spout-sample.jar SampleTopology sample.properties RemoteMode
Release Notes
Release 1.1.1 (June 1, 2015)
- When a Kinesis Stream is resharded, "storm rebalance" can be invoked to refresh the shard list and distribute the latest shards across the Spout tasks.
Release 1.1.0 (October 21, 2014)
- Added support for retrying failed records
- Added region name support
Future Work
- Automatically handle closed, split, and merged shards.
Related Resources
Amazon Kinesis Developer Guide
Amazon Kinesis API Reference
Amazon Kinesis Client Library
Amazon Kinesis Connector Library