search

amazon-archives / amazon-kinesis-connectors

Apache License 2.0
326 stars 190 forks source link

amazon redshift connector deployment #38

Open mtrienis opened 9 years ago

mtrienis commented 9 years ago

Hi All,

I've managed to get the amazon redshift connector running locally on my virtual machine, however, I would like to use this library as part of our production deployment.

Does anyone have an example of what the deployment topology would look like? For example, what is the throughput to Redshift when running this library on single EC2 instance? Does it need to be deployed using a distributed mechanism? If it can be distributed, will it integrate with Apache Spark (Streaming)?

Thanks in advanced for any help, Mike.

IanMeyers commented 9 years ago

Hello,

You might find that this article from the AWS Big Data Blog (http://blogs.aws.amazon.com/bigdata/post/Tx2AS1MWDUK5DUK/Hosting-Amazon-Kinesis-Applications-on-AWS-Elastic-Beanstalk) gives you an example about how you can run Kinesis Applications in a highly available, Multi-AZ topology without significant coding. Throughput for loading to Redshift will be more a function of how quickly your application can write files to S3, and can be affected by the EC2 Instance Type for the Kinesis Application, as well as by how many Shards you have in the Kinesis Stream. Kinesis can certainly integrate with Spark Streaming (http://spark.apache.org/docs/1.2.0/streaming-kinesis-integration.html).

HTH,

Ian