Datanexus / kafka

kafka platform
0 stars 0 forks source link

support confluent connectors for apache install #16

Open christopherkeller opened 5 years ago

christopherkeller commented 5 years ago

confluent supports ES sinks with a library package. we should install this for apache mode by default.

https://www.confluent.io/hub/confluentinc/kafka-connect-elasticsearch

christopherkeller commented 5 years ago

Connector uploaded into S3 bucket, but need IAM work on each EC2 instance to make it seamless. listed are the manual steps after uploading into S3.

verify connector is running

On ansible host provision connectors as usual

AWS_PROFILE=slower ec2_region=us-east-2 ./deploy connect $platform_root/ucsd/inventory/hostsfile-us-east-2-demo.none demo aws us-east-2 development centos $platform_root/ucsd $key_path