search

awslabs / kinesis-kafka-connector

kinesis-kafka-connector is connector based on Kafka Connect to publish messages to Amazon Kinesis streams or Amazon Kinesis Firehose.
Apache License 2.0
153 stars 91 forks source link

VPC endpoint not supported #60

Closed dressst closed 3 years ago

dressst commented 3 years ago

We try to implement connector to forward data to AWS over private Kinesis / Firehose endpoint. As soon as we add our endpoint url to parameter "kinesisEndpoint", we get an connection error with following exeption:

com.amazonaws.services.kinesisfirehose.model.AmazonKinesisFirehoseException: Credential should be scoped to a valid region, not 'vpce'.  (Service: AmazonKinesisFirehose; Status Code: 400; Error Code: InvalidSignatureException; Request ID: d6f09f49-ff02-e2e9-8f2f-032e13758433; Proxy: null)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1811)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1395)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1371)

We can get it working by creating a local DNS record "firehose.eu-central-1.amazonaws.com" and let it point to one of the IP's of our VPC endpoint.

We also specify parameter "region" with correct region string. Seems this parameter is ignored if an endpoint parameter is provided and region is retrieved from endpoint Url. In case of the VPC from the wrong sub domain in Url.

Can connector be changed to also support VPC's?