search

logstash-plugins / logstash-input-kinesis

Logstash Plugin for AWS Kinesis Input
Apache License 2.0
45 stars 57 forks source link

Cross Account DynamoDB and Kinesis Settings #100

Closed erickertzvfc closed 1 year ago

erickertzvfc commented 1 year ago

I have Logstash running in an EC2 instance on a completely different AWS account than Kinesis and cannot get this input plugin working. There are also no errors in the logs to begin debugging, which I saw there was already a separate open issue for. I suspect there is an issue with connecting to the DynamoDB table as it will fail on error connecting to kinesis on startup (which i have worked out those issues).

A couple of questions with this setup:

  1. Does the DynamoDB table need to exist in the same account as the Kinesis stream? If not and it looks for the table in the same account as the Logstash instance, how would I provide separate credentials? I am already using profiles if needed.
  2. Does the DynamoDB need anything additional setup for this to work or just a basic table?
  3. Is there any way to have more verbose logging?

I do have a DynamoDB table setup and matching application_name and the AWS creds are dynamodb: resource for testing.

Thanks in advance for any help.

erickertzvfc commented 1 year ago

i figured this out by using the role_arn setting. However, unfortuanetly I'm also dealing with an older version of logstash, 6.8 and it fails silently without creating the dynamodb table and in turn not pulling messages from kinesis. i tested with logstash 7.x and everything works as should.

erickertzvfc commented 1 year ago

im going to close this as 6.8 is at end of life support and updating to 7.x seems fine.

erickertzvfc commented 1 year ago

also, unrelated, but for anyone using this with opensearch as an output set ilm_enabled => false in your elasticsearch output to disable the 4xx calls to _xpack endpoints. so far, opensearch version of ilm still works with it disabled.