search

amazon-archives / logstash-input-dynamodb

This input plugin for Logstash scans a specified DynamoDB table and then reads changes to a DynamoDB table from the associated DynamoDB Stream.This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline. This gem is not a stand-alone program.
Apache License 2.0
105 stars 27 forks source link

logstash-input-dynamodb crashing logstash #15

Open gcarreto opened 8 years ago

gcarreto commented 8 years ago

Having a problem with plugin once an error ocurred with dynamodb. Dynamodb fails to retrieve a success response and after that logstash loses data. How can we avoid the lost of data? how can we implement a logic to help Kinesis to rollback to a previous sequence number range so it can go thru the data again and avoid loses.

Here is the error:

ERROR: com.amazonaws.services.kinesis.leases.impl.LeaseCoordinator: LeasingException encountered in lease renewing th read com.amazonaws.services.kinesis.leases.exceptions.DependencyException: com.amazonaws.AmazonServiceException: Unable to parse HTTP response content (Ser vice: AmazonDynamoDBv2; Status Code: 500; Error Code: null; Request ID: null) at com.amazonaws.services.kinesis.leases.impl.LeaseManager.convertAndRethrowExceptions(LeaseManager.java:558) at com.amazonaws.services.kinesis.leases.impl.LeaseManager.renewLease(LeaseManager.java:372) at com.amazonaws.services.kinesis.leases.impl.LeaseRenewer.renewLease(LeaseRenewer.java:106) at com.amazonaws.services.kinesis.leases.impl.LeaseRenewer.renewLeases(LeaseRenewer.java:85) at com.amazonaws.services.kinesis.leases.impl.LeaseCoordinator.runRenewer(LeaseCoordinator.java:199) at com.amazonaws.services.kinesis.leases.impl.LeaseCoordinator$RenewerRunnable.run(LeaseCoordinator.java:135) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: com.amazonaws.AmazonServiceException: Unable to parse HTTP response content (Service: AmazonDynamoDBv2; Status Code: 500; Error Code: null; Request ID: null) at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1219) at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:803) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:505) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:317) at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:1799) at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.updateItem(AmazonDynamoDBClient.java:1614) at com.amazonaws.services.kinesis.leases.impl.LeaseManager.renewLease(LeaseManager.java:363) ... 11 more log4j, [2016-02-16T02:13:11.953] ERROR: com.amazonaws.services.kinesis.leases.impl.LeaseCoordinator: LeasingException encountered in lease renewing thread com.amazonaws.services.kinesis.leases.exceptions.DependencyException: com.amazonaws.AmazonServiceException: Unable to parse HTTP response content (Service: AmazonDynamoDBv2; Status Code: 500; Error Code: null; Request ID: null) at com.amazonaws.services.kinesis.leases.impl.LeaseManager.convertAndRethrowExceptions(LeaseManager.java:558) at com.amazonaws.services.kinesis.leases.impl.LeaseManager.renewLease(LeaseManager.java:372) at com.amazonaws.services.kinesis.leases.impl.LeaseRenewer.renewLease(LeaseRenewer.java:106) at com.amazonaws.services.kinesis.leases.impl.LeaseRenewer.renewLeases(LeaseRenewer.java:85) at com.amazonaws.services.kinesis.leases.impl.LeaseCoordinator.runRenewer(LeaseCoordinator.java:199) at com.amazonaws.services.kinesis.leases.impl.LeaseCoordinator$RenewerRunnable.run(LeaseCoordinator.java:135) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: com.amazonaws.AmazonServiceException: Unable to parse HTTP response content (Service: AmazonDynamoDBv2; Status Code: 500; Error Code: null; Request ID: null) at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1219) at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:803) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:505) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:317) at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:1799) at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.updateItem(AmazonDynamoDBClient.java:1614) at com.amazonaws.services.kinesis.leases.impl.LeaseManager.renewLease(LeaseManager.java:363) ... 11 more