Closed ttaranov closed 7 years ago
Same here
I'm having the same error, any solution ?
I'm having the same error :-(
same error - replication does work but constantly getting this error. I think error is related to some kinesis dependency, as it got updated and didn't account for this library?
Getting the same error here too
Same here. After a couple of these errors the Java process dies silently. Can't get replication to work.
worked for me finally! use the following dependencies in build.sbt
"org.apache.spark" % "spark-streaming-kinesis-asl_2.11" % "2.1.0", "org.apache.spark" % "spark-streaming_2.11" % "2.1.0", "org.apache.spark" % "spark-core_2.11" % "2.1.0", "org.apache.spark" % "spark-sql_2.11" % "2.1.0"
I'm also seeing this issue.
@nandita1711 where have you put those dependencies? I cannot find build.sbt in the repo.
@dolphy17 if you have built your spark application in scala, you will have a build.sbt file where external dependencies are added. This is similar to pom.xml file in maven.
@nandita1711 any ideas for a fix in maven / pom.xml?
I'm pretty sure the code to push metrics to cloudwatch is broken, specifically https://github.com/awslabs/dynamodb-cross-region-library/blob/master/src/main/java/com/amazonaws/services/dynamodbv2/streams/connectors/DynamoDBReplicationEmitter.java#L411
Even if you update all the AWS versions in the pom.xml, you will still get the error.
If you update the bleeding edge, like this:
<aws.java.sdk.version>1.11.119</aws.java.sdk.version>
<amazon.kinesis-client-library.version>1.7.5</amazon.kinesis-client-library.version>
<amazon.kinesis-connectors.version>1.2.0</amazon.kinesis-connectors.version>
<dynamodb-streams-kinesis-adapter.version>1.1.0</dynamodb-streams-kinesis-adapter.version>
You will get compilation errors on that exact line (which is also throwing the exception).
Since it's broken, and I don't want a see a bazillion lines of the same exception in the logs, i just commented that line and rebuilt.
I wish I knew how to fix it, since it would be beneficial to have cloudwatch metrics, but I really don't know to fix it :(
Oh, it looks like someone created a working PR here: https://github.com/awslabs/dynamodb-cross-region-library/pull/35 Awesome!!!
This issue will be fixed by the SDK / KCL / Connector version update described elsewhere. Closing as a duplicate
I'm seeing a lot of stack traces around AmazonCloudWatch.putMetricData in logs while running this library:
steps to reproduce: clone this repo , mvn clean install cd target/ java -jar dynamodb-cross-region-replication-1.1.0.jar --sourceEndpoint https://dynamodb.us-east-1.amazonaws.com --sourceTable --destinationEndpoint https://dynamodb.us-west-2.amazonaws.com --destinationTable
replication does appear to work, however the kcl.log is full of these stack traces
2016-08-14 17:01:37,044 ERROR com.amazonaws.services.kinesis.metrics.impl.CWPublisherRunnable - Caught exception thrown by metrics Publisher in CWPublisherRunnable java.lang.NoSuchMethodError: com.amazonaws.services.cloudwatch.AmazonCloudWatch.putMetricData(Lcom/amazonaws/services/cloudwatch/model/PutMetricDataRequest;)Lcom/amazonaws/services/cloudwatch/model/PutMetricDataResult; at com.amazonaws.services.kinesis.metrics.impl.DefaultCWMetricsPublisher.publishMetrics(DefaultCWMetricsPublisher.java:63) at com.amazonaws.services.kinesis.metrics.impl.CWPublisherRunnable.runOnce(CWPublisherRunnable.java:144) at com.amazonaws.services.kinesis.metrics.impl.CWPublisherRunnable.run(CWPublisherRunnable.java:90) at java.lang.Thread.run(Thread.java:745)