Closed anand-chandrashekar closed 2 months ago
Hey @anand-chandrashekar, could you please try replacing the endpoint
with just the region
in the config, as follows?
source:
type: dynamodb
table: anand-mig-1
- endpoint:
- host: dynamodb.us-east-1.amazonaws.com
- port: 8000
+ region: us-east-1
credentials:
…
Hi @julienrf I got a different error. cc: @gcarmin
Exception in thread "main" com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: The security token included in the request is invalid. (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: UnrecognizedClientException; Request ID: HQS4CFKMN28L2FSUKQQCVUQG4VVV4KQNSO5AEMVJF66Q9ASUAAJG)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1799)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1383)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1359)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1139)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:796)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:764)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:738)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:698)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:680)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:544)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:524)
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:5110)
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:5077)
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.executeDescribeTable(AmazonDynamoDBClient.java:1981)
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.describeTable(AmazonDynamoDBClient.java:1947)
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.describeTable(AmazonDynamoDBClient.java:1993)
at com.scylladb.migrator.readers.DynamoDB$.readRDD(DynamoDB.scala:52)
at com.scylladb.migrator.readers.DynamoDB$.readRDD(DynamoDB.scala:19)
at com.scylladb.migrator.alternator.AlternatorMigrator$.migrate(AlternatorMigrator.scala:20)
at com.scylladb.migrator.Migrator$.main(Migrator.scala:43)
at com.scylladb.migrator.Migrator.main(Migrator.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:845)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Could please confirm that your credentials work for the region us-east-1? Do they work in the AWS console?
Yes, my credentials work.
I still suspect there is something wrong with the credentials (see this discussion). Could you please double-check the permissions associated with the actual credentials (see the documentation)?
I've setup aws properly and I can list-tables.
Exception in thread "main" com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: The security token included in the request is invalid. (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ...
ubuntu@ip-10-0-0-129:~/install/spark/spark-2.4.4-bin-hadoop2.7/bin$ aws dynamodb list-tables
{
"TableNames": [
"anand-mig-1",
]
}
Is it possible for migrator to connect via https (port 443). The aws dynamodb list-tables
goes via that port.
It was indeed a permission issue. I was able to get past it and get the tool worling. Closing this. Thank you @julienrf
I tried to run migrator as per the config below. Spark version 2.4.4. OS: ubuntu (AWS)
Error message: