hitting request rate limit for AWS KMS when I run CQLreplicator for multiple tables

[frozensky]

I am hitting AWS KMS request rate limit when I start to run CQLreplicator for the 11th table. I have total 24 tables I need to continue to run CQLreplicator before production cutover.

Please advise.

2024-01-28 05:14:51,998 ERROR [main] glue.ProcessLauncher (Logging.scala:logError(77)): Exception in User Class: com.amazonaws.services.s3.model.AmazonS3Exception : Please reduce your request rate for operations involving AWS KMS. (Service: Amazon S3; Status Code: 503; Error Code: SlowDown; Request ID: Q0C47JQSM3ABR8HF; S3 Extended Request ID: t1MkUbq8rFo9W/0k8jlM+nRp3G0BJAqjhJOQXqz52pYFH4jgXtjq1wCWzZdpAKuwoRVMVQILVmY=; Proxy: null)
com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1879)
com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1418)
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1387)
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157)
com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)
com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)
com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)
com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5437)
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5384)
com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:421)
com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:6508)
com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1856)
com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1816)
com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:3982)
GlueApp$.putStats$1(CQLReplicator.scala:328)
GlueApp$.$anonfun$main$44(CQLReplicator.scala:632)
GlueApp$.$anonfun$main$44$adapted(CQLReplicator.scala:576)
com.datastax.spark.connector.cql.CassandraConnector.$anonfun$withSessionDo$1(CassandraConnector.scala:104)
com.datastax.spark.connector.cql.CassandraConnector.closeResourceAfterUse(CassandraConnector.scala:121)
com.datastax.spark.connector.cql.CassandraConnector.withSessionDo(CassandraConnector.scala:103)
GlueApp$.$anonfun$main$43(CQLReplicator.scala:576)
scala.runtime.java8.JFunction1$mcVI$sp.apply(JFunction1$mcVI$sp.java:23)
scala.collection.Iterator.foreach(Iterator.scala:943)
scala.collection.Iterator.foreach$(Iterator.scala:943)
scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
scala.collection.parallel.ParIterableLike$Foreach.leaf(ParIterableLike.scala:982)
scala.collection.parallel.Task.$anonfun$tryLeaf$1(Tasks.scala:53)
scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
scala.util.control.Breaks$$anon$1.catchBreak(Breaks.scala:67)
scala.collection.parallel.Task.tryLeaf(Tasks.scala:56)
scala.collection.parallel.Task.tryLeaf$(Tasks.scala:50)
scala.collection.parallel.ParIterableLike$Foreach.tryLeaf(ParIterableLike.scala:979)
scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.internal(Tasks.scala:160)
scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.internal$(Tasks.scala:157)
scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.internal(Tasks.scala:440)
scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.compute(Tasks.scala:150)
scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask.compute$(Tasks.scala:149)
scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.compute(Tasks.scala:440)
java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189)
java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)

[nwheeler81]

@frozensky there are two quick workarounds: 1/ Stop the cqlreplicator with --state request-stop 2/ Open AWS Glue Console -> ETL Jobs -> CQLReplicator 3/ Find val WAIT_TIME = 30000 and update 30000 to 60000 or higher value 4/ Click Save 5/ Execute --state run again or 1/ Stop the cqlreplicator with --state request-stop 2/ Execute --state run with --override-rows-per-worker 500000 to slow down the replication process

[nwheeler81]

Another option is to create multiple S3 buckets to avoid KMS throttling

[frozensky]

Am I hitting the limit because I have too many rows in a table or because too many tiles running at the same time? Trying to see if I need to separate out 1 keyspaces per s3 bucket or per large table

[nwheeler81]

This is related to Amazon S3 not to Amazon Keyspaces. You might want to try to increase the KMS request quota.

2024-01-28 05:14:51,998 ERROR [main] glue.ProcessLauncher (Logging.scala:logError(77)): Exception in User Class: com.amazonaws.services.s3.model.AmazonS3Exception:Please reduce your request rate for operations involving AWS KMS. (Service: Amazon S3; Status Code: 503; Error Code: SlowDown; Request ID: Q0C47JQSM3ABR8HF; S3 Extended Request ID:; Proxy: null)On Jan 29, 2024, at 3:01 PM, Shaun Chan @.***> wrote: Am i hitting the limit because I have too many rows in a table or because too many tiles running at the same time? Trying to see if I need to separate out 1 keyspaces per s3 bucket or per large table

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you modified the open/close state.Message ID: @.***>