search

aws-samples / cql-replicator

CQLReplicator is a migration tool that helps you to replicate data from Cassandra to AWS Services
Apache License 2.0
16 stars 8 forks source link

[CQLReplicator on Glue] An error occurred (ConcurrentRunsExceededException) #107

Closed frozensky closed 9 months ago

frozensky commented 9 months ago

I am getting error on ConcurrentRunsExceededException when I run for the largest table. Attached running job count and quota page

Screenshot 2024-02-05 at 8 37 59 PM Screenshot 2024-02-05 at 8 38 44 PM

Also the Sampled primary keys per tile is 64838121 x 50 tiles = 3.24 billion rows only. It's way less than than estimate total row count (around 8 Billions)

[ec2-user@ip-10-144-111-195 bin]$ ./cqlreplicator --state run --tiles 50 --writetime-column modificationtime --landing-zone s3://cqlrep-prd-1  --region us-west-1 --src-keyspace quark --src-table personalcontentslists --trg-keyspace quark --trg-table personalcontentslists --override-rows-per-worker 1500000 --inc-traffic

[2024-02-06T03:56:21+00:00] OS: Linux
[2024-02-06T03:56:22+00:00] AWS CLI: aws-cli/2.15.17 Python/3.11.6 Linux/6.1.72-96.166.amzn2023.x86_64 exe/x86_64.amzn.2023 prompt/off
[2024-02-06T03:56:22+00:00] Incremental traffic for the historical workload is enabled
[2024-02-06T03:56:22+00:00] Incremental period: 1200 seconds
[2024-02-06T03:56:22+00:00] Starting discovery process...
[2024-02-06T03:56:22+00:00] TILES: 50
[2024-02-06T03:56:22+00:00] SOURCE: quark.personalcontentslists
[2024-02-06T03:56:22+00:00] TARGET: quark.personalcontentslists
[2024-02-06T03:56:22+00:00] LANDING ZONE: s3://cqlrep-prd-1
[2024-02-06T03:56:22+00:00] WRITE TIME COLUMN: modificationtime
[2024-02-06T03:56:22+00:00] TTL COLUMN: None
[2024-02-06T03:56:22+00:00] ROWS PER DPU: 1500000
[2024-02-06T03:56:22+00:00] START REPLICATING FROM: 0 (0 is disabled)
[2024-02-06T03:56:22+00:00] SAFE MODE: true
[2024-02-06T03:56:22+00:00] Checking if the discovery job is already running...
[2024-02-06T03:56:26+00:00] Starting the discovery job...
[2024-02-06T04:20:29+00:00] Sampled primary keys per tile is 64838121

An error occurred (ConcurrentRunsExceededException) when calling the StartJobRun operation: Concurrent runs exceeded for CQLReplicator
frozensky commented 9 months ago

nvm, found the issue

https://github.com/aws-samples/cql-replicator/blob/main/glue/bin/cqlreplicator#L392 https://github.com/aws-samples/cql-replicator/blob/main/glue/bin/cqlreplicator#L414