scylladb / scylla-migrator

Migrate data extract using Spark to Scylla, normally from Cassandra
Apache License 2.0
54 stars 34 forks source link

Multiple tables cannot be migrated simultaneously #96

Open bharatnailwal opened 1 year ago

bharatnailwal commented 1 year ago

Hello Team,

Please help me find someone who can assist me with migrating multiple tables from a keyspace at the same time using the Scylla migration tool? I tried making changes to config.yml, but it did not work.

error: Exception in thread "main" DecodingFailure(Attempt to decode value on failed cursor, List(DownField(table), DownField(source)))

bharatnailwal commented 1 year ago

config.yaml

source:
  type: cassandra
  host: <great-secret>
  port: 9042
  credentials:
    username: great-secret
    password: 'great-secret'
  keyspace: <testkeyspace>
  table: <testtable1>
  consistencyLevel: ONE
  preserveTimestamps: true
  splitCount: 256
  connections: 8
  fetchSize: 1000
target:
  type: scylla
  host: <great-secret>
  port: 9042
  credentials:
    username: great-secret
    password: 'great-secret'
  keyspace: <testkeyspace>
  table: <testtable1>
  consistencyLevel: ONE
  connections: 16
  stripTrailingZerosForDecimals: false
savepoints:
  path: /app/savepoints
  intervalSeconds: 300
renames: []
skipTokenRanges: []
validation:
  compareTimestamps: true
  ttlToleranceMillis: 60000
  writetimeToleranceMillis: 1000
  failuresToFetch: 100
  floatingPointTolerance: 0.001
  timestampMsTolerance: 0     
tarzanek commented 1 year ago
<testtable1> you likely need to escape or quote the value in source/table

e.g. 
  table: "<testtable1>"

but I doubt that is a table name, so you likely need to use proper name there
tarzanek commented 1 year ago

also you can migrate only one by one (I usually script it with bash scripts) (check https://github.com/scylladb/scylla-code-samples/tree/master/spark-scylla-migrator-demo )

you can even have bash templates and replace the values in them from env (look at envsubst )

@bharatnailwal ^