scylladb / scylla-migrator

Migrate data extract using Spark to Scylla, normally from Cassandra
Apache License 2.0
54 stars 34 forks source link

preserve writetimestamp for migrating collections #70

Open tarzanek opened 2 years ago

tarzanek commented 2 years ago

try to preserve timestamp or give a choice which writetimestamp to use for collection

tarzanek commented 1 year ago

preserving timestamp is a no go, since we cannot query writetimestamp for a collection however this is fixed by giving a choice to set writetimestamp with https://github.com/scylladb/scylla-migrator/pull/72

so for collection tables you can use writeTTLInS and (or) writeWritetimestampInuS to set hardcoded value - ideally older than when dual writes have started

mykaul commented 1 year ago

@tarzanek - are you planning to fix it? You seem to be an active contributor ;-)

tarzanek commented 1 year ago

@mykaul this is fixed in master using explicit timestamp given BEFORE dual writes started writeTTLInS writeWritetimestampInuS https://github.com/scylladb/scylla-migrator/blob/master/config.yaml.example#L144 maybe I should close this, since afaik on read side there is no way to get timestamp of the collection and picking it from other column is tricky