scylladb / scylla-migrator

Migrate data extract using Spark to Scylla, normally from Cassandra
Apache License 2.0
54 stars 34 forks source link

Ignore collection types timestamp & ttl but preserve for regular columns #81

Open fee-mendes opened 2 years ago

fee-mendes commented 2 years ago

When preserveTimestamps: true determineCopyType throws when a collection is found within the given source table.

This approach - however - prevents one to preserve timestamps and TTLs for regular columns when a collection column exists, which which may not be acceptable for some use cases. The only alternative existing today is to hardcode a writetime and TTL for the entire migration needlessly.

This issue, therefore, is an attempt to extend the preserveTimestamps, writeWritetimestampInuS and the writeTTLInS options so that they can be used in conjuction in such a way that only collection fields are involved during the hardcode process and regular columns have their batches split to preserve their original writetime() and ttl().

fee-mendes commented 2 years ago

@tarzanek

tarzanek commented 2 years ago

you can use the same timestamp for all rows and for collections too, see the new timestamp and ttl option at the bottom of sample conf file @fee-mendes , preservetimestamp has to be false for them to be used

tarzanek commented 2 years ago

ah, ok, let's see if this can be improved it won't be easy