Closed nwheeler81 closed 8 months ago
Left changes only for MemoryDB and OpenSearch connectors:
sparkSession.read.parquet
in dataReplicationProcess()
and keysDiscoveryProcess()
to glueContext.getSourceWithFormat
with DISK_ONLY
replicationPointInTime
to replicate data from a specific data pointClientConfiguration
with retries for the S3 clientdataReplicationProcess()
, read count.json
, aggregate, update count.json
. Proposed new structure: { "tile": 0, "primaryKeys": value "updatedPrimaryKeys": value "insertedPrimaryKeys": value "deletedPrimaryKeys": value "updatedTimestamp": value }
cqlreplicator
to return new stats --state stats
Proposed changes:
sparkSession.read.parquet
indataReplicationProcess()
andkeysDiscoveryProcess()
toglueContext.getSourceWithFormat
withDISK_ONLY
boolean
data type inrowToStatement
replicationPointInTime
to replicate data from a specific data pointClientConfiguration
with retries for the S3 clientdataReplicationProcess()
, readcount.json
, aggregate, updatecount.json
. Proposed new structure:{ "tile": 0, "primaryKeys": value "updatedPrimaryKeys": value "insertedPrimaryKeys": value "deletedPrimaryKeys": value "updatedTimestamp": value }
cqlreplicator
to return new stats--state stats