alex-thc
commented
1 month ago But this could be 24hr ago, right? So worst case scenario it’ll take time to finish?
Alex
On Thu, Oct 17, 2024 at 11:40 PM adiom-mark @.***> wrote:
@.**** commented on this pull request.
In connectors/dynamodb/stream/helper.go https://github.com/adiom-data/dsync/pull/89#discussion_r1805953738:
- }
- for _, shard := range shards {
- if shard.SequenceNumberRange.EndingSequenceNumber != nil {
- continue
- }
- openShards = append(openShards, shard)
- }
- streamARN := aws.String(s.streamARN)
- state := NewStreamState(s.streamARN)
- eg, egCtx := errgroup.WithContext(ctx)
- for _, shard := range openShards {
- eg.Go(func() error {
- shardIteratorRes, err := s.streamClient.GetShardIterator(egCtx, &dynamodbstreams.GetShardIteratorInput{
it starts reading from the earliest entries in the shards that are still open (no ending sequence number)
— Reply to this email directly, view it on GitHub https://github.com/adiom-data/dsync/pull/89#discussion_r1805953738, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHT75Z5YNP533OTU5TLF4GDZ4CUPJAVCNFSM6AAAAABP5PCFW6VHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDGNZXGEZTCMRUGU . You are receiving this because you commented.Message ID: @.***>
adiom-mark
Currently a rough prototype for dynamodb, support as a source, and as a destination its initial sync only to bootstrap
To try it out you need to set up localstack.
You need to create a dynamodb table using the cli. Then you can try using testconn as a source and feed data into dynamodb.
Then you can use dynamodb as a source to feed data into mongo.