Closed heyhughes closed 2 years ago
@heyhughes you can change it to the new list_objects_v2 and use the 'start after' property. You can lookup the last processed file from the Dynamo controller table, and have list_objects_v2 start after that file in S3. this way, you then get the next 1000 that haven't been processed. Worked like a charm for me
That did the trick! Thank you @sheridan06
I just updated the code with this suggestion.
Any thought to the 1000 limit with list_objects? We're finding that we have more than 1000 objects with the particular prefix (DMS syncing a volatile table). The Controller python uses s3conn.list_objects to identify if any new files need to be incrementally loaded. If you have over 1000 2*.parquet files the incremental job will never kick off.