Update S3 based on DocDB

[helen-m-lin]

User story

This is part of the AIND Metadata Update POC. As a service admin, I want to user updates to metadata in DocDB to also be reflected in S3, so I can ensure data is in sync.

The new AIND metadata update process will allow users to update metadata directly to DocDB. We can leverage existing AIND Data Asset Indexer to make the appropriate downstream changes to S3.

Acceptance criteria

• [ ] Nightly job checks records in DocDB and make appropriate updates in S3 if required.
• [ ] Ensure data updates between DocDB and S3 do not run in a loop.
• [ ] If desired, retain a history of updates by writing to logs or other store.

Sprint Ready Checklist

• [ ] 1. Acceptance criteria defined
• [ ] 2. Team understands acceptance criteria
• [ ] 3. Team has defined solution / steps to satisfy acceptance criteria
• [ ] 4. Acceptance criteria is verifiable / testable
• [ ] 5. External / 3rd Party dependencies identified
• [ ] 6. Ticket is prioritized and sized

Notes

The new DocDB to S3 workflow should only run in Dev as part of the POC.

[mekhlakapoor]

#check how to retrieve changes from docdb
cursor = collection.watch(full_document='updateLookup’)
for change in cursor:
    updated_record = change['fullDocument']
    s3_key = f'{updated_record["_id"]}.json'

    # Update corresponding data in S3
    s3_client.put_object(Bucket=bucket_name, Key=s3_key, Body=json.dumps(updated_record))

Do a check where you update a record in mongo compass and see if the cursor change looks as we expected. Otherwise, you can change the code as needed but here's a basic idea of how we can do this.