feat: copy core schema jsons in s3 and keep in sync with docdb - Githubissues

AllenNeuralDynamics / aind-data-asset-indexer

MIT License

0 stars 0 forks source link

feat: copy core schema jsons in s3 and keep in sync with docdb #57

Closed helen-m-lin closed 1 month ago

helen-m-lin commented 1 month ago

closes #55

Utils:

copy_then_overwrite_core_json_files: Copies core jsons to /original_metadata, then update all top level jsons with new metadata.nd.json.
sync_core_json_files: Syncs top level jsons with fields from metadata.nd.json. Use md5hash for checking updates.
added some other helper functions

AindPopulateMetadataJsonJob:

When a metadata.nd.json is created, call copy_then_overwrite_core_json_files

AindIndexBucketJob:

_process_docdb_record: on docdb updates, call sync_core_json_files, or copy_then_overwrite_core_json_files if /original_metadata doesn't already exist.
_process_prefix: If adding new metadata.nd.json to docdb, call copy_then_overwrite_core_json_files