AllenNeuralDynamics / aind-data-asset-indexer

MIT License
0 stars 0 forks source link

Check if _id exists in metadata.nd.json #64

Closed helen-m-lin closed 1 month ago

helen-m-lin commented 1 month ago

Is your feature request related to a problem? Please describe. If the metadata.nd.json exists in s3 but not in docdb, AindIndexBucketJob._process_prefix will write it to DocDB. There is an unhandled error if the metadata.nd.json does not have _id field, causing the indexer to crash.

Describe the solution you'd like Add a check that _id exists in the metadata.nd.json. We should also check that other services are not creating the .nd.json files, or if they do, they should use something like:

metadata_dict = Metadata.model_construct(**metadata_dict).model_dump_json(warnings=False, by_alias=True)

Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.

Additional context Add any other context or screenshots about the feature request here.