Is your feature request related to a problem? Please describe.
If the metadata.nd.json exists in s3 but not in docdb, AindIndexBucketJob._process_prefix will write it to DocDB. There is an unhandled error if the metadata.nd.json does not have _id field, causing the indexer to crash.
Describe the solution you'd like
Add a check that _id exists in the metadata.nd.json. We should also check that other services are not creating the .nd.json files, or if they do, they should use something like:
Is your feature request related to a problem? Please describe. If the metadata.nd.json exists in s3 but not in docdb,
AindIndexBucketJob._process_prefix
will write it to DocDB. There is an unhandled error if the metadata.nd.json does not have_id
field, causing the indexer to crash.Describe the solution you'd like Add a check that
_id
exists in the metadata.nd.json. We should also check that other services are not creating the .nd.json files, or if they do, they should use something like:Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.
Additional context Add any other context or screenshots about the feature request here.