AllenNeuralDynamics / aind-metadata-service

Service to retrieve Labtracks/Sharepoint info using a REST API
MIT License
3 stars 1 forks source link

Missing procedures from docdb #250

Open mekhlakapoor opened 2 months ago

mekhlakapoor commented 2 months ago

Describe the bug There are ~25 data assets in docdb that have procedures, but when those subject_ids are input into the metadata-service, they have no procedures (or some of the procedures are missing). Look into what could be causing this.

To Reproduce Steps to reproduce the behavior:

  1. Look up and compare 614608 in docdb vs prcoedures endpoint

Expected behavior A clear and concise description of what you expected to happen.

Screenshots image (1)

Desktop (please complete the following information):

Smartphone (please complete the following information):

Additional context Speak with Saskia and Mae about this to get list of subject_ids. I believe there may have been a breaking change in NSB, so check for updates in the NSB schema model

jtyoung84 commented 2 months ago

Let's figure out what the root issue is and we can scope out the fix

Sun-flow commented 1 month ago

referenced asset has been giving an internal server error for ~2-3 weeks now when looking it up in metadata service. looks fine in docdb.

Sun-flow commented 1 month ago

Issue assets were originally identified because they contained procedures that lacked a procedure_type field. This occurred at the general Procedures level (which usually contains surgeries), not at the specific procedure level.

Sun-flow commented 1 month ago

A number of assets turn up with the filter query: {$and: [{"procedures.subject_procedures.procedure_type": {"$exists": false}}, {"procedures.subject_procedures": {"$ne": []}}, {"procedures.subject_procedures": {"$ne": null}}]} in mongoDB.

When looking up the corresponding subject IDs in metadata service, a number of assets return different procedures. Usually, the docdb asset contains a single incomplete subject procedure from NSB, while the data asset in the metadata service contains multiple surgeries at varying levels of completeness, such as containing multiple surgeries with no listed procedures.

Sun-flow commented 1 month ago

List of subjects found using this method: ['689383', '675481', '665840', '665840', '665847', '688267', '689383', '675481', '677330', '665840', '665840', '686314', '688267', '690487', '669705', '669702', '669702', '689383', '669705', '677330', '688267', '690487', '665847', '688267', '669702', '669702', '686314']