[Metadata Improvement]: Fix NIAID SysBio metadata in DDE

[gtsueng]

Issue Name

Fix NIAID SysBio metadata in DDE

Issue Description

This is more of a NIAID SysBio issue than an NDE issue, but it affects our ability to use SysBio as a compelling use case/example

The NDE parser to ingested data from the DDE currently delineates between NIAID SysBio, NIAID Data Ecosystem, and other Data Discovery Engine ingested records based on the context file, however many batch-uploaded SysBio records do NOT have an @context object in the record. This causes it to be attributed only to the Data Discover Engine, and NOT NIAID SysBio

To do:

• [x] Use nde API to find all records missing the @context file (i.e. - they're being attributed only to the Data Discovery Engine
• [x] Use the Biothings DDE API to get the raw json objects for these records
• [x] Add the require @context information to each json object for these records
• [x] Create batch json files for upload to the DDE
  • Batch files can be found here: https://github.com/gtsueng/sysbio_dde_fixes/tree/main/data
• [x] Batch upload the records to update them in the DDE
  • @DylanWelzel, I tried to do this, but it looks like updates can only be done from privileged accounts. Can you try uploading from your account? (Tried, does not work)
  • @newgene, is there any way to confirm that these were uploaded to Laura's account and if so, is there a way for us to update them? (Confirmed that these were all uploaded via Laura's account)
  • @marcodarko, can you help with the batch update?
• [x] Request a refresh of the Data ingested from the DDE by the NDE team

Issue Discussion

The issue with the NIAID SysBio data was discussed at the Biweekly meeting dated 2024.04.03

Please select the type of metadata improvement

• [ ] Standardization (normalizing free text to an ontology)
• [ ] Augmentation (adding values for metadata fields missing values)
• [X] Clean up (addressing redundancy or messy metadata)
• [ ] Structure (changing the structuring of the metadata to support front end UI features)

Meta URL

https://github.com/gtsueng/sysbio_dde_fixes/tree/main/data

Related WBS task

https://github.com/NIAID-Data-Ecosystem/nde-roadmap/issues/12

For internal use only. Assignee, please select the status of this issue

• [ ] Not yet started
• [ ] In progress
• [ ] Blocked
• [ ] Will not address

Status Description

No response

Request status check list

• [ ] This metadata improvement has yet to be discussed between NIAID, Scripps, Leidos
• [ ] This metadata improvement does not need to be discussed between NIAID, Scripps, Leidos
• [X] This metadata improvement has been discussed/reported between NIAID, Scripps, Leidos
• [ ] This metadata improvement has been implemented locally to generate data for review
• [ ] This metadata improvement has been implemented on Dev
• [ ] This metadata improvement has been implemented on Dev and the results have been reviewed and approved for staging
• [ ] This metadata improvement has been implemented on Staging
• [ ] This page/documentation/change has been approved for Production
• [ ] This page/documentation/change has been implemented on Production

[DylanWelzel]

It seems that my account can only edit datasets uploaded by me. I only have 17 datasets available to edit and I cannot override the existing datasets with a bulk upload. Laura's account was the one responsible for uploading these sources originally https://discovery.biothings.io/contributor/flaneuse

[gtsueng]

Thanks for trying, @DylanWelzel. @newgene is there any way to confirm that these were uploaded to Laura's account and if so, is there a way for us to update them?

[newgene]

@gtsueng @DylanWelzel use /api/dataset/<id>?meta=1 you should now get back_meta field. The _meta.username field indicates the owner of the record. And yes, we can update it on the backend (can ping Marco on this).

[gtsueng]

Thanks @newgene I just checked all the records with missing @context, they are all tied to Laura's account. I'll get Marco's help to fix it.

[gtsueng]

@marcodarko, can you help with this issue?

[gtsueng]

As of 2024.05.02, the Data has been fixed on the DDE. A new build of the data will be generated once a fix has been made to the parser to address https://github.com/NIAID-Data-Ecosystem/nde-portal/issues/237

[gtsueng]

This issue has been marked as pending close out and will be closed after a week if there are no further comments on this issue