[Parser Fix]: DDE DefinedTerms `curatedBy` fix

[gtsueng]

Issue Name

DDE DefinedTerms curatedBy fix

Issue Description

The records ingested via the DDE are generally manually curated, so it is strange that fields where the expected type is DefinedTerm is being treated as if they were augmented. We should only consider them augmented if the value for the field is a non-URI text.

Suggested fix- Have a separate handler for URIs

• Rationale: There are fields like keywords, measurementTechnique, variableMeasured which may potentially have URI values when ingested via DDE, but currently there is no standardization method to apply if these fields have a free text value.
• If the value is a URI:
  • Use the URI to pull all necessary information so that the front-end can continue to function as normally; HOWEVER.
  • Do not treat as augmented -- The submitter already went through the effort of submitting an ontology value
    • For the curatedBy value (if it's required), use NIAID SysBio (if the record is coming from the NIAID SysBio portal on the DDE), or NIAID Data Ecosystem (if the record is coming from the NDE portal on the DDE). @jal347 should be familiar with such a conditional as he has created one for the includedInDataCatalog field for the DDE parser
  • If the value is NOT a URI, but plain text:
  • Treat as augmented -- we are using our standardization pipeline to improve the metadata

Fields that are potentially affected (i.e. - could potentially have URI values for conversion to DefinedTerm object):

• species
• infectiousAgent
• healthCondition
• measurementTechnique
• keywords
• variableMeasured
• topicCategory

Issue Example

TB Portals is a resource catalog that was added via the DDE: https://discovery.biothings.io/dataset/9142024b72770a67 As seen in this image, the fields infectiousAgent, healthCondition, and species already have ontology uri values so we should not consider them augmented fields, and they should NOT be considered curatedBy PubTator, or BioThings. They were curated to begin with.

However, they currently display in the ecosystem (staging for this example, but most records from NIAID SysBio/DDE in production have the same issue) as being curatedBy PubTator or BioThings. Here is the same record as it appears in data-staging.niaid.nih.gov: https://data-staging.niaid.nih.gov/resources?id=DDE_9142024b72770a67

Related WBS task

• [X] Check here if this repository is funded by NIAID (https://github.com/NIAID-Data-Ecosystem/nde-roadmap/issues/11)
• [X] Check here if this repository only has IID data (https://github.com/NIAID-Data-Ecosystem/nde-roadmap/issues/12)
• [ ] Check here if this is a generalist repository with IID data (https://github.com/NIAID-Data-Ecosystem/nde-roadmap/issues/12)

For internal use only. Assignee, please select the status of this issue

• [ ] Not yet started
• [ ] In progress
• [ ] Blocked
• [ ] Will not address

Status Description

No response

[jal347]

I made a new release on staging. It should be fixed.

[gtsueng]

variableMeasured field appears to still be just a link of urls, are we not able to pull in the name of term?

[gtsueng]

Per discussion on 2024.02.21, PubTator is still greedily assuming credit for these fields for links ingested via the DDE. This behavior needs to be changed.

[gtsueng]

Awesome! It looks like variableMeasured is pulling and displaying terms now! Can we do this for keywords? See https://data-staging.niaid.nih.gov/resources?id=DDE_9142024b72770a67

[jal347]

Thanks for catching that. Ill get that done today.

[gtsueng]

The improvements are on available on staging