[Metadata Improvement]: Generate Benchmark measurementTechniques data for standardization

[gtsueng]

Issue Name

Generate Benchmark measurementTechniques data for standardization

Issue Description

To readily benchmark measurementTechnique standardization approaches, it would be good to have a dataset of terms as a benchmark.

Aside from manually mapping existing terms, this benchmark could be generated with a few approaches:

• [x] Method 1: Programmatic generation of terms for standardization
  • Identifying mappings between the measurementTechnique ontologies (see related issue)
  • Creating a list of terms that exist in more than one ontology
    • Note, the ontology IDs for each term should be saved/stored somewhere for QC after running T2T
  • Adding in a few sample terms from ontologies that did not get represented by the terms in the previous list
  • Manipulating terms in the list to alter capitalization, hyphenation, or pluralization
  • Add dummy stopwords like “administration”, “collection”, “analysis”, “study” to random subset of list
• [x] Method 2: Use ChatGPT to generate the terms
  • An example prompt can be found at: https://github.com/gtsueng/openai_topic_classification/blob/covid/prompt6.txt
  • For a variety of results, a small sample of records from the following repositories (25 per repository) should be run:
    • ClinEpiDB (Clinical techniques)
    • VDJ Server
    • OMICS DI
    • Zenodo
• [x] Method 3: Generate benchmark from existing measurementTechniques
  • Some repositories have more standardized values for measurementTechniques
  • For these repositories, generate a frequency table of the measurementTechnique values and map them manually. Then inspect how well the standardization and extraction processes work. These repositories include:
    • [x] NCBI GEO
    • [x] REFRAMEDB

Issue Discussion

This issue is expected to be discussed only in the context of the results of the measurementTechnique standardization

Please select the type of metadata improvement

• [X] Standardization (normalizing free text to an ontology)
• [ ] Augmentation (adding values for metadata fields missing values)
• [ ] Clean up (addressing redundancy or messy metadata)
• [ ] Structure (changing the structuring of the metadata to support front end UI features)

Meta URL

No response

Related WBS task

https://github.com/NIAID-Data-Ecosystem/nde-roadmap/issues/13

For internal use only. Assignee, please select the status of this issue

• [ ] Not yet started
• [ ] In progress
• [ ] Blocked
• [ ] Will not address

Status Description

No response

Request status check list

• [X] This metadata improvement has yet to be discussed between NIAID, Scripps, Leidos
• [ ] This metadata improvement does not need to be discussed between NIAID, Scripps, Leidos
• [ ] This metadata improvement has been discussed/reported between NIAID, Scripps, Leidos
• [ ] This metadata improvement has been implemented locally to generate data for review
• [ ] This metadata improvement has been implemented on Dev
• [ ] This metadata improvement has been implemented on Dev and the results have been reviewed and approved for staging
• [ ] This metadata improvement has been implemented on Staging
• [ ] This page/documentation/change has been approved for Production
• [ ] This page/documentation/change has been implemented on Production

[gtsueng]

@ZubairQazi can you generate the data described in Method 2 and dump it into a spreadsheet with the columns: Repository, identifier, name, description, measurement terms?

• Repository: One of the four repositories listed
• identifier: The _id of the record in NDE
• name: The name of the record (used in the prompt)
• description: The description of the record (used in the prompt)
• measurement terms: A list of the measurementTechniques generated by chatGPT

[ZubairQazi]

Dumped into this spreadsheet: (Updated Link) https://docs.google.com/spreadsheets/d/1lG4hS-PQJ_IRxCc02W3Oz2OFMdYDMURRPbpm3Jg2kk0/edit#gid=1373622361

[gtsueng]

Great! Thanks @ZubairQazi that was useful for a sanity check. After the improvements, I think we can use this info, please take all the GPT predictions and generate a frequency table.

[ZubairQazi]

Frequency Table: https://docs.google.com/spreadsheets/d/1lG4hS-PQJ_IRxCc02W3Oz2OFMdYDMURRPbpm3Jg2kk0/edit#gid=1081886756

[gtsueng]

@DylanWelzel Once you figure out how to get and parse the mapping from NCBO BioPortal, you can use that to generate test data (Method 1) for evaluating your measurementTechnique standardization approach.