[Augmentation] add topicCategory values using ChatGPT

[gtsueng]

Discussion on slack: https://suwulab.slack.com/archives/CPF4AEHHC/p1696874796709359 Andrew's GH repository: https://github.com/andrewsu/openai_topic_classification

Chunlei's API key should be used to facilitate project billing

Details: The use of the more pricey ChatGPT4 is not necessary as ChatGPT3.5 appears to perform this task sufficiently

Potential testing/evaluation subset: Records with citation.pmid

• PubMed has curated MeSH terms associated with pmids
  • This curation is done using machine-assisted curation. NCBI/NLM biocurators use tools to facilitate the process, but do not rely strictly on tools for the curation.
• MeSH terms can be mapped to EDAM topics ontology with the mappings in NCBO BioPortal
• With both approaches resulting in EDAM topics, the results can be compared

[ZubairQazi]

Pricing estimate for 3,000,000 data points is between $2000 to $3000. Fluctuation is due to a few factors, but mainly that failed API requests are still charged and that inputs vary in token sizes.

[gtsueng]

@ZubairQazi Please do an initial run with the following two repositories:

• ClinEpiDB
• VDJServer

There should be about 100-120 records total. These repositories are "themed" so we can check the conceptual similarity against the themes of the repository to ensure that GPT3.5 turbo is performing as expected:

Themes/Topics for each repository:

• ClinEpiDB: http://edamontology.org/topic_3324, http://edamontology.org/topic_3305, http://edamontology.org/topic_3379
• VDJ Server: http://edamontology.org/topic_2830, http://edamontology.org/topic_3948, http://edamontology.org/topic_0081

Please dump the results and the metrics into a google spreadsheet for review.

[ZubairQazi]

Posting link for future reference:

https://docs.google.com/spreadsheets/d/1lG4hS-PQJ_IRxCc02W3Oz2OFMdYDMURRPbpm3Jg2kk0/edit#gid=1557475722

[gtsueng]

The results for ClinEpiDB and VDJ Server look good. We will proceed to scale up the application to ~1000 records by applying them to the following two repositories:

• LINCS: http://edamontology.org/topic_3534, http://edamontology.org/topic_3172, http://edamontology.org/topic_0602
• ImmPort: http://edamontology.org/topic_3324, http://edamontology.org/topic_3400, http://edamontology.org/topic_0804

[gtsueng]

The results from LINCS and ImmPort also look good, though there appears to be a strange formatting issue for a small number of results. Next steps:

• [x] Ensure that the results from GPT are cached in some fashion so that if there are changes needed in the way the results are parsed, we only need to re-do the parsing, not getting the results from GPT
• [x] Focus on getting the results for these 4 repositories properly formatted and into the dev environment. This way, Leandro can work on developing the front end while you run the data for the repositories that have been approved for display on production.
• [x] Run the repositories that have been approved for production in the order of least number of records to most number of records
• [x] For each repo, take a 10% random sampling of the records (up to 25 max) and dump it in a new spreadsheet. Include only the title and abstract. I will go through and annotate these so that we can use them for QA/QC purposes.

[gtsueng]

The sample data was generated on 2024.03.06 and can be found here for evaluation

[gtsueng]

@ZubairQazi Can you do a topicCategory run for the new repos that @DylanWelzel recently added?

• [ ] MalariaGEN
• [ ] MassIVE
• [ ] MicrobiomeDB

Additionally, please work with @DylanWelzel to ensure that any new records added during the updates since the original topicCategory run also get topicCategories assigned to them.

[gtsueng]

@ZubairQazi please also include

• [ ] Human Cell Atlas