NCATSTranslator / Feedback

A repo for tracking gaps in Translator data and finding ways to fill them.
7 stars 0 forks source link

"Streptomycin treats Leukemia" based on bad text-mining results #956

Open khanspers opened 2 months ago

khanspers commented 2 months ago

Query: What drugs may treat conditions related to Leukemia?

https://ui.test.transltr.io/results?l=Leukemia&i=MONDO:0005059&t=0&r=0&q=e83261ac-4200-45b4-8d4d-9f0abebefbdc

Streptomycin (an antibiotic) is returned (score of 4.99), from BTE and Unsecret, based on text mining. However, looking at the publication evidence for path a and the supporting paths for path b in the below screenshot, all the papers reference using Streptomycin in culture of cancer cell lines (presumably to avoid contamination), and nothing about the use of streptomycin to treat Leukemia. This can be seen from the "Snippet" that is provided in the UI, but I also opened each paper at PubMed and confirmed this with an in-page search for Streptomycin.

Screen Shot 2024-09-27 at 11 15 44 AM

Similarly, Streptomycin is returned for treats queries for other cancer types, for example lymphoma (score 4.94), renal carcinoma (score 4.94), also with snippets from papers describing cell culture.

Not sure if this is considered a bug or just how text-mining works. But it seems to be different from papers mentioning a drug/chemical in the same sentence as the queried disease, which is what is often in the snippet? Maybe its an ordering/scoring issue?

khanspers commented 1 month ago

Just tested the "what treats Leukemia" query again and while the results look different, Streptomycin is still returned (score 4.96), and the papers listed as evidence are similar to what was reported before, they mention streptomycin/penicillin in terms of cell culture.

https://ui.test.transltr.io/results?l=Leukemia&i=MONDO:0005059&t=0&r=0&q=6467fc16-f157-40cf-ae99-2766cd58069a