Feedback from SME meeting D. Peden

[karafecho]

Posting feedback from meeting with Dave Peden (SME), Kara, Gwen, and Skye on July 8, 2024. This issue will need to be unpacked, but I’m posting the feedback just to capture Dave’s comments and the testing we completed while fresh in my mind and before I lose my scrap-paper notes. BTW, we tested the ui.test.transltr.io endpoint.

What drugs may treat asthma? PK=86545425-9a66-4f52-ad5d-15278005018e (TEST - eel) [Note that this PK is from a rerun and returns somewhat different results than the results that Dave reviewed. My laptop battery died during the call, and apparently, I lost the screenshot intended to capture the PK after the results were complete. Sigh. Plus, we started reviewing results before they were complete, but most/all of the notes/comments below still apply.]

1. The system is “glacially slow”. He’s not sure what the ideal waiting time is for returning query results; however, he noted that "it depends on how desperate I am for answers”. [Maybe remove the requirement to remain on page while answers are being generated? FWIW, I find this to be very annoying, too. An email notification for users who logged in might be helpful, too.]
2. Why is an experimental compound (Gsk-679586a2) returned as the top answer?
3. Descriptions of drugs/chemicals:
   • Descriptions are sometimes, but not always, available; moreover, they are sometimes, but not always, available only after clicking on the drug/chemical to expand the answer. Examples of these scenarios include linolenic acid, itraconazole, pranlukast, fluticasone, salmeterol.
   • Descriptions are too wordy. Dave would like to see a concise description of the chemical class and (for drugs) mechanism of action.
   • Example is terbutaline:
4. Generic results are not helpful. Example is sodium chloride.
5. Scoring:
   • Description of the scoring algorithm is not clear.
   • Scoring/ranking does not seem accurate or intuitive. Why is methylprednisone ranked lower than r-linolenic acid? r-linolenic acid is not a drug. Why is influenza virus vaccine ranked so high?
6. Supporting graph shows an inferred “in clinical trials for” edge, but the KL/AT points only to the ChEMBL wiki page, not the clinical trials. Example is valategrast, which also includes TTD in addition to ChEMBL, but no clinical trials:
   • Dave found many other examples where the ChEMBL wiki page was the supporting evidence in the absence of the clinical trial.
   • Dave felt strongly about replacing ChEMBL clinical trials data with clinicaltrials.gov data. (More support for prioritizing Gwen’s effort and the CQS!)
7. ChEBI role classification categories are not helpful and seem incomplete. For example, terbutaline is a beta agonist, but that’s not highlighted as one of the chemical role categories.
   • Chemical class would be more helpful than the current ChEBI role classification categories.
   • Drug approval status is more helpful than the ChEBI role classification categories.
   • "But where is the approved indication?”
8. Wiki pages are not terribly helpful without edge-specific interpretations. Example records (as opposed to edges) were especially unhelpful.
9. Grouping by chemical class or pathway would be helpful.
10. Description and visualization of lookup versus inferred required explanation, as the current definition was not only insufficient but not easy for Dave to find and the visualization was not intuitive.
11. Dave would like to use Translator for identifying treatments for difficult-to-treat patients, much like U Alabama's PMI docs. Gwen pointed out the disclaimer and that the system is not intended for clinical use, but I'm not sure that matters to a practitioner such as Dave. As someone who has worked with Dave for close to a decade, I feel pretty confident in stating that he wouldn't actually prescribe a drug based on Translator results, but he would very much appreciate a recommendation for difficult-to-treat patients, and he would be happy to wait for answers in order to find a good recommendation ... but only if he could navigate off of the page while results are compiling or otherwise receive an email notification when results are complete. :-) Additionally, allowing users to browse results before they are complete is probably not helpful and leads to more confusion (and weird results) than the quick intermediary results are worth.

What chemicals may downregulate MYD88?

We ran out of time before being able to take a deep dive into the result set, but Dave seemed more satisfied with these results than the “may treat” results, in agreement with Andy's observation that we set the bar too high with MVP1.

[gprice1129]

Adding the label just to communicate that the UI team plans on looking into this feedback which may alter our plans for Fugu.

[gaurav]

NodeNorm has a shorter description for terbutaline ("A member of the class of phenylethanolamines that is catechol substituted at position 5 by a 2-(tert-butylamino)-1-hydroxyethyl group."), but I personally prefer the description we currently use (provided by the Annotator, I think?).

[sharatisrani]

Comment 2: GSK679.... have entered comment to issue #840, tying this issue there - Improving are working on a fix.

Comment 5B: Things have changed. r-linolenic acid is no longer returned. Three linolenic acids are returned, and the highest is at #79, while methylprednisone is at #46. Influenza virus vaccine is at #587, unlikely to see the light of day.

Comment 10: These features are underway, hopefully in Guppy. Very useful to get this endorsement.

[sharatisrani]

Comment 8: Are we not switching to ATC Level 4? @Genomewide @jh111

[sharatisrani]

Comments 4, 6: it will be interesting to see how these are solved. These will have an overall improvement in result quality, and similar comments have been raised in other issues.

[khanspers]

"Additionally, allowing users to browse results before they are complete is probably not helpful and leads to more confusion (and weird results) than the quick intermediary results are worth." -> I agree with this, it is kind of confusing and not intuitive. If the purpose of showing intermediate results is to keep the user engaged and show that something is indeed being returned, maybe the intermediate results should be greyed out and not clickable, to indicate that they are not complete? If the purpose is specifically to allow and encourage users to browse/view incomplete results, then maybe the status message at the top that tells you there are new results should be more prominent (color, size...).

[sstemann]

Adding the label just to communicate that the UI team plans on looking into this feedback which may alter our plans for Fugu.

@gprice1129 are there UI changes expected for Fugu for this? otherwise, lets remove the label

[sierra-moxon]

agree with @khanspers on "load new results" -- @Genomewide also mentioned in a call yesterday, that the slow return of text as tools like chatgpt are running signals the user that the system is still thinking...

[sstemann]

Comment 8: Are we not switching to ATC Level 4? @Genomewide @jh111

based on latest NCATS/UI call we are not switching to ATC Level 4 in the UI without additional user testing. This is pending.

[sstemann]

@capasfield @sierra-moxon for TAQA to really test this, I think this should be converted to a discussion and then individual tickets made and assigned to a release and developer/team,. I can't tell what is expected for Guppy. I'm not sure any TAQA testers can tell.