User testing & review

[maxkfranz]

Bug testing (everyone):

Run through the basic user workflow yourself and keep an eye out for software bugs. Note them on Github.

User testing (Toronto in the new year):

• End-to-end walkthrough with potential users
  • The instructor has a second computer (probably laptop) that he uses to perform system actions in the admin panel. The instructor uses the admin panel to approve the document request and to send the invitation email.
  • The user fills out the "get started" form on the homepage.
  • Does the user know what a Pubmed link is?
  • Does the user get tripped up by the first author question?
  • Does the user receive the email? Does he easily find the link/button to open the editor?
  • The user edits the pathway figure.
  • Does the user understand how to add entities and interactions?
  • Does the user run into any grounding issues?
  • Is there anything that the user wants to represent that the system does not support?
  • Does the user share to Twitter after he's finished the figure?
• Misc. questions
  • What are the user's thoughts on the wording on the homepage?
  • What kind of things could provide value to users in the read-only explorer view?

[jvwong]

Info

• Dec 19, 2019
• Two testers, in-person
• Version
  • GitHub

Setup

Context

Congratulations! You have just published a research article. You have become aware of ‘Bio factoid’, a website where you can share the biological pathway information in your paper with colleagues.

Background

A combination of genetic and biochemical evidence showed that the human KRAS binds to, phosphorylates and contributes to the activation of the three closely related RAF proteins, including BRAF.

Provided Article

Ras recruits Raf‐1 to the plasma membrane for activation by tyrosine phosphorylation. R. Marais, Y. Light, H.F. Paterson, C.J. Marshall EMBO J (1995)14: 3136-3145

Notes

User 1 (Background: Bioinformatics)

• Home/Signup
  • Went to PubMedCentral to retrieve a PMCID
  • Entered PMCID in signup form
• Email
  • Immediately knew to click 'START' button
  • Would have deleted email otherwise
• Editor
  • Mousing over the logo: Expected the tooltip with 'Menu' text to be a menu button
• Twitter/Sharing
  • How do I know if it is saved? I want to save it.
  • How do I access this pathway again later?
  • Can I share privately / preview it before it is shared / share it with just myself?
  • Who's account will this be tweeted from?

User 2 (Background: Experimental/Bioinformatics)

• Home/Signup
  • General comments: Be more explicit. Please provide up-front, information about: What to expect, what information I require and for what purpose
  • 'Summarize, Share, Discover' what?
  • 'Get Started' doing what?
  • Suggested rearranged banner/above the fold content:
  • Move up 'Summarize' section text
  • Move up Molecular Cell author examples as they are more concrete
  • Describe what information I will need to input and why
  • Get started: Initially not sure what we meant by 'Pubmed link'
• Email
• Editor
• Twitter/Sharing
  • Before clicking, expected 'Share' to provide some checks or validation
  • Could not recall 'Sharing to Twitter' indicated in information/Home/Signup
  • Unsure of who's account or where this would be Tweeted from

[sacdallago]

599 should be here

[jvwong]

from #599, Factoid internal

Homepage

• Center summarize, share, discover text: https://unstable.factoid.baderlab.org/
• Not sure why "Molecular Cell" is below the guy presenting (if I were a naive new user).

Document management panel

• I would make the icons on the admin page more self-described: like the thumbs and the email icon. They could just be replaced with words like "Corresponding Author" for the email icon.

App

• There is nothing to prompt the user that a match was successfully done. The current interface makes me think I need to click on 'Select a better match for "sirt1"'.
• After I turn off the initial tutorial modal, I don't easily know how to get instructions back. It would be nice if there was a "?" question mark icon somewhere.
• The instructions on the modal don't tell me anything about the interaction or node types
• Should there be a link to the paper on the Twitter post not just Factoid?

Twitter

• Most people like colors, but I don't see this happening anytime soon.

[jvwong]

from Thursday, Jan 9, Factoid plenary

Homepage

• Change "subcellular biology" phrase?
• "hidden gems"? change? Direct connection to your paper?
• Say something about depositing content to some database? Some "freely available database"
• "View only mode" is not specified for example. Might annoy users that it is not interactive? Should it link to the Twitter output? No information that clicking on a node gives a popup with info. Added guided tour? or video instructions?
• Add video to homepage?
• Why is Molecular Cell there?
• Say example on the bottom examples?
• Add PubMed link example
• Change background color to black?
• NLP missing?
• Add a list of extracted entities to simplify the author work?
• Tell authors to "record the most important interactions"

Editor

• How to accept the gene identifier is selected by the system?
• Human as default organism?
• Should the node name change to the official symbol?
• How to draw interaction? Do you have to press the keyboard shortcut? How to draw the interaction.
• Submit button?
• Inhibit complex possible?
• No inhibit interaction?
• Interaction type is ambigous
• Help button bigger? More explanation. Long press not explained
• Share on Twitter? Caption
• How to BioPAX share?
• Tweet but share with other people
• Populate box with two entity name for complex?
• Interactions within complex? Name lost for complex. Pulling apart nodes slowly maintains complex.
• Corresponding author important?
• On homepage, "PMID: XYZ" as the input. No example generated
• Everything filled out and error message
• "Contributing for" clarify text

Admin

• Invites are not sorted by time

[jvwong]

Date: November 2019 User background: Bioinformatics and Computational Biology

Comments:

• I think it is nice and simple enough for authors to use. However, making it so simple, leaves all context out.
• I think a description of the scope of the tool is needed from the homepage it seems that it is a summarization tool for anything. Not every paper will be suitable for factoid based on the relations and entities captured (e.g., variants and phenotypes). Only is clear in the help snapshot that talks about pathway.
• Related to previous there is no help indicating the inventory of relations and example of use, and the entities and DBs/ontologies that are available.
• You may want to consider ORCID as a general method for signing in.
• As entities you have genes and chemicals, how would you represent complexes? For example many receptors are complexes.
• I was looking into this example https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3561766/
  • The relation P53 activates the transcription/translation of miRNA-194-1 activates the transcription/translation sounds odd in context of miRNA, as it is a non coding molecule
  • In this case should I select Other as the relation?
• How do I indicate the evidence for the relation, the PMID..?
• Also the relation of "binding with" shouldn't it be bidirectional?
• Do you have in the backend mapping of relation to some ontology like relation ontology (RO)?

[jvwong]

Date: Jan 16/17 Users: Two Reactome curators, simultaneously

• Home
  • Missing clear purpose - why?
    • "Visual abstract" is confusing
  • Requires more detail, fuller description
  • "Highlight critical interactions"
  • "Made computationally/easily accessible"
  • Mol Cell
  • Need a clearer example of purpose, other than a specific paper example that exists
• Email
• Editor
  • Grounding
  • Autoground
    • Did not understand that grounding was applied automatically
    • For chemical "bi-2582", expected chemical but top hit returned was gene, so was confused by 'Type' option
  • "Select better match" confused with a message prompt
    • "OK" or "Submit" missing
  • Delete
  • At first, unsure of how to do this
  • The popup on nodes and edges distracts/interferes with selection
  • Finishing/Saving
  • "What do I do now"? "How do I save"?
  • "'Share' sounds like publish"; "'Submit' is clearer"
  • Twitter/Sharing (aside: Testers were admitted reluctant Twitter users)
  • "Where is it 'published' and by whom"?
• General
  • Where is the data going? Where does it end up?
  • Why would people come here?
• Further suggestions
  • Getting the ball rolling
  • Pre-curation then ask author to verify in the confines of a publication/peer-review pipeline (e.g. Molecular Cell).
  • However, sometimes have seen software where its just easier to start from scratch
  • Accessing data
  • Search a list of annotations
  • Part of Pathway Commons or Reactome context

[jvwong]

Summarized

Updated: Jan 20 2010

Common concerns

• Homepage
  • Text
    • Vague descriptions of goal, lacking sufficient context
      • Increase the level of detail and concreteness in descriptions
    • Highlight
      • “Highlight important interactions in your paper”
      • Computational/easy accessible
  • Molecular Cell banner
    • purpose of section is unclear
      • Indicate that these are examples
      • Need more to support the idea of our purpose
• Email
• Editor
  • Expected to accept or apply grounding
    • Acknowledge status of grounding
    • ‘Select better match’ should be clear alternative
  • Help
    • Difficult to access and find tutorial
    • Some is absent
      • Complex
  • Complex/compound
    • Unaware of existence
    • Creation and destruction has behaviours not easily discovered
  • Completion
    • Expect save or submit action
    • Expect input validation
      • Acknowledge status of the data / doc
• Sharing
  • Unsure which account will share
    • Clarify outcome of sharing
  • No article (journal) hyperlink
• General
  • Scope
    • Insufficient detail of biological concepts, types of data desired
      • Type and scope of data desired
  • Goals
    • Unclear is the destination of the data - where does it end up?

Individual concerns

• Homepage
  • Text
    • No indication of where data goes or can be accessed
    • No clear relationship with data and article
  • Call to action
    • Tried to enter a PMCID
      • Support other major paper identifiers
    • Error message remains with input
      • BUG:
      • “PMID: 31876050”
      • cccsander@...
  • Molecular Cell banner
    • Examples should linkout to Twitter post or something more useful than the rather underwhelming read-only view
• Email
• Editor
  • Ground
    • Type: Not obvious that top hit was gene vs chemical (e.g. ‘bi-2582’)
  • Delete
    • Popup distracts/interferes with selection
  • Complexes
    • Drag out continuously (removal) vs multi-click drag short distance (remains) - BUG?
    • Improve name
      • Populate box with entities by default
  • Interaction types
    • ‘transcription/translation’ is awkward with respect to non-coding RNA, e.g. (miRNA-194-1)
    • Binding with should indicate bi-directionality vs ‘no direction’
  • Unsure how to provide evidence for interaction
    • PMID
• General
  • Seeding data
    • Under peer-review, could pre-curate and author-validate
  • Discover
    • Get a plan together
      • Simple search listing
      • Integration into third-party sites