search

biothings / biothings_explorer

TRAPI service for BioThings Explorer
https://api.bte.ncats.io
Apache License 2.0
8 stars 9 forks source link

Create regression testing infrastructure #117

Closed andrewsu closed 3 years ago

andrewsu commented 3 years ago

We would like to create a regression testing framework to quantitatively assess BTE's performance. As a gold standard, we can use the orphan drug indication dataset mentioned in https://github.com/NCATSTranslator/Relay/discussions/123 or the mechanistic paths from https://sulab.github.io/DrugMechDB/. For each of those gold standards, we should create a TRAPI query (examples), send it to BTE using a small library of plausible metapaths focused on drug repurposing, and then assess whether BTE was able to retrieve the right drug among the results. (Later we can also assess where that drug ranked among all potential drugs retrieved.) We would want to execute this test on a regular basis (weekly?), and then have a simple web page where results can be viewed/browsed.

tagging @ariutta and @AlexanderPico

andrewsu commented 3 years ago

Still plenty left to be done, but just remembered that @kevinxin90 did some initial analysis of the OOPD dataset in https://github.com/biothings/bte_regression_test, and we also created some issues there. Would be good to build on that...

ariutta commented 3 years ago

@andrewsu I'm copying this from Slack over to here too:

Brief summary: we now have a workflow bte_regression_test.yml that is configured to run every Saturday as well as on demand. (There's also an abbreviated version of this same workflow: bte_regression_test_quick_demo.yml).

It pulls the latest BTE TRAPI Docker image to create a local instance of the API and then runs every query template in src/query_templates against that API. It currently saves the resulting full_results.csv file as a GH Action artifact, but the future plan is to create a static page showing summary stats over time as well as a Jupyter notebook for interactively exploring the results, potentially hosted via colab or mybinder. Another future improvement: use batch queries to speed up the tests. Right now, it takes 1 hour to run the six query templates for the first two lines of the OOPD file and the better part of a day to run the entire test.

If you want to see some completed workflow runs, you can take a look at my fork: https://github.com/ariutta/bte_regression_test/actions You'll notice the full test timed out, but the quick demo did finish, so you can see the result saved as an artifact.

ariutta commented 3 years ago

I'll close this issue in favor of the new issue created in the bte_regression_test repo: https://github.com/biothings/bte_regression_test/issues/9