search

callahantiff / PheKnowLator

PheKnowLator: Heterogeneous Biomedical Knowledge Graphs and Benchmarks Constructed Under Alternative Semantic Models
https://github.com/callahantiff/PheKnowLator/wiki
Apache License 2.0
159 stars 29 forks source link

TODO: Perform OWL Reasoner Evaluation #18

Open callahantiff opened 4 years ago

callahantiff commented 4 years ago

TODO: Perform comparison of OWL reasoners.

During today's meeting with @bill-baumgartner, we outlined how we will perform an evaluation of OWL reasoners on the PheKnowLator V.2.0 KG.

OWL Reasoner Selection Criteria
Using the following reviews (shown below), we selected reasoners that met the following criteria:

  1. Low response time
  2. Available via the OWLAPI
  3. Open source 
Khamparia A, Pandey B. Comprehensive analysis of semantic
web reasoners and tools: a survey. Education and Information
Technologies. 2017 Nov 1;22(6):3121-45.
Parsia B, Matentzoglu N, Gonçalves RS, Glimm B, Steigmiller A.
The OWL reasoner evaluation (ORE) 2015 competition report.
Journal of Automated Reasoning. 2017 Dec 1;59(4):455-82.

Eligible Reasoners:

Reasoner Language OWLTools
ELK EL Yes
ELepHant EL No
Pellet DL Yes
RACER DL No
FACT++ DL No
Chainsaw DL No
Konclude DL No
Crack DL No
TrOWL DL+EL No
MORe DL+EL No

@bill-baumgartner - to determine what reasoners were available in OWLTools, I ran the following: 

./owltools -h

Next Steps:

Evaluation Steps:

  1. Benchmark each of the algorithms on HPO+Imports

    • Run-time
    • Justifications
    • Count of inferred axioms
    • Consistency

  2. For all algorithms that pass the benchmark, run them against PheKnowLator

    • Including disjointness axioms
    • Excluding disjointness axioms

  3. Clinician Evaluation via @jwyrwa

    • Create a spreadsheet of the inferred axioms by algorithm and mark them as:
      • Correct/Incorrect
      • Definitely Clinically relevant, Maybe clinically relevant, not clinically relevant

@bill-baumgartner - did I forget anything?

callahantiff commented 4 years ago

@bill-baumgartner - I won't be able to meet during our normal Wednesday time, but can meet anytime today after 1pm and anytime tomorrow (for the most part). Let me know that works best for you!

bill-baumgartner commented 4 years ago

Evaluation steps look good. What was the result of ./owltools -h? I can meet @ 1pm today. Tomorrow looks dicey due to the weather forecast.

callahantiff commented 4 years ago

Evaluation steps look good. What was the result of ./owltools -h? I can meet @ 1pm today. Tomorrow looks dicey due to the weather forecast.

I have something at 1pm. I could do 12pm or 2pm. Do either of those work?

The result of running ./owltools -h indicated that of our list of potential ontologies, only ELK and Pellet were included.

bill-baumgartner commented 4 years ago

Noon is great!

callahantiff commented 4 years ago

@bill-baumgartner - following up from our meeting yesterday. Can you please take a look at OWLTools to see how difficult it might be to add a reasoner from the OWLAPI? Thank you!!

bill-baumgartner commented 4 years ago

@callahantiff - can you post the list of reasoners we selected?

callahantiff commented 4 years ago

@callahantiff - can you post the list of reasoners we selected?

No problem, @bill-baumgartner the reasoners we chose are shown in the table below.

Reasoner Language OWLTools
ELK EL Yes
ELepHant EL No
Pellet DL Yes
RACER DL No
FACT++ DL No
Chainsaw DL No
Konclude DL No
Crack DL No
TrOWL DL+EL No
MORe DL+EL No
bill-baumgartner commented 4 years ago

First pass - the biggest issue is that most of the available reasoners support OWL API v3, and not v4 or v5.

I'll take a look at integrating Konclude first as it looks to be doable, and then perhaps we take a closer look and prioritize the rest.

callahantiff commented 4 years ago

First pass - the biggest issue is that most of the available reasoners support OWL API v3, and not v4 or v5.

I'll take a look at integrating Konclude first as it looks to be doable, and then perhaps we take a closer look and prioritize the rest.

Thanks for the update @bill-baumgartner! I think your plan sounds great! Let me know if I can do anything in the meantime! I will also be in tomorrow if you prefer to meet in person.

callahantiff commented 4 years ago

@bill-baumgartner - did you want to touch base on this stuff today?

bill-baumgartner commented 4 years ago

Current status of attempting to use Konclude though the OWL-API is summarized here: https://github.com/ignazio1977/owllink-owlapi/issues/2

Probably not worth meeting today unless you have something you want to discuss.

bill-baumgartner commented 4 years ago

Turns out that Konclude works well from the command-line (in conjunction with Docker). So far I have not figured out how to retrieve explanations for unsatisfiable classes, but it is straightforward to:

callahantiff commented 4 years ago

Turns out that Konclude works well from the command-line (in conjunction with Docker). So far I have not figured out how to retrieve explanations for unsatisfiable classes, but it is straightforward to:

  • check the consistency of an ontology, e.g. docker run --rm --name "konclude" -v /local/path/to/ontologies:/data konclude/konclude consistency -i /data/cl.owl
  • classify an ontology, e.g. docker run --rm --name "konclude" -v /local/path/to/ontologies:/data konclude/konclude classify -v -i /data/cl.owl -o /data/cl.konclude.out
  • and, check the satisfiability of a specific class, e.g. docker run --rm --name "konclude" -v /local/path/to/ontologies:/data konclude/konclude satisfiability -v -i /data/cl.owl -x http://purl.obolibrary.org/obo/CL_2000096

Awesome, thanks Bill! Once I have the graph built I will give this a try! Happy to help dig around to find the output for unsatisfiable classes too.

callahantiff commented 4 years ago

Morning @bill-baumgartner!

Below is a link to download a zipped directory of the ontologies we can use for the reasoner challenge. I have spent the last few days extensively cleaning these files (i.e. removed punning, fixing erroneous Literal typing, correcting inconsistent imported ontology identifiers, etc.). I also went ahead and ran each cleaned file through ELK to make sure that the cleaning didn't inadvertently cause other errors. All of them passed.


Download Link: reasoner_challenge.zip


Directory Contents Aside from the clean single ontology files, you will also find:

Note. All ontologies in the directory contain all original entities (i.e. not had all annotation assertions removed to create the minimal symbolic representation). I figured we should start with them intact and if need be, apply the reduction transformation later on (it's super easy to do).

Question: @bill-baumgartner - do we want all triple-, quad-, ...n-1-wise merge files? It is easy to generate these, but thought I would ask just in case you had something else in mind.

Ok, that's all for now, see you in a few hours! 😄 🎉

callahantiff commented 4 years ago

@bill-baumgartner - here is the Jupyter Notebook I mentioned during our meeting today (incase you are curious).

callahantiff commented 4 years ago

@bill-baumgartner - just wanted to check-in and see how the great reasoner challenge of 2020 was going? Still OK to meet at our usual time tomorrow?

bill-baumgartner commented 4 years ago

Yep, let's meet. I should have some preliminary data to show you.

callahantiff commented 4 years ago

Yep, let's meet. I should have some preliminary data to show you.

@bill-baumgartner - Woo hoo! Sounds great, I am looking forward to it!