Open callahantiff opened 4 years ago
@bill-baumgartner - I won't be able to meet during our normal Wednesday time, but can meet anytime today after 1pm and anytime tomorrow (for the most part). Let me know that works best for you!
Evaluation steps look good. What was the result of ./owltools -h
?
I can meet @ 1pm today. Tomorrow looks dicey due to the weather forecast.
Evaluation steps look good. What was the result of
./owltools -h
? I can meet @ 1pm today. Tomorrow looks dicey due to the weather forecast.
I have something at 1pm. I could do 12pm or 2pm. Do either of those work?
The result of running ./owltools -h
indicated that of our list of potential ontologies, only ELK
and Pellet
were included.
Noon is great!
@bill-baumgartner - following up from our meeting yesterday. Can you please take a look at OWLTools to see how difficult it might be to add a reasoner from the OWLAPI? Thank you!!
@callahantiff - can you post the list of reasoners we selected?
@callahantiff - can you post the list of reasoners we selected?
No problem, @bill-baumgartner the reasoners we chose are shown in the table below.
Reasoner | Language | OWLTools |
---|---|---|
ELK | EL | Yes |
ELepHant | EL | No |
Pellet | DL | Yes |
RACER | DL | No |
FACT++ | DL | No |
Chainsaw | DL | No |
Konclude | DL | No |
Crack | DL | No |
TrOWL | DL+EL | No |
MORe | DL+EL | No |
First pass - the biggest issue is that most of the available reasoners support OWL API v3, and not v4 or v5.
I'll take a look at integrating Konclude first as it looks to be doable, and then perhaps we take a closer look and prioritize the rest.
First pass - the biggest issue is that most of the available reasoners support OWL API v3, and not v4 or v5.
elephant - C code, "Support for OWL API is planned"
pellet - OWL API v3
racer - OWL API v3
chainsaw - OWL API v3
konclude - no direct connect to OWL API, but may be possible to use: https://www.derivo.de/en/products/konclude/usage-owl-api/
crack - listed as unmaintained here: http://owl.cs.manchester.ac.uk/tools/list-of-reasoners/
trowl - OWL API v3
more - OWL API v3
fact++ - supports OWL API v4
snorocket - not on our list, but supports OWL API v4
I'll take a look at integrating Konclude first as it looks to be doable, and then perhaps we take a closer look and prioritize the rest.
Thanks for the update @bill-baumgartner! I think your plan sounds great! Let me know if I can do anything in the meantime! I will also be in tomorrow if you prefer to meet in person.
@bill-baumgartner - did you want to touch base on this stuff today?
Current status of attempting to use Konclude though the OWL-API is summarized here: https://github.com/ignazio1977/owllink-owlapi/issues/2
Probably not worth meeting today unless you have something you want to discuss.
Turns out that Konclude works well from the command-line (in conjunction with Docker). So far I have not figured out how to retrieve explanations for unsatisfiable classes, but it is straightforward to:
docker run --rm --name "konclude" -v /local/path/to/ontologies:/data konclude/konclude consistency -i /data/cl.owl
docker run --rm --name "konclude" -v /local/path/to/ontologies:/data konclude/konclude classify -v -i /data/cl.owl -o /data/cl.konclude.out
docker run --rm --name "konclude" -v /local/path/to/ontologies:/data konclude/konclude satisfiability -v -i /data/cl.owl -x http://purl.obolibrary.org/obo/CL_2000096
Turns out that Konclude works well from the command-line (in conjunction with Docker). So far I have not figured out how to retrieve explanations for unsatisfiable classes, but it is straightforward to:
- check the consistency of an ontology, e.g.
docker run --rm --name "konclude" -v /local/path/to/ontologies:/data konclude/konclude consistency -i /data/cl.owl
- classify an ontology, e.g.
docker run --rm --name "konclude" -v /local/path/to/ontologies:/data konclude/konclude classify -v -i /data/cl.owl -o /data/cl.konclude.out
- and, check the satisfiability of a specific class, e.g.
docker run --rm --name "konclude" -v /local/path/to/ontologies:/data konclude/konclude satisfiability -v -i /data/cl.owl -x http://purl.obolibrary.org/obo/CL_2000096
Awesome, thanks Bill! Once I have the graph built I will give this a try! Happy to help dig around to find the output for unsatisfiable classes too.
Morning @bill-baumgartner!
Below is a link to download a zipped directory of the ontologies we can use for the reasoner challenge. I have spent the last few days extensively cleaning these files (i.e. removed punning, fixing erroneous Literal typing, correcting inconsistent imported ontology identifiers, etc.). I also went ahead and ran each cleaned file through ELK to make sure that the cleaning didn't inadvertently cause other errors. All of them passed.
Download Link: reasoner_challenge.zip
Directory Contents Aside from the clean single ontology files, you will also find:
n=10
) n=45
; naming scheme: ont1_ont2_merged.owl
)all_ontologies_merged.owl
)Note. All ontologies in the directory contain all original entities (i.e. not had all annotation assertions removed to create the minimal symbolic representation). I figured we should start with them intact and if need be, apply the reduction transformation later on (it's super easy to do).
Question: @bill-baumgartner - do we want all triple-, quad-, ...n-1
-wise merge files? It is easy to generate these, but thought I would ask just in case you had something else in mind.
Ok, that's all for now, see you in a few hours! 😄 🎉
@bill-baumgartner - here is the Jupyter Notebook I mentioned during our meeting today (incase you are curious).
@bill-baumgartner - just wanted to check-in and see how the great reasoner challenge of 2020 was going? Still OK to meet at our usual time tomorrow?
Yep, let's meet. I should have some preliminary data to show you.
Yep, let's meet. I should have some preliminary data to show you.
@bill-baumgartner - Woo hoo! Sounds great, I am looking forward to it!
TODO: Perform comparison of OWL reasoners.
During today's meeting with @bill-baumgartner, we outlined how we will perform an evaluation of OWL reasoners on the PheKnowLator V.2.0 KG.
OWL Reasoner Selection Criteria
Using the following reviews (shown below), we selected reasoners that met the following criteria:
Eligible Reasoners:
@bill-baumgartner - to determine what reasoners were available in OWLTools, I ran the following:
Next Steps:
Evaluation Steps:
Benchmark each of the algorithms on HPO+Imports
For all algorithms that pass the benchmark, run them against PheKnowLator
Clinician Evaluation via @jwyrwa
@bill-baumgartner - did I forget anything?