Pipeline to suggest new patterns

[matentzn]

We need a pipeline to suggest new DOSDP patterns from existing EQ definitions in the ontologies.

• [x] familiarise yourself with src/scripts/suggest_new_patterns.sh and get it to run successfully (I didn't test it)
• [x] Implement src/scripts/suggest_new_patterns.py to suggest new patterns. For now:
• [ ] Read all tables from upheno_patterns and get only the "defined_class" column from all the tables, store in a variable like "already_defined".
• [ ] Read all tables from generic_patterns and remove all the rows where "defined_class" is in "already_defined"
• [ ] Group the remaining data by generic pattern and "quality" column, and sort them by how many rows per quality and print them out in markdown, something like:

abnormalQualityOfThing:

morphology (10 phenotypes)

- Example row 1
- Example row 2
- Example row 3

increased size (10 phenotypes)

[rays22]

Progress as of 2022-02-11:

• I have modified the script in in https://github.com/obophenotype/upheno-dev/commit/859f26c22fd580043f2666074708a241dc56f50a and managed to run suggest_new_patterns.sh. I have a collection of tsv files with patterns from ontologies "mp hp xpo wbphenotype".
• I have a prototype python script that compares tsv tables and finds the overlap or disjunction as desired. I need to modify/expand the script so that it aggregates all the already defined patterns from the freshly downloaded "mp hp xpo wbphenotype" ontologies and so it finds the undefined patterns.

[rays22]

I have added the first version of the script that helps identify new patterns (src/scripts/suggest_new_patterns.py) in commit https://github.com/obophenotype/upheno-dev/commit/43e5b53aff6dec1972d7f50d84d45d86fd86a86b. It still has some known limitations including:

1. It uses a library (petl) that is not part of standard python distributions.
2. It is not optimised for a large number of tsv files, so it frequently runs out of steam.

[matentzn]

Thank you @rays22

Do you want to start adding pull requests to the uPheno repo then with new patterns?

[rays22]

Thank you @rays22

Do you want to start adding pull requests to the uPheno repo then with new patterns?

Yes, that is my plan.

[matentzn]

@rays22 has completed this sufficiently.

There are some patterns we still have to create but nothing we need to focus on now.