Normalise the extracted entities to an ontology

[ireneisdoomed]

To be able to extract the 1:M relationships between diseases and phenotypes, it would be useful to follow an ontology based approach of grounding the identifying terms to common term definitions.

Using an ontology gives us 2 important benefits:

• More definition. A disease is identified with different ways or referring to the same ontological term across the literature. Being able to have a common definition will improve the clustering algorithm in a later stage when we want to identify the associated phenotypes.
• Structure. By giving the constraints of the ontology, we make sure that the terms do not live in the void. This is important in the part where we distinguish between phenotypes and diseases. When we map a label to a HP term, we know that the text most likely refers to a sign and not a disease.

As per the ontologies we should be using, I see 2 interesting possibilities and I don't know which will yield better results:

• Using the Experimental Factor Ontology (EFO). a) I'd define as phenotype every child of these terms:

  • phenotype
  • measurement b) Diseases will be defined as any other term included in the OT Disease index that are not included in a)

• Using MONDO and the Human Phenotype ontology

The benefits of using EFO is that I am familiar with the dataset structure OT provides, and the benefits of using MONDO/HP is that I have the impression that their coverage and ontological representation might be slightly for this task.

[ireneisdoomed]

In terms of vector stores there are many we can use: FAISS (simple and local), LlamaIndex, ChromaDB or Pinecone (cloud-based). Langchain has wrappers around the main vector stores that accommodate common methods for similarity search and clustering, so I'd suggest using implementing them with Langchain in case we want to change the library in the future.

Regarding the embeddings strategy, I'd use here a high quality tokenizer to make sure that the most semantic value is captured (OpenAI's models, basically).

[ireneisdoomed]

I did a very similar exercise one year ago with a "basic" BERT model Tokenizer. The code for inspiration can be seen here: https://github.com/ireneisdoomed/random_notebooks/tree/main/text_similarity

[ireneisdoomed]

We have implemented:

• [x] The extraction of the embeddings of all the disease labels that represent one EFO ID
• [x] The construction of a vector database in Pinecone that we can query with an API token. The URL of the database: phenomena-38e5d95.svc.us-west4-gcp.pinecone.io
• [X] Code to find the most likely EFO ID to a string of an user by embedding the string and querying the Pinecone db

Todo: implement in the FE like a confidence-based ranking system, that shows suggested terms and allows the user to choose from the top 5 most similar EFO IDs.