Describe the process(es) for continuous integration (CI) of KP data into an aggregate knowledge graph (AKG)

[patrickkwang]

Questions to answer:

• What interface(s) do KPs need to provide in order to enable such integration?
• What interface(s) should the AKG provide for ARAs?

Questions not to answer:

• What technologies should be employed (GitHub Actions, cron jobs, Neo4j, etc.)? Except where it impacts the aforementioned interfaces, this can be decided by the AKG developers.
• How should the biomedical data from KPs be fused? This is a data modeling issue, and should be addressed collectively, but not here.

[dkoslicki]

Additionally, some "legacy KG's" (eg. ARAX/KG1 and ARAX/KG2) utilize information ETL'd from non-translator KS's. A process for CI of non-Translator KS's may also want to added to this issue (or pulled out as a separate one).

[cbizon]

@cmungall @deepakunni3 what kinds of interfaces do you think would make sense here?

[cmungall]

My answer is predicated on my assumption that there will be many purpose specific AKGs (as well as potentially a Translator 'uber' AKG). The assumption is that a local AKG is pragmatically required for certain inference algorithms; the local AKG does not need to be the union of all KGs, but it should have the parts its needs locally to avoid network latency on high-frequency traversal operations.

The source KGs may be available via queries (one-hop APIs, query interfaces such as sparql/cypher) or via dumps.

I think the operations to enable integration should be:

• extract
  • extract a complete source KG
  • extract a KG based on a "seed" (e.g. list of IDs)
• normalize
• filter
  • include/exclude a subset by white/blacklist of
    • edge labels
    • node types (e.g. just get a disease-to-phenotype edges from a trusted KG)
    • edge or node sources
    • evidence types (e.g. remove automated inferences)
    • other properties
• merge
  • basic 'concatenation'
  • merge graphs and also perform clique merge on nodes using node normalization
• validate
  • does merging the graphs introduce incoherency or errors?
• diff

I would also point people at our command line tool robot (http://robot.obolibrary.org/) for making aggregate ontologies. The distinction between ontology and KG is fuzzy. However, I think the use cases and approaches are analogous.