GOC / GOA Joint Pipeline

kltm commented 4 months ago

Project link

https://github.com/orgs/geneontology/projects/159

Project description

As part of the discussion from the UNC GO meeting, we decided to go ahead with a joint GO/GOA pipeline. The intended division of labor would be:

GO Central produces:

Ontology
User, groups, dbxrefs, and other metadata
Curation tool resources
- PAINT annotations
- Noctua-derived annotations (standard and GO-CAM)
- "Automated upstreams" (MGI)
Derived products to drive GO Central interfaces and downloads
- Solr index, Blazegraph, GO API and required files, etc.

GO Central consumes from GOA:

GAFs and GPAD/GPI(?) files from GOA
GOA annotation error reports

GOA produces:

GAFs and GPAD/GPI(?) files from GOA
GOA annotation error reports

GOA consumes from GOC:

Ontology
Metadata
Noctua GPADs
PAINT
"Automated upstreams" (MGI)

This joint pipeline would solve several ongoing issues that GO faces wrt the pipeline

speed and stability of releases
detailed error reporting
ontology synchronization
consistence of data products for the GO community
infrastructure expense (leveraging EBI compute)

PI

Chris

Product owner (PO)

Pascale

Technical lead (TL)

Seth/Alex

Other personnel (OP)

Dustin (at need)

Technical specs

Rolling technical discussions w/Alex and Pascale: https://docs.google.com/document/d/1klJX7do0JGaWkFY3fcGsXBXGPPG9a42y_4YM7arCPK4/edit?pli=1 Explanatory slides: (IN PROGRESS)

Other comments

N/A

kltm commented 4 months ago

Tagging @pgaudet, noting that this not "officially" exists.

kltm commented 4 months ago

Items to be added to project at need.

We still need to work out a lot of technical details (e.g. file granularity, downloads, timing and freezes wrt ontology and curation tools for release processing, etc.). Early days.

kltm commented 1 month ago

Current Pascale working doc: https://docs.google.com/document/d/1klJX7do0JGaWkFY3fcGsXBXGPPG9a42y_4YM7arCPK4/edit?pli=1

geneontology / project-management