Task list and timing to complete MGI import

[ukemi]

ping @loricorbani

• [x] 1. Mar 1-4/Repeat QC roundtripping analysis to ensure data integrity after import into MGI (David, Dustin, Lori, Seth, Sierra)
  • [x] 1a Mar 1-2/Dustin or Seth migrates the Noctua production data into noctua-models dev .
  • [x] 1b Mar 1-2/Dustin imports MGI production data into noctua models dev (http://www.informatics.jax.org/downloads/custom/go_cam_mgi.gpad and http://www.informatics.jax.org/downloads/custom/mgi.gpi)
  • [x] 1c Mar 2/Dustin creates files for Lori from the MGI pipeline branch (actually "wormbase 238"), made available on skyhook, to import into her test environment.
    Lori: Concatenate these make File 2 ( http://skyhook.berkeleybop.org/issue-238-wormbase-test-pipeline/products/annotations/noctua_mgi.gpad.gz and http://skyhook.berkeleybop.org/issue-238-wormbase-test-pipeline/products/annotations/noctua_pr.gpad.gz)
  • [x] 1d Mar 2-3/Lori picks up Dustin’s files and loads them into Lori-test with all the other MGI loads. Produces these outputs:
  • [x] mgd.gpad
  • [x] mgi_nonoctua.gpad (mgd.gpad minus Noctua and minus PAINT)
  • [x] 1e Mar 2-3/MGI files are put on the ftp server---- mgd.gpad & mgi_nonoctua.gpad to Dustin/Seth. File 3. Used for GOC pipeline, AmiGO2 etc. Keep in mind, this step is only for testing. Once everything goes live, we will no longer supply MGI-curated annotations to the GOC. It will be a one-way street into MGI from curators working in Noctua.
  • [x] 1f Mar 3/Dustin reloads the file from Lori (as in 1b).
  • [x] 1g Mar 3/Dustin creates a new file from the round-tripped annotations. File 4 ( http://skyhook.berkeleybop.org/issue-238-wormbase-test-pipeline/products/annotations/noctua_mgi.gpad.gz) (as in 1c).
  • [x] 1h Mar 4/Sierra runs the diff on files 2 and 4 (step 1c/gpad vs step 1g/gpad)
  • [x] 1i Mar 4/David will check the diff created by Sierra. The only differences should be the 100 or so annotations that will be regenerated due to another round of reasoning over some ‘regulates’ annotations. There was only one difference in the number of times a GO ID was used. This made David's job easy!
  • [x] 1j Mar 4/If all is good move to step 2 below otherwise repeat this roundtripping test again after troubleshooting.
• [x] 2 Mar 4 or Mar 7/Dustin loads noctua-models dev with MGI production GPAD annotations for curator training and MGI front-end testing (http://www.informatics.jax.org/downloads/custom/go_cam_mgi.gpad and http://www.informatics.jax.org/downloads/custom/mgi.gpi); Dustin reruns MGI test pipeline (actually "wormbase 238")
• [x] 3 Mar 7/Lori: reloads the data from the MGI test pipeline (actually "wormbase 238") pipeline (http://skyhook.berkeleybop.org/issue-238-wormbase-test-pipeline/products/annotations/noctua_mgi.gpad.gz and http://skyhook.berkeleybop.org/issue-238-wormbase-test-pipeline/products/annotations/noctua_pr.gpad.gz). This is the step that will occur as a regular job once the switchover is made.
• [x] 4 Mar 7/Lori: load Lori’s backup; load it into Test server; run FEWI pipeline
  David: This load represents ‘the real thing’. Lori will create all the MGI reports that we will produce after the switchover, including the MGI ‘slim’ annotation files that only contain non-Noctua loads.
• [x] 5 David: Tell the MGI curators to stop curating in the MGI interface. Note that we still don’t want to disable it at this point because GOC QC will still be happening on ‘legacy’ annotations. This cannot be enforced.
• [x] 6 MGI/Mar 8,9,10: curator training using noctua-dev that contains MGI annotations (i.e. noctua-models dev). ( second week of March during the NO-GO calls week).
• [x] 7 Mar 9: Lori: test turn off MGI/PWI/GO Annotation; read-only on Test Server
• [x] 8 Mar 11 : Lori: install new PWI tag on MGI Production. - [ ] 7 MGI/Mar 11: turn off MGI/PWI/GO Annotation; that is, read-only/no more changes allowed at MGI ( @krchristie will test this modification with @loricorbani)
  • This will mean that Annotations that were created at MGI will remain static and no GOC QC can be done on them until they are in Noctua production.
  • • All new annotations will be made in Noctua production.
  • All annotation review/changes of original MGI annotations will be suspended until the switchover is completed.
• [x] 9 Mar 29 Check final schedule MGI Production release with GO group
• [x] 10 Mar 29 Kimberly or Suzi need to send out notice re Noctua down time
• [x] 11 Mar 31: Lori: turn off all GO loads
• [x] 12Mar 31: GO switchover day--GO completes transition to new MGI curation stance
  • [x] Dustin produces final TTL files and PRs them into noctua-models master
    (Dustin processes files to make final PR into noctua-models master, similar to 1b (new go_cam file))
    https://github.com/geneontology/noctua-models/pull/224
  • [x] go-site gets PR for all necessary metadata (i.e. new MGI sources, noctua merging, and pipeline changes(?))
    https://github.com/geneontology/go-site/pull/1819
  • [x] pipeline gets any PR improvements back to production branches (needs to be checked)
    In a side-by-side examination, I believe that the main change, an update to ontobio, has already been lapped by the production branches; would like a second opinion from @sierra-moxon or @dustine32; confirmed by @dustine32
  • [x] noctua production stop; flushed; changes committed
  • [x] noctua-models PR taken (after commit above)
  • [x] noctua production starts
  • [x] all pipeline/metadata changes taken--production pipeline and production noctua is now operating on new MGI curation stance
  • [x] snapshot pipeline run manually started; release run delayed
  • [x] Dustin processes new go_cam file; outputs = noctua_mgi.gpad.gz, noctua_pr.gpad.gz
    /data/downloads/snapshot.geneontology.or
    /products/annotations/noctua_mgi.gpad.gz
    /data/downloads/snapshot.geneontology.org/products/annotations/noctua_pr.gpad.gz
• [x] 12a Apr 1: David sanity checks MGI models in Noctua production
• [x] 13 Apr 4: DaveM: makes Git tags
• [x] 14 Apr 4: Dave : loads Test database from newest MGI Production Backup
• [x] 15 Apr 4: Dave : runs GO migration
• [x] 16 Apr 4: Dave : runs FEWI Pipeline
• [x] 17 Apr 4: Joel : run MouseMine Pipeline
• [x] 18 Apr 5: David : verify test-PWI, verify test-FEWI
• [x] 19 Apr 6: Install/Run on MGI Production
• [x] 20 Apr 6: Turn Production MGI-GO loads back on
• [x] 21 Apr 7????: Dave: loadadmin tag; turn off go_cam script, remove custom/go_cam files
• [x] 22 Apr 8 Lori and David go out for a beer

[dustine32]

@ukemi Beautiful ticket! Just FYI, I'm currently working on step 1b.

[kltm]

@loricorbani @ukemi Clarifying for 1d and 1e that we do not use GPADs for pipeline ingest and will need GAFs.

[kltm]

@loricorbani @ukemi For 1e, I believe we discussed that these would be at a publicly available location and not files through email. (As well as GAF formatting.)

[kltm]

@loricorbani @ukemi For 8, there is currently some time travel. What should the correct date be there? Please feel free to correct directly in the list at the top.

[kltm]

@loricorbani @ukemi For 11, I'm not sure why there are files listed here?

[dustine32]

@loricorbani Step 1c is complete and the output GPAD files are now available in the skyhook/issue-238-wormbase-test-pipeline location.

[leemdi]

1d/1e files are here: http://www.informatics.jax.org/downloads/custom/noctua/ gene_association.mgi gene_association_nonoctua.mgi gene_association_pro.mgi mgi.gpad mgi.gpi mgi_nonoctua.gpad

The "nonnoctua" files are those that contain no Noctua annotations and no PAINT annotations. that is, the Annotations MGI loads from GOA/Human, GOA/Mouse, Rat, etc.

fyi, GO:0007569 failed because it was recently obsoleted but was in noctua_mgi.gpad file. Let me know if you want to send me a new file or just continue with this test.

[ukemi]

11 must have been a weird cut and paste error from my transfer of Lori's ticket. Dustin needs to produce the files that Lori will pick up for our production load.

[dustine32]

Thanks @loricorbani! The new custom/noctua folder should work great. Question: is gene_association_nonoctua.mgi simply the GAF version of mgi_nonoctua.gpad? If yes, then awesome, as GAF is the format we'll need for that file when we run the GO pipeline after the MGI import into Noctua.

On obsolete term GO:0007569, this is very weird. I couldn't find this term in the noctua_mgi.gpad file hosted at http://skyhook.berkeleybop.org/issue-238-wormbase-test-pipeline/products/annotations/noctua_mgi.gpad.gz as of 2022-03-03 7:30AM Pacific. Are you using an older version of this file perhaps?

[leemdi]

Dustin,

Ok, let’s try again…new files are available here:

http://www.informatics.jax.org/downloads/custom/noctua/

Lori

[dustine32]

@loricorbani Rad! No validation errors on my end. I'm producing the new models now (step 1f) and will let you know when the new GPAD files are available on skyhook (after step 1g is complete, likely late, late tonight or early tomorrow Friday).

[sierra-moxon]

https://github.com/sierra-moxon/mgi_import_qc/tree/main/03_02_2022 - diffs for file2 and file4 look pretty good. :)

[ukemi]

Wow! This is awesome. The changes that @balhoff made to the GPAD output must have gotten rid of the regulates differences.

[dustine32]

Noting where we're at:

• @ukemi Step 2 - new models have been generated and pushed to dev. The noctua-dev environment has been reloaded so the new models are now viewable.
• @loricorbani I'm running the issue-238-wormbase-test-pipeline right now to generate the new GPAD files. They should be available on skyhook early tomorrow 3/8.

[leemdi]

I am at Step 3 - reload of MGI test using Dustin's Step 2 file

[ukemi]

Great! So now we have two parallel tasks occurring. @ukemi will begin the training session with the MGI curators today using dev as the workspace. @loricorbani will reload MGI and push things all the way through so @ukemi can check the MGI web display of GO annotations in a testing environment. The checking may take a few days because I suspect I will be occupied with the training.

[leemdi]

step 3 : done; ready for David to review: MGI Test Server

[leemdi]

step 8 Mar 11 : Lori: install new PWI tag on MGI Production : done MGI editing interface (PWI), GO Annotations is read-only. MGI curators must use Noctua for all GO Annotations.