Complete production transfer from Stanford to Berkeley

[kltm]

TODO/gross timeline from the STanford meeting with @stuartmiyasato @cmungall :

• [ ] hibernate (RO) svn cvs
• [ ] stop all pipline tasks for the above
• [x] (tanya/TAIR) look at archiving jira into elastic search? Will either push into GitHub or mothball
• [x] get static site creation crawler script for s3 (ben)
• [ ] unwinding into s3
  • [ ] archive.geneontology.org (in process)
    • [ ] once it has been unwound, dns switches to s3
  • [ ] secure domains for cvs + svn archive
  • [ ] start process to unwind svn and cvs monthlies into s3
  • [ ] turn on new site
• [ ] explore aws x-fer of machines for wiki, drupal, etc.
  • [ ] maybe maintained for a wee bit more
  • [ ] look at dumping different versions of the drupal, make it static
  • [ ] create or keep wiki EC2 GitHub wiki also an option: https://github.com/geneontology/go-site/issues/313#issuecomment-285170973
• [x] ask mike about mail lists, okay to use stanford mail for the time being no overhead, right?
• [x] x-fer domain, and see what that actually might involve stuart did for AGR, should be aight
  • [x] this will (probably) include keeping the stanford mail list maps
  • [x] go-helpdesk will need some TLC? gmail vacation message?
• [x] mike's script?
• [x] new jenkins
• [x] turn on new pipeline

(Epic, sorry. We can add concerned users as we go.)

This ticket can be closed when the bulk of production has been moved to Berkeley.

[stuartmiyasato]

Confirmed with Mike Cherry on a couple of points:

• Mike's filtering script will continue to run until the replacement is ready for production.

• Mike is okay with keeping the MX record for geneontology.org pointing to our local mail server at Stanford, so forwarding addresses @geneontology.org will continue to work as they have in the past.

[kltm]

For better or worse, the GO Helpdesk issue has been triaged, assuming my inbox doesn't detonate.

[kltm]

@stuartmiyasato We are crawling forward on this; I can now hopefully start devoting more attention to the issues. I'm currently looking at how to do a new Jenkins installation, so we pipeline is next on the plate.

In the meantime, I'm trying to finish off the helpdesk. As a final piece there, I was wondering if I could get the following CNAME placed?

help.geneontology.org -> geneontology.github.io

This will be one piece of trying to spread the infrastructure out a bit more.

[stuartmiyasato]

@kltm I forgot to mention to you that I'm on vacation all of this week, so I probably won't be very responsive to requests until I get back into the office. Today is a quiet day at my parents' house in LA, so I'm able to get to this. :) help.geneontology.org should be set now:

miyasato% host help.geneontology.org
help.geneontology.org is an alias for geneontology.github.io.
geneontology.github.io is an alias for github.map.fastly.net.
github.map.fastly.net has address 151.101.24.133

[kltm]

@stuartmiyasato Whoops! Sorry to bother then. At least you can rest well knowing that the GO Helpdesk has a new home. It looks like it has already propagated, so done and done!

@monicacecilia Moving forward, any reference to the GO Helpdesk should be http://help.geneontology.org as the most correct "forever home". I'll make updates as I run into them.

[monicacecilia]

@kltm copy that.

[kltm]

Given that we're trying to free up as much overhead as possible, I did a little experiment into what would happen if we converted the current mediawiki installation into GH wiki: : pandoc -s -f mediawiki -t markdown_github -o /tmp/gw.txt /tmp/mediawiki.txt The results are surprisingly good--I think that if we did it right, most of the linking and structure could be preserved.

Now, that said, I really like mediawiki and it is super easy to maintain; but the option is out there.

[cmungall]

From experience trying to migrate wikis like that of google code, are there not a lot of edge cases? E.g. how media like images and ppt is handled.

[kltm]

Well, I've dug around a bit and tried some scratch stuff with the GitHub wiki, and I must say that while the text/link preservation works a treat, the media preservation would be quite poor, with no obvious way to deliver stuff internally from the wiki repo--they seem to have specifically set out to do that. That would be a roadblock for a complete port, but not the end of the world if we decide to abandon mediawiki (again, not advocating, just looking) in general.

[kltm]

@stuartmiyasato Sorry, slow progress here. I was wondering if we could get the CNAME build.geneontology.org pointed to wok.lbl.gov? I'll be initially figuring out the pipeline for AmiGO (although we have to get monitoring and logging setup here as well); once that is done, we might want to go ahead and flip AmiGO over here.

[stuartmiyasato]

@kltm Okay, this is now in place.

-bash-3.2$ host build.geneontology.org
build.geneontology.org is an alias for wok.lbl.gov.
wok.lbl.gov has address 131.243.192.20

[kltm]

Thank you! Back to the salt mine for me.

[kltm]

A temporary holding list while we consider the next steps:

• AmiGO 1.x as discussed above
• AmiGO 1.x pipeline any remaining functioning MySQL and other dumps
• GO wiki I believe that this is in AWS, EC2, with snapshots?
• ftp I believe this is just a view of the filesystem of one of the below?
• CVS ontology and files
• SVN ontology and files [still in use as a stopgap for some files]
• SVN viewvnc [still in used as a stopgap for some files]
• archive.geneontology.org a fairly large filesystem archive of GO product until fairly recently

[kltm]

Talking to @stuartmiyasato , all remaining items are now better represented by issues in the migration project: https://github.com/orgs/geneontology/projects/39 This planning and execution super is no longer needed.