create EC2 instance which mirrors current pipeline machine

[sheridancbio]

• [x] need appropriate users (cbioportal_importer)
• [x] need clones of codebases / credential files
• [x] need matching system binary tools (java, python, mysql, ...)
• [x] clone needed data repositories (possibly on a persistent volume) for : dmp_importer, cmo_importer, triage_importer [not genie, not public]

[averyniceday]

Followed instructions here: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/docker-basics.html

Then also added sheridan and wanga5 to docker group so we can execute

[averyniceday]

I've tested an import from the new machine to the database and that has completed successfully. I had to create a hacked importer to stop it from connecting to Redcap/CDD which we don't have access to yet. There's already a request sent in by Rob for opening up those ports. Once approved we can move to testing the entire run.

Things left:

• [x] script for installing dependencies
• [x] clone mercurial repositories

[sheridancbio]

This work required making a request to open ports in the MSK firewall for connections coming from the AWS EKS cluster address ranges. A request was submitted May 10 to allow connection coming from private network addresses: digits AWS EKS dev cluster subnet A CIDR 10.1.16.0/25 [range 10.1.16.0 -> 10.1.16.127] any http, https, mysql, db2, ssh digits AWS EKS dev cluster subnet B CIDR 10.1.16.128/25 [range 10.1.16.128-> 10.1.16.255] any http, https, mysql, db2, ssh digits AWS EKS dev cluster subnet C CIDR 10.1.17.0/25 [range 10.1.17.0 -> 10.1.17.127] any http, https, mysql, db2, ssh digits AWS EKS prod cluster subnet A CIDR 10.1.20.0/25 [range 10.1.20.0 -> 10.1.20.127] any http, https, mysql, db2, ssh digits AWS EKS prod cluster subnet B CIDR 10.1.20.128/25 [range 10.1.20.128 -> 10.1.20.255] any http https, mysql, db2, ssh and going to internal msk network hosts / ports: pipelines.cbioportal.mskcc.org 22, 80, 443, 3306 ssh, http, https, mysql ddp.mskcc.org http, https ramen.cbio.mskcc.org 22, 80, 443 ssh, http, https dashi.cbio.mskcc.org 22, 80, 443, 8080, 28080, 58080 ssh, http, https dashi2.cbio.mskcc.org 22, 80, 443, 8080, 28080, 58080 ssh, http, https lynx.mskcc.org 9770 http, https draco.mskcc.org 9775 http, https plcrdbd2.mskcc.org 1526 oracle database client driver pidvudb1.mskcc.org 51013 db2 database client driver github.mskc.org 443 https

Status on May 14:

Request has been acted on by digits, we have tested connections from various points in the source address block ranges for the dev cluster. The 3306 mysql port to pipelines cannot be tested because there is a firewall on the target machine not allowing incoming connections. Also ports 80 and 443 and 8080 on dashi and dashi2 are not deploying services we need currently. Aside from these, only the db2 and oracle ports seem to have not been opened to the dev cluster.

Testing of the prod cluster cannot be done until we are granted authorities to create keypairs for launched ec2 instances.

[averyniceday]

Isolated Testing

• [x] import into EKS db
• [x] redcap pipeline
• [x] CRDB fetch
• [x] DDP fetch
• [ ] Darwin fetch
• [ ] CVR fetch
• [x] local build [ rework the build-importer-jar script ]

[sheridancbio]

We believe the darwin fetcher is able to reach the darwin db2 port ... but we are seeing a failure to connect from the db2 driver: [2021-05-18 19:47:44.272] boot - 23430 ERROR [main] --- AbstractStep: Encountered an error executing step mskimpactTimelineBrainSpineStep in job mskCaisisJob java.lang.RuntimeException: DB2 SQL Error: SQLCODE=-1060, SQLSTATE=08004, SQLERRMC=DVCBPAPS, DRIVER=4.15.134 at com.querydsl.sql.SQLQueryFactory$DataSourceProvider.get(SQLQueryFactory.java:47) \~[querydsl-sql-4.0.9.jar!/:?] at com.querydsl.sql.SQLQueryFactory$DataSourceProvider.get(SQLQueryFactory.java:34) \~[querydsl-sql-4.0.9.jar!/:?] at com.querydsl.sql.AbstractSQLQuery.connection(AbstractSQLQuery.java:660) \~[querydsl-sql-4.0.9.jar!/:?]

We think this may require an adjustment of the firewall, or the incoming host authorization list for the db2 system. We will need to reach out to darwin for them to allow incoming connects from any ip address in the ranges possible for the eks development and production clusters.

[sheridancbio]

Follow up tasks identified during review:

• follow up with darwin db2 connection whitelist : Mike Eubank for incoming connection to db2 on pidvudb1.mskcc.org (port 15031?) --- >> email Ta-ching Chien... and cc Mike and Benjamin
• adjust automation_environment.sh script and build script for the new server
• mercurial problems --- figure out whether to upgrade mercurial client version --- adjust the history of cbio-portal-data --- or do a full switch to git and dump mercurial now
• complete fetcher testing

[sheridancbio]

remaining tasks

• [x] clone cbio-portal-data and move in (or clone in) the other repositories needed (clone of cbio-portal-data in progress)
• [x] cvr import scripts: comment out the consumption step on the dev server
• [x] cvr import scripts: make sure to enable the "test" mode during the fetch operation [prevents requeues]
• [x] automation environment script .. a clean / maintainable and minimally disruptive approach for setting up a new pipelines server while using our checked in scripts
• [x] add mercurial tool version 4.0 download .rpm module and installation into the setup system script
• [x] need to enable redcap database backups (see repo redcap-snapshot) via crontab this is just an export from redcap with a mercurial checkin, doesn't really need to be tested in the full run, especially since we are committing the same files in production
• [x] setup system script : setting up .ssh key used for cloning / updating github.mskcc.org repositories ... the current key in use on pipelines is in \~/.ssh/id_rsa_shahcompbio. ~/.ssh/config file must be updated to have an entry for the github server like:

  Host github.mskcc.org
  HostName github.mskcc.org
  User res-ski-cbioportal
  IdentityFile ~/.ssh/id_rsa_shahcompbio

  we have figured out that by using [HostFingerprints] in the \~/.hgrc file, we can accept the expired certificate on pipelines and can avoid using the "-insecure" option ... which was causing a "Connection reset by peer" error for http: based urls. Now, we are using https: protocol URLs and clone and pull work smoothly. To configure the machine to do this, the fingerprint here needs to be added to the \~/.hgrc file like this: (note : case is sensitive)

• [x] setup system script : set up the /home/cbioportal_importer/.hgrc file
  • auth section entries
  • fingerprint section
  • username

    [hostfingerprints]
    data.cbioportal.mskcc.org=63:C9:3B:00:FC:24:D5:8D:42:53:54:BB:62:54:46:73:32:A7:AA:2E

• [x] a new script perhaps to clone cbio-portal-data and all embedded repositories [ already started ]