Deploy with optional VPC

[chrisammon3000]

Description

It is now possible to deploy gfe-db with or without a VPC. If deployed without a VPC it can be configured to use an existing VPC and resources using environment variables.

It is also easier to deploy multiple instances of gfe-db to the same account by applying the stage convention to .env files → .env.dev, .env.qa, .env.prod etc.

A new safeguard is in place when running make commands to ensure that changes are deployed to the correct environment → STAGE=dev make deploy.

Summary of Changes

• Allows optional VPC and networking resources deployment using environment variable

  CREATE_VPC=true/false # If false, VPC_ID and PUBLIC_SUBNET_ID are required
  VPC_ID=<vpc_id> # Required if CREATE_VPC=false
  PUBLIC_SUBNET_ID=<public_subnet_id> # Required if CREATE_VPC=false

• Environment variables are now segregated by stage

  .
  ├── .env.dev
  ├── .env.prod
  ├── .env.qa

• Makefile usage now requires the stage to be specified to avoid deploying changes to the wrong environment

  STAGE=dev make deploy

• Updated README

Usage

Using external VPC

1. Retrieve the VPC ID and subnet ID from the AWS console or using the AWS CLI.
2. Purchase or designate a domain in Route53 and create a hosted zone with an A record for the subdomain. You can use 0.0.0.0 for the A record because it will be updated later by the deployment script.
3. Acquire a subscription for the Bitnami Neo4j AMI through AWS Marketplace.
4. Install prerequisites.
5. Set environment variables including the ones from the previous steps. You must store these in a file named .env.<stage>, for example .env.dev or .env.prod:
   • CREATE_VPC=false
   • VPC_ID
   • PUBLIC_SUBNET_ID
   • HOSTED_ZONE_ID
   • HOST_DOMAIN
   • SUBDOMAIN
   • NEO4J_AMI_ID
6. Check the config JSONs (parameters and state) and edit the values as desired.
7. Run STAGE=<stage> make deploy to deploy the stacks to AWS.
8. Run STAGE=<stage> make database.load.run releases=<version> to load the Neo4j, or STAGE=<stage> make database.load.run releases=<version> limit=<limit> to run with a limited number of alleles.
9. Run STAGE=<stage> make database.get.credentials to get the username and password for Neo4j.
10. Run STAGE=<stage> make database.get.endpoint to get the URL for Neo4j and navigate to the Neo4j browser at the subdomain and host domain, for example https://gfe-db.cloudftl.com:7473/browser/.

Creating a new VPC

1. Purchase or designate a domain in Route53 and create a hosted zone with an A record for the subdomain. You can use 0.0.0.0 for the A record because it will be updated later by the deployment script.
2. Acquire a subscription for the Bitnami Neo4j AMI through AWS Marketplace.
3. Install prerequisites.
4. Set environment variables including the ones from the previous steps. You must store these in a file named .env.<stage>, for example .env.dev or .env.prod:
   • CREATE_VPC=true
   • HOSTED_ZONE_ID
   • HOST_DOMAIN
   • SUBDOMAIN
   • NEO4J_AMI_ID
5. Check the config JSONs (parameters and state) and edit the values as desired.
6. Run STAGE=<stage> make deploy to deploy the stacks to AWS.
7. Run STAGE=<stage> make database.load.run releases=<version> to load the Neo4j, or STAGE=<stage> make database.load.run releases=<version> limit=<limit> to run with a limited number of alleles.
8. Run STAGE=<stage> make database.get.credentials to get the username and password for Neo4j.
9. Run STAGE=<stage> make database.get.endpoint to get the URL for Neo4j and navigate to the Neo4j browser at the subdomain and host domain, for example https://gfe-db.cloudftl.com:7473/browser/.

Next Steps

• Implement CloudFormation change sets for more control and visibility of deployments
• Commit-level tracking of new data (ready)
• Integration of notifications, validation and backup functionality to the state machine (ready)
• Unit & integration testing