Write plan to migrate tenants off us-east and into GovCloud

[dlapiduz]

We need to write up a timeline that allows people to migrate their apps at their own pace.

Ideally it would look something like this:

• X/10: First version of GovCloud cloud.gov. Endpoint created for API at api.gov.cloud.gov (?)
• X/29: GovCloud ready to receive tenants.
• X - X+2: Tenants self-migrate
• X+2/1: Switch endpoints api.gov.cloud.gov -> api.cloud.gov, api.cloud.gov -> api.old.cloud.gov (?)
• X+4/1: Terminate us-east environment (?)

We need to write up comms about why we are doing this and what we are going to do to help people move but we need to make sure that this is smooth for everyone.

@mogul @brittag @18F/cloud-gov What do you think?

[afeld]

This is going to be a major pain in the butt for people (copying databases and switching domains over, etc.)...wonder how much we can automate, or at least provide scripts for?

[mogul]

Diego's described one way to do it (eg entirely customer-driven via redeployments and individually-driven data migrations) and to be honest that's the cleanest way, but customers who are not using CD to manage their deployment may get sore about it, and the data migration is still on them regardless. There may be ways we can make the process easier; we'll get there in time and will refine this story as it floats to the top of the backlog.

[dlapiduz]

@afeld I think that we can write up tools to help them but we can't run 2 environment on the long run.

@mogul right, this is the "cleanest" way. We can try to migrate stuff but its going to be a mess.

I think that we should work in batches of moving things but basically the main point of this story is that an announcement needs to be made so we don't have this situation getting worse over time. I think that part of the announcement needs to be about how we are going to help users move.

[brittag]

Yeah, it makes sense to me to announce this relatively soon so that people can start planning/preparing (and also can start new projects in ways that minimize future migration pain, if possible). I'd approach it like this:

• [ ] Write up the timeline of changes (and briefly explain the reason) in a clear and helpful way for customers. Gently reassure people that this is just a one-time request (if that's the case).
• [ ] Write down brief basic recommendations for next steps they may need to take. Probably: typical tasks they will need to do for existing projects + how to set up new projects to minimize migration labor.
• [ ] Show all of that to a couple of customers to learn about any additional information/recommendations/tools/reassurance they need, then try to add those things.
• [ ] Test a little more, or just announce if it seems fine.

And:

• [ ] To help us incrementally systematize how we communicate with customers about product changes, decide ahead of time how we'll announce this to people and format the content for those ways (even just making a list of "email, cloud.gov update post, #cloud-gov-support announcement").
• [ ] Find and update any docs that also need to communicate these changes.

[dlapiduz]

We should plan to announce this around the time the assessment ends on 6/24. That way new tenants can deploy to govcloud directly.

If I had to make a rough plan I'd say:

• cloud.gov properties (move all the urls)
• sandboxes (terminate sandboxes in e/w)
• 18F owned properties (tock, myusa,etc)
• GSA stuff (oasis, calc)
• EPA?
• College Scorecard?

Also, we should try to get Diego as soon as possible in GovCloud so tenants don't have to move twice (and there is something "new" waiting on the other side). This is being discussed here: https://github.com/18F/cg-atlas/issues/16

[dlapiduz]

Also, this plugin will become handy to people wrangling more than one environment: https://github.com/guidowb/cf-targets-plugin

[mogul]

I agree we should try to arrange this around the deployment of Diego, so that people get a benefit rather than just the inconvenience: "Now you're going to be on GovCloud for additional ATO goodness, and you'll be able to opt into a backend which enables you to SSH directly into apps!" Will be doing that with high priority, but I expect some complications we can't foresee will arise; setting a 6/24 target seems premature.

[dlapiduz]

I am not saying we should move everyone on 6/24... I doubt we will have diego production ready then.

[mogul]

Notes on this topic from retro today, to end up in separate issues later:

• Diego wants us to get a draft plan up and announced that makes it clear what's going to happen, how users can transition now, even ahead of DEAgo being available, if they wish, and provides guidance on how to do that (eg dumping/restoring their service instances, changing API endpoints).
• Team wants to help people understand the inventory of what's their responsibility to move vs ours.
• Team wants to try doing a contingency plan exercise to see how much we can automate bringing everything up in a neighboring AWS account
• Team wants to work on secrets management as part of this
• We'll limit our investment to a spike initially to avoid going overboard in automating everything

[mogul]

From today's grooming, here's a draft backlog

Outline of the transition based on the discussion today:

• GovCloud ready to receive tenants (internal apps all deployed and functional)
• Write comms about why we're doing this, what benefits they'll get, and what we'll do to help people move smoothly. Prompt for questions!
• Draft user-empathetic comms with lots of context for each of the milestones below
• Announce the plan to E/W users
• Provide guidance on capturing the existing E/W deployment with manifests for repeatability
• Provide guidance on migrating stateful services
• Provide guidance on DNS transition
• Write tools to help with the most common/difficult migrations
• Provide guidance on minimizing the interruption in service
• Invite users to GovCloud (pre-populate roles?)
• Mirror orgs, spaces, roles, and quotas (in the interim: strip Manager roles? mirror changes?)
• Mirror ELBs and certs, and stop accepting changes to them in E/W
• Tenant self-migration period
• Chase up laggards ("no app left behind!!" -jbarnicle)
• Announce endpoint switch, then switch endpoints api.fr.cloud.gov -> api.cloud.gov, api.cloud.gov -> api.old.cloud.gov -

Termination of the us-east environment is explicitly out of scope for the PI, although desirable ASAP. We want to keep it around for the inevitable "oh crap, I left something behind!" requests.

@dlapiduz, @brittag, @afeld, @berndverst, can you review and ask questions/make comments in the next couple days? After that we'll start breaking this out into the backlog.

[brittag]

This makes sense to me overall! I'll have suggestions for the individual items (the initial announcement should include an approximate timeline of planned changes, etc.) but I can file those after they get broken out.

[afeld]

Just heard that the legacy DEA backend is being phased out as of the end of 2016, in case we needed additional incentive to move folks.

https://docs.google.com/document/d/1L6behogKHG5GCs1suB-BLplb6_xB9L4LK0U0pAkuhYc/edit

[datn]

My thoughts on dividing this task further:

1) enumerate all clients with services/assets/datastores/certs to be moved

2) document migration process for each type of service/asset/datastore/etc

3) create migration plan for each client incl. contacts for client, service dependencies and tasks to be performed by client

4) identify missing infra in govcloud env, if any

5) define process for backups

I stopped short of creating actual issues for these because they potentialy imply some cross-squad work (like obtaining client contact info), and after discussing it with John I thought a comment like this would be better -- Bret, I think we could use some guidance about how you envision creating cross-squad work under a single task.

[brittag]

Reference to Google Doc as well: https://docs.google.com/document/d/11aO1sPeCQXEbzb1l7FgQ2Z0wkboZpMxrmrYaw72C31U/edit

[mogul]

Everything we've noted above has now been broken out into separate issues... We're going to close this one and just work from the backlog now.