search

deptofdefense / move.mil

The entry point website to the Defense Personal Property System (DPS).
https://www.move.mil
Other
15 stars 15 forks source link

Do not automatically blow away and re-seed the database in production #403

Open jamesatheyDDS opened 6 years ago

jamesatheyDDS commented 6 years ago

This change removes the rails.config file, which currently runs db:setup on every app deploy.

This script is run every time an EC2 instance is spun up as well as every time the app is deployed to an already running instance. In production, it's problematic to blow away the DB and re-seed it every time this happens. Seeding takes multiple minutes, and during that time, various features of the site stop working while they wait for their data to appear in the DB, especially the Locator Maps and the PPM Estimator.

Once this reaches production, changes to the static data in the DB will require either running db:setup manually, or making specific ad-hoc DB changes.

jamesatheyDDS commented 6 years ago

FYI, I have already set the EB_ENVIRONMENT variable appropriately in both move-mil-prod and move-mil-staging.

aileendds commented 6 years ago

I'm a bit confused about how this works. Does this mean we should stop using seeds to make changes to the DB? Or does db:migrate seed? How does it know if it should delete an existing row? Also, I think staging should mirror production. There might be differences between staging and prod that arise because of this change.

jamesatheyDDS commented 6 years ago

@aileendds We can continue to use seeds to make changes to the DB, but in production, we should just run bin/rails db:setup manually when we need to apply a change to the database. db:setup is overkill for most changes, but it does get the job done. I'd just like to run it once, when a new version is pushed to production that requires DB changes, instead of every time an instance is spun up.

jamesatheyDDS commented 6 years ago

@jgarber623-gov Since @aileendds pointed out that staging and production should be as close as possible, I am now just removing the rails.config file altogether, since all I think it does is run db:setup. Now, all deployed environments, whether they be staging, production, or per-branch environments (if anyone ever creates another one like I did for the first draft of the PPM estimator) need someone to SSH in and make database changes manually. Thoughts?

jgarber623-gov commented 6 years ago

@jamesatheyDDS If/when you proceed with this change, you'll want to update the developer documentation on the wiki, too. I believe it references the current seeding/deployment process. You'll also want to alert @rich-allen-gov and team so they're aware of this change.

rich-allen-gov commented 6 years ago

I'll have our team follow the issue and try to give them a heads up.

jamesatheyDDS commented 6 years ago

@jgarber623-gov @aileendds @rich-allen-gov I am prepared to alter https://github.com/deptofdefense/move.mil/wiki/Working-with-Seed-Data once this change lands. Is everyone comfortable with proceeding?

jamesatheyDDS commented 6 years ago

Also, looking at the Events log on the AWS console, this change will only resolve maybe 1 out of 3 or 1 out of 4 of the "X% of requests are failing with HTTP 5xx" events. Without looking more closely at the logs on the instances, I don't know where the rest of the 5xx errors are coming from.