Deploying on ES 5

[divergentdave]

This is a follow-up from #177, with nice checkboxes!

• [x] Create second AWS Elasticsearch domain, running version 5.1.1
• [x] Tweak the per-node storage setting in the domain, as we did with the old one previously
  • For the record, our configuration currently looks like "Instance count: 1, Instance type: t2.small.elasticsearch, Storage type: EBS, EBS volume type: General Purpose (SSD), EBS volume size: 20GB, Automated snapshot start hour 00:00 UTC (default)"
  • The access policy attached to the domain allows "...:root" to perform "es:*" actions on "...:domain/oversight/*". (There's also an old clause with IP-based access from testing, but we don't need to carry that forward)
• [x] Update user-data files with new DNS name for Elasticsearch domain
• [x] Merge the PR to master (easier than juggling branches when deploying)
• [x] Terminate the old scraper instance, start a new one, and SSH in
• [x] Run the elasticsearch:init, elasticsearch:alias_write, elasticsearch:alias_read, and elasticsearch:init_dashboard rake tasks to prepare mappings
• [x] Populate ES with tasks/inspectors.js --since=1
• [x] Start a new web instance
• [x] Kick the tires on staging.oversight.garden for a while
• [x] Promote from staging to production
• [x] Terminate the old web instance
• [x] Delete the old AWS Elasticsearch domain

[divergentdave]

When the new ES domain came up, it was stuck in "yellow" status. I fired up irb from the working directory on the server and did the following.

> require './config/environment'
> puts Environment.client.cat.shards
.kibana            0 p STARTED    1 3.1kb x.x.x.x 79k86C7
.kibana            0 r UNASSIGNED                       
dashboard          1 p STARTED    0  130b x.x.x.x 79k86C7
dashboard          2 p STARTED    0  130b x.x.x.x 79k86C7
dashboard          4 p STARTED    0  130b x.x.x.x 79k86C7
dashboard          3 p STARTED    0  130b x.x.x.x 79k86C7
dashboard          0 p STARTED    0  130b x.x.x.x 79k86C7
oversight-20170325 1 p STARTED    0  130b x.x.x.x 79k86C7
oversight-20170325 2 p STARTED    0  130b x.x.x.x 79k86C7
oversight-20170325 4 p STARTED    0  130b x.x.x.x 79k86C7
oversight-20170325 3 p STARTED    0  130b x.x.x.x 79k86C7
oversight-20170325 0 p STARTED    0  130b x.x.x.x 79k86C7
 => nil

I tried deleting the .kibana index, since I've never been a big fan of Kibana, but it came back on it's own. I think I'll have to set number_of_replicas to 0 on the .kibana index.

[divergentdave]

Environment.client.indices.put_settings index: '.kibana', body: { index: { number_of_replicas: 0 } } solved it.

> puts Environment.client.cat.health
1490497605 20:06:45 786276019377:oversight-es5 green 1 1 11 11 0 0 0 0 - 100.0%

[divergentdave]

It's all migrated! I'm going to wait a bit before deleting the version 1.5 stuff, but it's all looking good.

[konklone]

Hot damn, good job!

[divergentdave]