Technical > Move OpenSearch to ec2 instance

[chrisaballard]

This should save 40-50% on costs

• [ ] ~Backup current opensearch index to accessible location~
• [x] Set up equivalent spec ec2 instance with shareable EBS (if we want to use the same storage with a different EC2 machine). Volume is mounted at /data and set to remount on reboot, but I haven't tested this yet
• [x] install opensearch
• [x] reload data
• [x] Need a fixed domain name for opensearch and web instances (otherwise will change each time!!)
• [ ] change password from default
• [x] set prototype to run on machine start
• [x] set prototype to use ec2 opensearch instance
• [x] tweak parameter in API which is sensitive to number of shards (we're now using 1, not 5). *Changed from default of 120 to 1205=600**
• [x] Set up a schedule to bring both the web and opensearch instance down overnight and weekends
• [ ] ~Enable SSL?~
• [x] Auto start the docker containers when brought back up
• [x] resize opensearch service instance to free tier
• [x] only enable opensearch access through prototype. Changed security rules so only the prototype EC2 instance can access port 9200

[kdutia]

Should we use this AWS CloudFormation template for scheduling the EC2 instance? https://aws.amazon.com/solutions/implementations/instance-scheduler/

Alternatively we could manually set up a schedule using AWS Lambda functions https://aws.amazon.com/premiumsupport/knowledge-center/start-stop-lambda-cloudwatch/

[kdutia]

Tried using elasticdump to export the index to my local machine so I could re-index it in an EC2 opensearch instance, but it was very slow. I'm going to set up the new opensearch instance and re-run the load instead.

[kdutia]

The new instance has been set up with 1 shard rather than 5 shards which exist on the managed Opensearch instance. From Elasticsearch guidance this seems like the correct thing to do, and may help us reduce our resource usage (each shard has separate overhead).

[kdutia]

I set up a cron job to auto-start opensearch and opensearch-dashboards on reboot. Tested that it works both when restarting the ec2 machine, and stopping it then starting it again.

[kdutia]

Schedule set up on EventBridge for online time 6am-10pm weekdays