SolrCloud warm standby - Githubissues

User Story

In order to be able to recover from a failed SOLR cloud, data.gov admins want a warm SOLR cloud standby that we could switch to in order to keep the site up.

Acceptance Criteria

[ACs should be clearly demoable/verifiable whenever possible. Try specifying them using BDD.]

[ ] GIVEN a SOLR cloud environment standby is in place \ WHEN the in use SOLR cloud is not available \ THEN a data.gov admin can swap to using the warm standby (to keep the site working) \ AND a task can be used to seed any missing data in the SOLR instance

Background

The SOLR sync process on a full production instance takes 4 days. We're not sure how long a backup takes to restore, but probably some percentage of that (even if it takes 10% of time, we're talking 10 hours to recover). Since an outage of that time isn't acceptable, we want a SOLR Cloud instance ready to take over if SOLR crashes. Ideally we would use the native backup process from live SOLR, and restore it into the backup regularly (utilizes s3, see documentation here).

We are also considering a different approach, where we have a separate SOLR cluster spun up and synced every 1-4 days; ready to take over if necessary.

Security Considerations (required)

None, data does not leave secure environment

Sketch

[ ] Research ways to backup and restore SOLR data
[ ] Plan out necessary resources
[ ] Implement process in dev
[ ] Test backup occurs
[ ] Test SOLR swap
[ ] Document recovery process

Alternate Sketch

[ ] Setup another ckan-solr-standby-sync cloud.gov app, with no running jobs
[ ] Connect this to another (new) solr cluster, but the same DB and redis as production catalog
[ ] Setup a task that runs the only missing solr sync m-thur
[ ] Setup a task that runs the full solr index on fridays (should be done by monday?)
[ ] Document process for swapping them in real time in case of solr production outage

GSA / data.gov