Q4 Disaster Recovery Exercise

kiwix / operations

Kiwix Kubernetes Cluster

http://charts.k8s.kiwix.org/

5 stars 0 forks source link

Q4 Disaster Recovery Exercise #126

Closed kiwixbot closed 8 months ago

kiwixbot commented 9 months ago

In order to strengthen our infrastructure, we want to:

Document our Disaster Recovery Plan
Simulate a Disaster and recover the service to ensure documentation exists and is up to date

This ticket requests the assignees to:

[x] Pick a Service from Kiwix/openZIM/Offspot
[x] Redeploy it completly using its backup, using the documentation or writing it.

See Zimfarm documentation.

Note: this is an automatic reminder intended for the assignee(s).

rgaudin commented 9 months ago

@kelson42 @benoit74 which service should we do?

kelson42 commented 9 months ago

@rgaudin The website? The wiki? The object-cache service?

rgaudin commented 9 months ago

Might be a good opportunity to document the website setup since it's not on our infra and AFAIK we don't backup it (maybe there's a backup feature in cyon)

In the other hand, it's entirely different than the rest of our service and maintained by martin. In a crash scenario he'd probably be the one restoring it

rgaudin commented 9 months ago

Can we take 15mn early this week to discuss it all together ?

S3 is not backed up (the policies and all) and that's somewhat important we do

kelson42 commented 9 months ago

@rgaudin Agree. Usually such outsourced services are the less resilient exactly because its outsourced and we "trust" the provider. Here is the danger.

rgaudin commented 9 months ago

Opened #128 and #129. We'll do the Kiwix Wiki this week.

benoit74 commented 8 months ago

Done. Procedure is now documented (https://github.com/kiwix/k8s/wiki/Restore-Wiki-from-a-backup) and tested on both wikis.