search

Charcoal-SE / PulseMonitor

PulseMonitor is the chat interface to Halflife.
MIT License
4 stars 3 forks source link

Safe storage beyond Redunda #25

Open tripleee opened 4 years ago

tripleee commented 4 years ago

The items we had in Redunda were lost. The instance is now back up but with a clean slate. Ideally, there should be a way to store the current state in multiple places (back up your backups!)

In particular, it would be nice to have a checkpoint feature to pull the current data out of a running client prior to rebooting its host and/or container before there is a crisis involving one or more of metasmoke, Redunda, and AWS.

tripleee commented 4 years ago

There is now a simple nightly cron job pulsebackup on the EC2 instance where Halflife runs which copies the dot directory from the running instance to a local Git repository in pulsedata. I'll see about mirroring this repo to a proper place.

tripleee commented 4 years ago

Just restored (most) notifications from yesterday from backups. The prestart script should be refactored to take a backup, stop the old instance, start a new instance in offline mode, copy in the backup, and only then restart the bot.

For what it's worth, I now have an offsite backup of the pulsedata Git repo, though the automation is probably still somewhat brittle.

tripleee commented 4 years ago

(Or , of course, I should figure out how to get Redunda to do exactly this!)

tripleee commented 4 years ago

Tangentially, reminder to self, how to get useful JSON diffs out of Git:

echo '*.json diff=json' >.gitattributes
git config diff.json.textconv 'jq .'

Adapted from https://t-a-w.blogspot.com/2016/05/sensible-git-diff-for-json-files.html

tripleee commented 4 years ago

I have refactored the restart script to copy in a fresh version from backup but it still looks like Redunda is rolling back the fresh data to an older version. Not really in a place where I can experiment, and Redunda seems to be down anyway ... /-: