tiddlyspace.com emergency migration post checklist

[cdent]

Here as it seems like a nice way to keep track and keep it safe and because separate tickets need more information than able to gather at this point.

• [x] listener.py (used with websocket stuff) hang as it puts a tiddler into beanstalkd, unclear if a listener or beanstalk problem. upshot is that websocket feeds (frontpage, tapas, etc) don't update
• [x] whoosh index needs to be rebuilt from scratch (in progress)
• [x] twp.tiddlyspace server3 branch merge to master and release (hand built on new server)
• [x] twp.mysql3 utf8mb4 branch merge to master and release (hand built on new server)
• [x] deploy.sh test and update (the change in tiddlyweb and sudo handling has large impact here)
• [x] document change in use of tiddlyweb account
• [x] confirm correct boot sequence for daemons (adjusted, but still unsure about listener (started by twdispatcher))
• [x] twp.tiddlyspace.profiles warns on a 204 response, bad logic prior to log line
• [x] intermittent bizarre errors from mysql usually associated with a session being used across forks. needs investigation
• [x] twikifier still sometimes timing out
• [x] automysqlbackups needs confirmation of working (potential max_packet problems)
• [x] tscount related stats tools need set up (in place needs confirmation)
• [x] webalizer or other web stats, need set up (in place needs confirmation, will take time (tuesday) because of log format issues)
• [x] collectd or monit need set up
• [x] trim and test the sbp.sh backup script
• [x] adjust nginx and uwsgi config using an awake analytical mind
• [x] twp.prettyerror assumes existince of SCRIPT_NAME in environ at line 181. not present under uwsgi mounted on /
• [x] (OperationalError) (2013, 'Lost connection to MySQL
• [x] Once stable, the amount of logging twikifier does needs to be toned done, the new syslog no likey
• [x] nginx sometimes report: upstream prematurely closed connection while reading response header from upstream
• [ ] mental audit by someone else
• [x] inclue hostname in web logs (if possible)
• [x] locate and record the keys to the kingdom for managing tiddlyspace.com DNS
• [x] adjust backup times to be more aligned with automysql backup times
• [x] twikifier memcache setting errors not proper handled
• [x] document collectd and collection3 setup
• [x] twikifier is leaking quite badly, growing to well over 1GB in the children
• [x] tiddlywiki.org rewrites not present does it matter?
• [ ] add 2nd (and maybe 3rd and 4th) database dump
• [x] take http stuff out of logwatch

[cdent]

On twikifier see this commit. The problem is not fixed, but is managed.

[pads]

Just to let you know, I modified log watch to exclude the httpd service: Service="-httpd" and so the output is much much smaller.

[cdent]

Thanks @pads

[cdent]

• Twikifier timeouts and other errors are happening rarely since the "kill after N requests" stuff was put in place.
• nginx is now 12 workers, uwsgi is 8 workers, 8 threads, single interpreter, kill after 500MB, will push rigorous performance tweaking out into the future as things are currently capable beyond actual use
• http://tsdbup.tiddlyspace.com/Collectd
• http://tsdbup.tiddlyspace.com/TiddlyWebAccountChange

[cdent]

Only a rather small number of 404s in tiddlywiki.org logs, so not gonna bother putting rewrites in. They are a performance killer.

[cdent]

• Colm and I have access to tiddlyspace.com dns
• mysql timeout errors happen during uwsgi restarts and are basically expected (given the current way of doing things)

Still need mental audit by someone else and fixes to deploy.sh

[cdent]

Good enough, closing.