Everything is broken! "Database connection error" when loading Feedback Friends

[ttencate]

I'm looking into it and will post my findings here.

[ttencate]

Looks like MySQL crashed because it ran out of memory:

$ tail error.log
160831 17:23:40 InnoDB: Completed initialization of buffer pool
160831 17:23:40 InnoDB: Fatal error: cannot allocate memory for the buffer pool
160831 17:23:40 [ERROR] Plugin 'InnoDB' init function returned error.
160831 17:23:40 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
160831 17:23:40 [ERROR] Unknown/unsupported storage engine: InnoDB
160831 17:23:40 [ERROR] Aborting

160831 17:23:41 [Note] /usr/sbin/mysqld: Shutdown complete

160831 17:23:42 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended

Why did it run out of memory? Might be related to this:

$ ps aux|grep php|wc -l
475

There are a LOT of scrapers running in parallel. Maybe it got into a bad state because something was briefly slow, and new ones kept getting started and making it even slower.

I'm killing all the scrapers like this:

$ ps aux|grep scraping.php|awk '{print $2}'|xargs kill

Then restarting mysql.

[ttencate]

I don't understand how this is wired up. /etc/init.d/mysql seems to defer to systemctl. Indeed systemctl status mysql shows "active (exited)". But neither systemctl start mysql nor /etc/init.d/mysql start seems to do anything.

Right now I'm running it manually (connected to my terminal) using sudo /usr/bin/mysqld_safe, so the site is up again while I figure out how I should properly restart it...

[ttencate]

OK, just restart instead of start. That took me way too long.

Going to add a timeout to the scrapers so they don't run over their allotted time.

[ttencate]

There is already a timeout, set to 40 seconds and passed to set_time_limit with an additional 30 seconds. Methinks it's not quite working. Probably PHP only checks it when it's not fetching files, and file_get_contents has a default timeout of 60 seconds but only applied (when no bytes are being received](http://stackoverflow.com/questions/10236166/does-file-get-contents-have-a-timeout-setting) (I think).

curl might be a better way to fetch files, but it probably requires modules, and I vaguely recall it being a hassle in the past. Not going to risk that.

No matter: we'll just add a sleep and kill to the cronjob command line.

[ttencate]

Prefixing timeout 60s to the command in the crontab seems to work great. Some concerns exist about data integrity, but the next scrape should fix that.