Add crash recovery to server

[Patrick-Bay]

Currently when the server crashes there is no recovery -- it sits dormant until manually restarted. Explore the following to add automated restarts to the server: https://www.digitalocean.com/community/tutorials/how-to-configure-a-linux-service-to-start-automatically-after-a-crash-or-reboot-part-1-practical-examplesAdd a crash count and limit, along with recorded timestamps (per crash) to the server log, or some other recording system, so that we can stop restarts if the limit is reached within a specific time period (e.g. 15 restarts within 30 seconds). Also include a reporting system so that administrators are notified of a server restart and are sent a copy of the crash report / error log(s). Use email notifications to start (text messages would be nice to have).

[RiskingTime]

@Patrick-Bay how long do you estimate it will take to complete this work?

[Patrick-Bay]

I expect crash recovery implementation to take approximately 2 hours which would include both the automation script(s) and the startup email functionality.

[RiskingTime]

ok, start this task next, whenever you are ready

[Patrick-Bay]

I've enabled crash recovery and control on the server and documented how to set up the environment.

https://github.com/CoinRoster/slotmachine/wiki/Enabling-and-Administering-Crash-Recovery-for-the-Game-Server

[RiskingTime]

@Patrick-Bay this might be a silly question, but can you confirm that when the game server crashes and then is restarted automatically, there is a log of all action including some reason why it crashed?

regarding "As a final step, PM2 should generate a startup script for itself and the game server. To do this type: sudo pm2 startup" can you explain this in more detail? is this command what triggers the game server to auto startup?

so basically the game server will run as a process and also PM2 will run as a separate process?

[Patrick-Bay]

I'll answer your questions in order:

• Not in any way a silly question, and yes, we should have a copy of both the regular game log as well as the error log: http://pm2.keymetrics.io/docs/usage/log-management/
• This final step registers PM2 to automatically start when the server is started or restarted. I believe that by default this is off so that PM2 will keep your processes alive but only if the server isn't restarted. There's another command to unregister PM2.
• PM2 is basically a container for Node.js which runs the game server. It automates the process of making Node.js a background process but not in a standard way. If you list running processes ("ps -auroot"), for example, you'll see "PM2" but no "nodejs", so you need to control the game server via PM2. I'm not to familiar with the all of the hows and the whys of this but it seems to work well.

Thanks, Patrick

On Tue, Mar 6, 2018 at 10:20 AM, chavis1082 notifications@github.com wrote:

this might be a silly question, but can you confirm that when the game server crashes and then is restarted automatically, there is a log of all action including some reason why it crashed?

regarding "As a final step, PM2 should generate a startup script for itself and the game server. To do this type: sudo pm2 startup" can you explain this in more detail? is this command what triggers the game server to auto startup?

so basically the game server will run as a process and also PM2 will run as a separate process?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/CoinRoster/slotmachine/issues/23#issuecomment-370815884, or mute the thread https://github.com/notifications/unsubscribe-auth/AHrbz7W2N5_1nIEqgcD-1dYGVuK_DOQ3ks5tbqkzgaJpZM4SUaH5 .

[RiskingTime]

Reopened this issue due to changes made while addressing database connectivity issues

[RiskingTime]

timestamp error log, game server log

[RiskingTime]

error notification e-mails to admins, containing the error msg, time/date, any other details (user account, action, etc)

[Patrick-Bay]

Timestamps have been added to error log as well as in-sequence error logging to the general debug log. In addition, administrator email functionality has been implemented so that any runtime errors, including full details, are sent to administrators (as specified in the game server configuration). All fixes have been pulled to the development server (http://myfruitgame.com) and are available for testing now.

Changes for this issue were pushed with commit: https://github.com/CoinRoster/slotmachine/commit/4cd77c0d8ac4c5d717df520c727eb208437531d6