Restarting the backend server after a crash

[anoopsarkar]

Is it worth adding a server auto-restart on crash?

[KonceptGeek]

As discussed yesterday, we could add a quick hack so that the backend keeps updating a file every minute and a shell script could keep checking if the file is being updated, otherwise it would kill the backend and restart again.

[avacariu]

Can't we get the init system to do it? CentOS uses Upstart, doesn't it?

[theq629]

Note that the backend should already allow crashes inside the query handling code without crashing the whole server. So the main issue is probably timing out queries rather than crashes. (Although external support would be a good idea as well.)

[anoopsarkar]

I'm looking for:

1. a small footprint (not much change to the existing code)
2. not much maintenance (no need to check often if the restart procedure itself is working)
3. easy to deploy on standard platforms

init.d or upstart or systemd might be suitable. This is a common enough problem that there should be a recipe somewhere, right?

[avacariu]

You'd only need something simple like this for Upstart:

start on runlevel 2
exec $BACKEND -c full.conf
respawn

You can get rid of the start on line if you want to start it up manually.

For systemd it's going to be similarly short.

[anoopsarkar]

Andrei can you check if natlang-web has upstart? I suspect it is still init.d

If it isnt already installed and if it can be installed via yum, then I can do it. Let me know. On Jan 14, 2015 6:07 PM, "Andrei Vacariu" notifications@github.com wrote:

You'd only need something simple like this for Upstart:

start on runlevel 2 exec $BACKEND -c full.conf respawn

— Reply to this email directly or view it on GitHub https://github.com/sfu-natlang/lensingwikipedia/issues/139#issuecomment-70029131 .

[KonceptGeek]

Isn't Upstart for starting application when the system is started? I thought we wanted a solution when the application crashed due to large queries.

[avacariu]

Upstart can handle starting services whenever you want, including manually. The respawn statement tells it to restart the service when it crashes.

[avacariu]

@anoopsarkar according to yum, the upstart package is installed.

[anoopsarkar]

I tried various ways to get upstart to launch and respawn the backend and failed.

Perhaps I can sit with @vlad003 and do this with him.

[anoopsarkar]

@vlad003 could you get upstart working on natlang-web.cs.sfu.ca? If you did, send me a shell file to execute that will get the backend running using upstart.

[avacariu]

Could I get sudo access on that server? I need it to write and run the .conf files in /etc/init.

[anoopsarkar]

Since the dockerize PR #192 was merged with master we no longer use upstart.