In production mode it should be impossible to crash server

[ahungry]

I'm still struggling with occasional errors that bubble up with backtraces based on bad user input (for instance, someone sent a PUT request with JSON and clack attempted to eval it, and crashed due to 'incomplete JSON' in a clack woo handler).

I think there needs to be a config setting (productionp based maybe) that prevents the environment from being able to crash/stop serving requests in all but the most unpredicted circumstances.

Providing a web server means that the input cannot and will not always be reliable (the user's remote requests) due to maliciousness or errors on the user's part.

Is there a way to cause the system to either drop threads/requests that error out (leaving the system to serve the other requests coming in and recreating the thread), force a server restart on error, or have the system drop out of the lisp instance (SBCL etc.) entirely? I've tried sbcl --no-debug and --script options to attempt to drop out entirely (and run the platform in a shell "while" loop) but when clack hits one of these errors, it seems to lock up on it (even with disabled debugger).

[jackcarrozzo]

I'm interested to hear the clean way to handle this as well!

[fukamachi]

Are you starting an application by clack:clackup with :debug nil?

On Mon, Jun 22, 2015 at 1:59 AM, Jack Carrozzo notifications@github.com wrote:

I'm interested to hear the clean way to handle this as well!

— Reply to this email directly or view it on GitHub https://github.com/fukamachi/caveman/issues/61#issuecomment-113927992.

[ahungry]

I set APP_ENV to 'production' in my shell rc file, so #'productionp returns true in config.lisp.

Then I start with:

 (com.ahungry:start :server :woo)

In the various defconfigs for caveman, I have:

(defconfig |production|
    (list :databases *env-based-database-info*
          :debug nil))

Where env-based-database-info is the list containing the database connection info.

For now I have 99.9% uptime without user intervention by running the following to launch the application (apache proxypasses to it):

while :; do sbcl --eval '(ql:quickload com.ahungry)' --eval '(com.ahungry:start :server :woo)'; done

And then another script that runs on a 5 minute CRON which makes a curl request to localhost:5000 (default caveman port) and if it doesn't see a reply (which is what happens when a fatal error bubbles up) it issues a kill -9 to the sbcl process.

[ahungry]

The script issuing a kill is required for the reasons mentioned in initial post (a bubbled up error just hangs indefinitely instead of allowing sbcl to drop out to the shell as should be expected).

[hijarian]

@ahungry You essentially ask for the functionality of forever. I guess it's out of scope for Clack. It should be separate project.

[hijarian]

@ahungry Although I support @fukamachi and suggest that you always run your server with :debug nil, as if you don't have Swank tunnel to your server, it's meaningless to make Lisp drop to debugger on unhandled error, it must die immediately.

[ahungry]

Ah, I thought that's what the debug nil in config.lisp environment setting was for. Maybe a mistake on my part.