Redis child process doesn't terminate

[chrismcg]

Using foreman to start up a project specific redis instance.

redis: redis-server ./config/redis.development.conf

It starts fine but doesn't get killed when I Ctrl-C foreman. I have daemonize no in the redis conf I'm giving it so it's not that. Looks like it might be related to issue #10

[chrismcg]

This is due to redis ignoring SIGHUP. When I compile a version of redis that doesn't ignore it it works as expected. Not sure what the best way around the issue is :/

[chrismcg]

I'm guessing this is because the SIGINT gets send to the sh -c that PTY.spawn creates, not to the actual redis instance itself. So bash turns that into SIGHUP and redis ignores that.

[chrismcg]

Worked round it by creating script/run_redis that is:

#!/usr/bin/env ruby
trap("SIGHUP") { Process.kill(:TERM, $pid) }

$pid = fork

if $pid == nil
  exec "redis-server ./config/redis.development.conf"
end

Process.wait

And having foreman run that

[ddollar]

This should be fixed in the latest foreman. We always send SIGTERM to terminate now.

Cheers, David

[deepthawtz]

Just tested this out on 0.17.0 Using redis 1.2.6

My Procfile

web: bundle exec ./script/server thin
redis: redis-server config/redis.development.conf
worker: bundle exec rake resque:work QUEUE=*

Results

➜  foreman start
10:54:25 web.1     | started with pid 20154
10:54:25 redis.1   | started with pid 20155
10:54:25 worker.1  | started with pid 20157
10:54:29 web.1     | => Booting Thin
10:54:29 web.1     | => Rails 2.3.10 application starting on http://0.0.0.0:3000
^C
SIGINT received
10:54:30 system    | terminating
10:54:30 system    | killing worker in pid 20157
10:54:30 system    | killing web in pid 20154
10:54:30 system    | killing redis in pid 20155
10:54:30 worker.1  | process terminated
10:54:30 system    | terminating
10:54:30 system    | killing web in pid 20154
10:54:30 system    | killing redis in pid 20155
➜ ps aux | grep redis-server
deepthawtz 20178   0.0  0.0  2425704    268 s005  R+   10:54AM   0:00.00 grep redis-server
deepthawtz 20160   0.0  0.0  2435788    964   ??  S    10:54AM   0:00.00 redis-server config/redis.development.conf

[ddollar]

Could you paste the contents of your redis.development.conf

Thanks, David

[deepthawtz]

# Redis configuration file example

# By default Redis does not run as a daemon. Use 'yes' if you need it.
# Note that Redis will write a pid file in /Users/deepthawtz/.zendesk/var/run/redis.pid when daemonized.
daemonize no

# When run as a daemon, Redis write a pid file in /Users/deepthawtz/.zendesk/var/run/redis.pid by default.
# You can specify a custom pid file location here.
pidfile /Users/deepthawtz/.zendesk/var/run/redis.pid

# Accept connections on the specified port, default is 6379
port 6379

# If you want you can bind a single interface, if the bind option is not
# specified all the interfaces will listen for connections.
#
# bind 127.0.0.1

# Close the connection after a client is idle for N seconds (0 to disable)
timeout 300

# Set server verbosity to 'debug'
# it can be one of:
# debug (a lot of information, useful for development/testing)
# notice (moderately verbose, what you want in production probably)
# warning (only very important / critical messages are logged)
loglevel debug

# Specify the log file name. Also 'stdout' can be used to force
# the demon to log on the standard output. Note that if you use standard
# output for logging but daemonize, logs will be sent to /dev/null
logfile /dev/null

# Set the number of databases. The default database is DB 0, you can select
# a different one on a per-connection basis using SELECT <dbid> where
# dbid is a number between 0 and 'databases'-1
databases 16

################################ SNAPSHOTTING  #################################
#
# Save the DB on disk:
#
#   save <seconds> <changes>
#
#   Will save the DB if both the given number of seconds and the given
#   number of write operations against the DB occurred.
#
#   In the example below the behaviour will be to save:
#   after 900 sec (15 min) if at least 1 key changed
#   after 300 sec (5 min) if at least 10 keys changed
#   after 60 sec if at least 10000 keys changed
save 900 1
save 300 10
save 60 10000

# Compress string objects using LZF when dump .rdb databases?
# For default that's set to 'yes' as it's almost always a win.
# If you want to save some CPU in the saving child set it to 'no' but
# the dataset will likely be bigger if you have compressible values or keys.
rdbcompression yes

# The filename where to dump the DB
dbfilename dump.rdb

# For default save/load DB in/from the working directory
# Note that you must specify a directory not a file name.
dir /Users/deepthawtz/.zendesk/var/db/redis/

################################# REPLICATION #################################

# Master-Slave replication. Use slaveof to make a Redis instance a copy of
# another Redis server. Note that the configuration is local to the slave
# so for example it is possible to configure the slave to save the DB with a
# different interval, or to listen to another port, and so on.
#
# slaveof <masterip> <masterport>

# If the master is password protected (using the "requirepass" configuration
# directive below) it is possible to tell the slave to authenticate before
# starting the replication synchronization process, otherwise the master will
# refuse the slave request.
#
# masterauth <master-password>

################################## SECURITY ###################################

# Require clients to issue AUTH <PASSWORD> before processing any other
# commands.  This might be useful in environments in which you do not trust
# others with access to the host running redis-server.
#
# This should stay commented out for backward compatibility and because most
# people do not need auth (e.g. they run their own servers).
#
# requirepass foobared

################################### LIMITS ####################################

# Set the max number of connected clients at the same time. By default there
# is no limit, and it's up to the number of file descriptors the Redis process
# is able to open. The special value '0' means no limts.
# Once the limit is reached Redis will close all the new connections sending
# an error 'max number of clients reached'.
#
# maxclients 128

# Don't use more memory than the specified amount of bytes.
# When the memory limit is reached Redis will try to remove keys with an
# EXPIRE set. It will try to start freeing keys that are going to expire
# in little time and preserve keys with a longer time to live.
# Redis will also try to remove objects from free lists if possible.
#
# If all this fails, Redis will start to reply with errors to commands
# that will use more memory, like SET, LPUSH, and so on, and will continue
# to reply to most read-only commands like GET.
#
# WARNING: maxmemory can be a good idea mainly if you want to use Redis as a
# 'state' server or cache, not as a real DB. When Redis is used as a real
# database the memory usage will grow over the weeks, it will be obvious if
# it is going to use too much memory in the long run, and you'll have the time
# to upgrade. With maxmemory after the limit is reached you'll start to get
# errors for write operations, and this may even lead to DB inconsistency.
#
# maxmemory <bytes>

############################## APPEND ONLY MODE ###############################

# By default Redis asynchronously dumps the dataset on disk. If you can live
# with the idea that the latest records will be lost if something like a crash
# happens this is the preferred way to run Redis. If instead you care a lot
# about your data and don't want to that a single record can get lost you should
# enable the append only mode: when this mode is enabled Redis will append
# every write operation received in the file appendonly.log. This file will
# be read on startup in order to rebuild the full dataset in memory.
#
# Note that you can have both the async dumps and the append only file if you
# like (you have to comment the "save" statements above to disable the dumps).
# Still if append only mode is enabled Redis will load the data from the
# log file at startup ignoring the dump.rdb file.
#
# The name of the append only file is "appendonly.log"
#
# IMPORTANT: Check the BGREWRITEAOF to check how to rewrite the append
# log file in background when it gets too big.

appendonly no

# The fsync() call tells the Operating System to actually write data on disk
# instead to wait for more data in the output buffer. Some OS will really flush
# data on disk, some other OS will just try to do it ASAP.
#
# Redis supports three different modes:
#
# no: don't fsync, just let the OS flush the data when it wants. Faster.
# always: fsync after every write to the append only log . Slow, Safest.
# everysec: fsync only if one second passed since the last fsync. Compromise.
#
# The default is "always" that's the safer of the options. It's up to you to
# understand if you can relax this to "everysec" that will fsync every second
# or to "no" that will let the operating system flush the output buffer when
# it want, for better performances (but if you can live with the idea of
# some data loss consider the default persistence mode that's snapshotting).

appendfsync always
# appendfsync everysec
# appendfsync no

############################### ADVANCED CONFIG ###############################

# Glue small output buffers together in order to send small replies in a
# single TCP packet. Uses a bit more CPU but most of the times it is a win
# in terms of number of queries per second. Use 'yes' if unsure.
glueoutputbuf yes

# Use object sharing. Can save a lot of memory if you have many common
# string in your dataset, but performs lookups against the shared objects
# pool so it uses more CPU and can be a bit slower. Usually it's a good
# idea.
#
# When object sharing is enabled (shareobjects yes) you can use
# shareobjectspoolsize to control the size of the pool used in order to try
# object sharing. A bigger pool size will lead to better sharing capabilities.
# In general you want this value to be at least the double of the number of
# very common strings you have in your dataset.
#
# WARNING: object sharing is experimental, don't enable this feature
# in production before of Redis 1.0-stable. Still please try this feature in
# your development environment so that we can test it better.
shareobjects no
shareobjectspoolsize 1024

[deepthawtz]

I notice that foreman logs "SIGINT received", I'm using ^C to stop foreman, is that the correct way? The README and man page don't really mention how to stop foreman properly.

[deepthawtz]

This might be related. Does foreman maintain its own pid mapping? I've just noticed that the pid foreman says it is starting is different from the actual pid the OS gives.

[ddollar]

No it doesn't, I wonder if redis has some parent process that starts, forks a child and then goes away. ^C is the normal way to stop foreman.

[deepthawtz]

Actually, none of the other pids seem to match the ones the OS knows about. I'm not sure how this is possible.

➜ foreman start
12:08:37 web.1     | started with pid 21573
12:08:37 redis.1   | started with pid 21574
12:08:37 worker.1  | started with pid 21576
^Z
[2]  + 21572 suspended  foreman start
➜ ps aux | grep thin
deepthawtz 21579 100.0  1.0  2483144  43472 s000  R+   12:08PM   0:05.08 ruby ./script/server thin
deepthawtz 21575   0.0  0.0  2435544    804 s000  Ss+  12:08PM   0:00.00 sh -c bundle exec ./script/server thin 2>&1
➜ ps aux | grep resque
deepthawtz 21615   0.8  0.0  2435120    536 s005  R+   12:10PM   0:00.00 grep resque
deepthawtz 21581   0.0  1.1  2483072  44096 s004  S+   12:08PM   0:05.76 resque-1.16.1: Waiting for *                                                  
deepthawtz 21578   0.0  0.0  2435544    816 s004  Ss+  12:08PM   0:00.00 sh -c bundle exec rake resque:work QUEUE=* 2>&1

[deepthawtz]

I see why now pstree reveals the processes are actually children of a foreman process, makes sense now.

[ddollar]

This should be fixed for you now. Foreman will now do the following when shutting down processes: SIGTERM, 3 second pause, SIGKILL (if still running)

Cheers, David

[deepthawtz]

That works David, thanks.