forever list empty after reboot but service is running

[mashhoodr]

Hi!

I configured forever-service on my Debian machine, it starts the service on reboot, but the I cannot access the service via forever list nor sudo service sName status. If I check using ps -aux I can see the node service running.

When I run sudo service sName status it says sName is not running

Any suggestions to what I could have configured incorrectly?

Linux development 3.2.0-4-amd64 #1 SMP Debian 3.2.65-1+deb7u2 x86_64 Forever version: v0.10.9 Forever service version: forever-service version 0.4.5 Node: v0.10.21 NPM: 1.3.15

[arvind-agarwal]

Can you post your /etc/init.d/sName file

[mashhoodr]

File contents for that file:

#!/bin/bash

### BEGIN INIT INFO
# Provides: staging
# Required-Start:    $network $remote_fs $local_fs
# Required-Stop:     $network $remote_fs $local_fs
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: forever-service startup script for staging
# Description: forever-service startup script for node script based service staging, uses forever to start the service
### END INIT INFO

#   CLI node /usr/bin/forever-service install staging --script staging/app.js -e NODE_ENV=staging USER=something PASSWORD=something DATABASE=something HOST=something PORT=1234 HTTP=1234 HTTPS=1234 --start
#   Working Directory /var/www

#Setup Environment variables (if any)
export NODE_ENV=staging

export USER=something

export PASSWORD=something

export DATABASE=something

export HOST= something

export PORT=1234

export HTTP=1234

export HTTPS=1234

# Check if any of $pid (could be plural) are running
PIDFILE="/var/run/staging.pid"
LOGFILE="/var/log/staging.log"

LOCKFILE="/var/lock/staging"

# introduce some gaps between restarts and throttle continous restarts
MIN_UPTIME="5000"
SPIN_SLEEP_TIME="2000"

# kill signal: Since default needs to be SIGTERM, it is important that services gracefully shutdown,
# specially if they are doing transactions or other work which should not be interuppted in between
# for exceptional situation where you dont care about abrupt shutdown, SIGKILL should be used
KILL_SIGNAL="SIGTERM"

# Wait time afer with SIGKILL will be sent to the process, in case SIGTERM is not fully finished
# This is required since when we use SIGTERM, some times if there is problem in code, it might take lot of time for process to exit
# or process may never exit, in such siutation we have to forcebly kill it so that shutdown or service restart can be done appropriately
# this wait time is in millisecond
KILLWAITTIME=5000

killtree() {
    local _pid=$1
    local _sig=${2:--TERM}
    kill -stop ${_pid} # needed to stop quickly forking parent from producing children between child killing and parent killing
    for _child in $(ps -o pid --no-headers --ppid ${_pid}); do
        killtree ${_child} ${_sig}
    done
    kill -${_sig} ${_pid}
}

checkpidexists() {
    [ -d "/proc/$1" ] && return 0
    return 1
}

start() {
    #this is to ensure forever is able to find out the correct root every time
    export FOREVER_ROOT=/root/.forever

    STATUS=$(forever --plain list | sed 's/data:\(\s*\[[0-9]*\]\s*\(staging\)\s.*\)/\2-status:\1/;tx;d;:x')
    if ! [ -z "$STATUS" ]; then
        echo "Service staging already running"
        return 0
    fi

    echo  "Starting staging"

    # move to the directory from where the inital forever script was launched so that even if it is relative it works as expected
    cd /var/www

    forever \
    --pidFile $PIDFILE \
    -a \
    -l $LOGFILE \
    --minUptime $MIN_UPTIME \
    --spinSleepTime $SPIN_SLEEP_TIME \
    --killSignal $KILL_SIGNAL \
     \
    --uid staging \
    start staging/app.js  2>&1 >/dev/null
    RETVAL=$?

    [ $RETVAL = 0 ] && touch $LOCKFILE
    return $RETVAL
}

stop() {
    #this is to ensure forever is able to find out the correct root every time
    export FOREVER_ROOT=/root/.forever

    echo -n "Shutting down staging: "

    STATUS=$(forever --plain list | sed 's/data:\(\s*\[[0-9]*\]\s*\(staging\)\s.*\)/\2-status:\1/;tx;d;:x')
    if [ -z "$STATUS" ]; then
        echo "Not running"
        return 0
    fi

    # PID=$(<$PIDFILE) - Changed to detection based on actual PID from forever, sicne due to watchDirectory pid could dynamically change
    PID=$(forever --plain list | sed -n -e '/data:\s*\[[0-9]*\]\s\(staging\)\s/p' | awk '{print $7}')

    if [ -z "$PID" ]; then
        echo "Could not get pid"
        return 0
    fi

    #run in background, since recent changes in forever, now blocks stop call with SIGTERM is finished
    #but we want to wait till some time and forcibly kill after elapsed time
    #without background script, we could be waiting forever
    forever stop staging 2>&1 >/dev/null &

    CURRENTWAITTIME=$KILLWAITTIME
    # wait for some time before forcefully killing the process
    while [ $CURRENTWAITTIME -gt 0 ]; do
        #check if the process is still running
        checkpidexists $PID
        if [ $? -ne 0 ]; then
            # if not running we can break, since no more wait is needed, service is stopped
            echo "Successful"
            break
        fi

        sleep 1
        CURRENTWAITTIME=$(( $CURRENTWAITTIME - 1000))

    done
    checkpidexists $PID
    if [  $? -eq 0  ]; then
        killtree $PID 9
        echo 'Forced shutdown'
    fi

    rm -f $PIDFILE 2>&1 >/dev/null
    rm -f $LOCKFILE 2>&1 >/dev/null
    return 0

}

status() {
    #this is to ensure forever is able to find out the correct root every time
    export FOREVER_ROOT=/root/.forever

    STATUS=$(forever --plain list | sed 's/data:\(\s*\[[0-9]*\]\s*\(staging\)\s.*\)/\2-status:\1/;tx;d;:x')
    if [ -z "$STATUS" ]; then
        echo "staging is not running"
        RETVAL=3
    else
        echo $STATUS
        RETVAL=0
    fi
    return $RETVAL
}

case "$1" in
    start)
    start
    ;;
    stop)
    stop
    ;;
    status)
    status
    ;;
    restart)
        stop
    start
    ;;
    *)
    echo "Usage: <servicename> {start|stop|status|restart}"
    exit 1
    ;;
esac
exit $?

[arvind-agarwal]

Can you try following as root.

Setup env variable FOREVER_ROOT=/root/.forever and then run forever list

and check if it gives you the required output

[mashhoodr]

Sorry, no luck.

Im logged in as root

set the ENV VAR: FOREVER_ROOT=/root/.forever

check:

root@development:/var/www# echo $FOREVER_ROOT
/root/.forever
root@development:/var/www# forever list
info:    No forever processes running
root@development:/var/www# ps -aux | grep forever
root      2668  0.0  4.7 605536 23944 ?        Ssl  01:24   0:00 /usr/bin/nodejs /usr/lib/node_modules/forever/bin/monitor staging/app.js
root      4748  0.0  0.1   7836   880 pts/0    S+   01:57   0:00 grep forever

Forever list was working fine before the reboot.

[arvind-agarwal]

Any specific reason you are not using latest forever? I would recommend moving to latest package and try. Since looks like forever is not able to get the list here from its local store in forever root.

[mashhoodr]

I have updated to the latest version (v0.14.1)

Still not showing. Do I need to reboot?

[arvind-agarwal]

Yup please reboot and try

[mashhoodr]

Its fixed after reboot! Thanks so much.