Job not started after server reboot

[visualight]

Hello,

I created a service for my app like this : $forever-service install ledcom --script ledcom_app.js

All work fine with start and stop but if i reboot the server the service do no start automatically. Can you help me ?

Edit : command (update-rc.d ledcom defaults) give me this output update-rc.d: /etc/init.d/ledcom : file does not exist

Thanks

[arvind-agarwal]

What OS you are using, have you checked logs in /var/log/ledcom.log file and analyzed if your service get executed and then stops for some reason?

[visualight]

My OS is Ubuntu ... I'm so much testing i must reinstall the server. I will communicate my ledcom.log file as soon as the system recovery is complete.

[visualight]

Here is my log

error: Forever detected script exited with code: 143 error: Script restart attempt #1 error: Forever detected script exited with code: null error: Script restart attempt #2 ...

I get this after reboot (fresh install) If i type : status ledcom : ledcom stop/waiting

But if i start manually : start ledcom it work fine

Have you a solution

Thanks

[visualight]

Hello,

I tested to start a node app with and without express framework. Result : i have the same error

error: Forever detected script exited with code: 143 error: Script restart attempt #1 error: Forever detected script exited with code: null

I really need a solution. Thanks

[arvind-agarwal]

Can you check the log file in /var/log/upstart also and send me the logs. Looks like the forever is attempting to start the service but failing for some reason.

Have you checked sudo forever list and see if it is showing your service as running?

[arvind-agarwal]

One more thing to check, is your service dependent upon some environment variables? and may be during system restart it is not able to find those?

[visualight]

Here is my environment variables

root:~# printenv
TERM=xterm-256color
SHELL=/bin/bash
SSH_CLIENT=
SSH_TTY=/dev/pts/0
USER=root
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.axa=00;36:*.oga=00;36:*.spx=00;36:*.xspf=00;36:
MAIL=/var/mail/root
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
PWD=/root
LANG=fr_FR.UTF-8
SHLVL=1
HOME=/root
LOGNAME=root
SSH_CONNECTION=
LESSOPEN=| /usr/bin/lesspipe %s
LESSCLOSE=/usr/bin/lesspipe %s %s
_=/usr/bin/printenv

Here is my log form UPSTART

^[[32minfo^[[39m:    Forever processing file: ^[[90mledcom_app.js^[[39m
ledcom started
^[[32minfo^[[39m:    Forever processing file: ^[[90mledcom_app.js^[[39m
ledcom started
Shutting down ledcom:
^[[32minfo^[[39m:    Forever stopped process:
^[[37m   ^[[39m ^[[37muid^[[39m    ^[[90mcommand^[[39m         ^[[90mscript^[[3$
[0] ledcom ^[[90m/usr/bin/nodejs^[[39m ^[[90mledcom_app.js^[[39m 1054    1058  $
ledcom shutdown
^[[32minfo^[[39m:    Forever processing file: ^[[90mledcom_app.js^[[39m
ledcom started
^[[32minfo^[[39m:    Forever processing file: ^[[90mledcom_app.js^[[39m
ledcom started

If i start manually my app (start ledcom) it work fine

[arvind-agarwal]

Looks like the service start attempt is made by forever, you will have to debug your service to figure out what is going wrong.

What i suspect is some environment mismatch during automated service startup.

Are you using any environment variables in your script or dependent upon HOME env variable?

You can console the process.env.HOME variable from your service and check in logs if that is consistent between automated start as well as manual start. (Do this for any other Env variable you might be using).

Also paste your /etc/init/ledcom.conf file to see if anything problematic is present.

[visualight]

Here is my basic app

var express = require('express');
var app = express();
app.use('/assets', express.static(__dirname + "/assets"));

app.get('/', function(req, res) {

    var data = {
        title : 'Hello World !',
        description : 'home page'
    };

    res.render('index.jade', {data:data});
});

console.log('Ledcom Server Running');
app.listen(3001);

Here is ledcom.conf

#!upstart
#       /etc/init/ledcom.conf
#
#       ledcom - Provisioned using forever-service
#
#       CLI node /usr/local/bin/forever-service install ledcom --script ledcom_$
#       Working Directory /root
#

description "forever-service startup script for node script based service ledco$

env FOREVER_ROOT=/root/.forever

start on (local-filesystems and net-device-up IFACE=eth0)
stop on shutdown

expect fork

# Check if any of $pid (could be plural) are running
env LOGFILE="/var/log/ledcom.log"

# introduce some gaps between restarts and throttle continous restarts
env MIN_UPTIME="5000"
env SPIN_SLEEP_TIME="2000"

# kill signal: Since default needs to be SIGTERM, it is important that services$
# specially if they are doing transactions or other work which should not be in$
# for exceptional situation where you dont care about abrupt shutdown, SIGKILL $
env KILL_SIGNAL="SIGTERM"

# Wait time afer with SIGKILL will be sent to the process, in case SIGTERM is n$
# This is required since when we use SIGTERM, some times if there is problem in$
# or process may never exit, in such siutation we have to forcebly kill it so t$
# this wait time is in millisecond
env KILLWAITTIME=5000

chdir /root

exec  forever -a -l $LOGFILE --minUptime $MIN_UPTIME --spinSleepTime $SPIN_SLEE$

post-start script
        echo "ledcom started"
end script

pre-stop script

 echo "Shutting down ledcom: "

        STATUS=$(forever --plain list | sed 's/data:\(\s*\[[0-9]*\]\s*\(ledcom\$
        if [ -z "$STATUS" ]; then
                echo "Not running"
                return 0
        fi

        # PID=$(<$PIDFILE) - Changed to detection based on actual PID from fore$
        PID=$(forever --plain list | sed -n -e '/data:\s*\[[0-9]*\]\s\(ledcom\)$
        if [ -z "$PID" ]; then
                echo "Could not get pid"
                return 0
        fi

        forever stop ledcom &

        killtree() {
            local _pid=$1
local _sig=${2:--TERM}
            kill -stop ${_pid} # needed to stop quickly forking parent from pro$
            for _child in $(ps -o pid --no-headers --ppid ${_pid}); do
                killtree ${_child} ${_sig}
            done
            kill -${_sig} ${_pid}
        }

        CURRENTWAITTIME=$KILLWAITTIME
        # wait for some time before forcefully killing the process

# wait for some time before forcefully killing the process
        while [ $CURRENTWAITTIME -gt 0 ]; do
                #check if the process is still running
                if [ ! -d "/proc/$PID" ]; then
                        echo "ledcom shutdown"
                        # if not running we can break, since no more wait is ne$
                        break
                fi
                sleep 1
                CURRENTWAITTIME=$(( $CURRENTWAITTIME - 1000))
        done

if [ -d "/proc/$PID" ]; then
                killtree $PID 9
                echo "ledcom Forced shutdown"
        fi

end script

Now if i make my own ledcom.conf in etc/init and in attempt to start manually i get a error : Unable to load "index.jade" in views (or something like this : i not remember exactly the message). But if i start manually with forever command (forever start ledcom_app.js) all work fine.

[arvind-agarwal]

Can you also send me output of sudo forever list after the reboot.

[visualight]

After reboot and sudo forever list i get :

info:    No forever processes running

[arvind-agarwal]

Please send me following information after ensuring you are running latest version of forever-service and forever, also give your exact version number of both forever and forever-service

1) Ensure ledcom service is not running, by checking in sudo forever list 2) If it is running stop it, 3) Delete /var/log/ledcom.log 4) Delete /var/log/upstart/ledcom.log 5) Reboot the machine, and after reboot is complete 6) Take output of the following /var/log/ledcom.log 7) /var/log/upstart/ledcom.log 8) sudo forever list

[visualight]

Ok guy, thanks for your support. I hosted my app on a VPS server made by OVH (this was the problem) In fact my server never reboot and i see in many post on the internet that OVH hosting is really a shit So, i created a account on amazon and now ALL WORK FINE !!

Thank you :-) you can close this post :-)