How to add pre-requisites for a process (commands the must be run once every restart)

innovia commented 9 years ago

Hi Kostya

I have tried all day but i m not getting to a solution

my issue is that i have a few process

elasticMQ (Amazon local SQS) simple queue service Local Amazon DynamoDB (database) server.js

before running server.js i have to create tables and queues on every restart since they are not persistant I have to create the queues (single command) I have to create the tables (single command)

so i tried with dependencies and setting up transitions to up with back ticks to run the commands but its not creating the tables and queues

i ended up running a bash wrapper that does all of that and start server.js at the end

but it wasn't monitoing the child process so i've added monitoring_children which gets the child but doesn't transfer the stop command (kill -2 {PID}) to the child and so i have left with a running process unattached to eye

can you please help me out here?

here's the wrapper

#!/bin/bash

echo "Creating Queues"
node scripts/createQueues.js Vagrant

echo "Listing Queues"
node scripts/listQueues.js

echo "Creating dynamodb Tables"
node scripts/createDynamoDBTable.js

echo "create mysql tables"
mysql -u root -h localhost --password=xxxx < scripts/createMysqlTables.sql

echo "initializing bot"
node scripts/initializeDynamoBot.js

echo "starting server at $PORT"
node server.js

here's my eye config

#!/usr/bin/env ruby
env_vars =  `cat /server/.vagrantenv`.split("\n")

Eye.config do
  mail :host => "email-smtp.us-east-1.amazonaws.com",
       :port => 587,
       :domain => 'us-east-1.amazonaws.com',
       :auth => :login,
       :user =>  'ABCDEF',
       :password =>  'xxxxx',
       :starttls => true,
       :from_mail => 'tech@server.me',
       :from_name => "eye - #{`hostname`.chomp}"

  contact :tech, :mail, 'ami@server.me'

  logger '/tmp/eye.log'
  logger_level Logger::DEBUG
end

class Eye::Notify::Mail < Eye::Notify
  def message
    h = []
    h << "From: #{from_name} <#{from_mail || user}>" if from_mail || user
    h << "To: <#{contact}>"
    h << "Subject: #{message_subject}"
    h << "MIME-Version: 1.0"
    h << "Content-type: text/html"
    h << "Date: #{msg_at.httpdate}"
    "#{h * "\n"}\n#{message_body}"
  end

  def message_subject
    "#{msg_name} on #{msg_host} - #{msg_message}"
  end

  def message_body
    public_ip = `/sbin/ifconfig eth0 | grep 'inet addr:' | cut -d: -f2 | awk '{ print $1}'`.chomp
    m = []
    m << "<body>"
    m << "<br><h3>#{msg_name.to_s} Queue Info: </h3><br>"
    m << "SSH:   <a href=\"ssh://vagrant@#{public_ip}\"> ssh vagrant@#{public_ip} </a><br>"
    m << "Branch: <b>#{ENV['GIT_BRANCH']}</b> <br>"
    m << "Commit: <a href=\"https://bitbucket.org/server/commits/#{ENV['GIT_COMMIT']}\"> Commit on BitBucket </a> <br><br>"
    m << "<b>StdErr Log Tail:</b><br> <pre>#{`tail -n 30 /var/log/queues/#{msg_name}_error.log`} </pre><br>"
    m << "<b>StdOut Log Tail:</b><br> <pre>#{`tail -n 30 /var/log/queues/#{msg_name}.log`} </pre><br>"
    m << "</body>"
    debug "message body: #{m * "\n"}"
    "#{m * "\n"}"
  end
end

Eye.application(:vagrantbox) do
  env_vars.each do |e|
    env e.split('=').first => e.split('=').last
  end

  env "PORT" => "8080"
  env "HTTP_PORT" => "8081"
  env "VAGRANTBOX" => 'true'

  queues_info = [
      {:name => 'apnQueue', :workers => 1},
      {:name => 'fbQueue', :workers => 1},
      {:name => 'picSaverProcessor', :workers => 1},
      {:name => 'contactRequestQueue', :workers => 1},
      {:name => 'multicastMessageQueue', :workers => 1},
      {:name => 'celebQueue', :workers => 1},
      {:name => 'notifyFriendsQueue', :workers => 1},
      {:name => 'loginCodesQueue', :workers => 1}
  ]

  singleton_tasks = [
      {:name => 'cacheFansThreadInS3', :workers => 1},
      {:name => 'meetNewPeopleCloud', :workers => 1},
      {:name => 'doubleRemind', :workers => 1}
  ]

  group "services" do

    process "sqs" do
      working_dir '/etc/aws_mock_services'
      pid_file "/home/vagrant/eye/sqs.pid"
      stdall "/home/vagrant/eye/sqs.log"
      start_command "/usr/bin/java -Djava.library.path -jar ./elasticmq-server-0.8.2.jar"
      stop_command   "kill -9 {PID}"
      restart_command  "kill -HUP {PID}"
      daemonize true
    end

    process "dynamodb" do
      working_dir '/etc/aws_mock_services'
      pid_file "/home/vagrant/eye/dynamodb.pid"
      stdall "/home/vagrant/eye/dynamodb.log"
      start_command "/usr/bin/java -Djava.library.path=./DynamoDBLocal_lib -jar DynamoDBLocal.jar -dbPath /data/dynamodb -port 4567"
      stop_command   "kill -9 {PID}"
      restart_command  "kill -HUP {PID}"
      daemonize true
    end

    process "mongo" do
      pid_file "/var/run/mongo/mongod.pid"
      start_command "sudo /sbin/start mongod"
      stop_command "sudo /sbin/stop mongod"
      restart_command "sudo /sbin/restart mongod"
    end

    process "redis" do
      pid_file "/etc/redis/redis.pid"
      start_command "/usr/local/bin/redis-server /etc/redis/redis.conf"
      stop_command   "kill -QUIT {PID}"
      restart_command  "kill -HUP {PID}"
    end

    process "server" do
      depend_on [:sqs, :dynamodb, :redis, :mongo]
      working_dir "/server"
      pid_file "/home/vagrant/eye/server.pid"
      stdall "/home/vagrant/eye/server.log"
      start_command "./startVagrantServer.sh"
      stop_on_delete true
      monitor_children do
        stop_command "kill -9 {PID}"
      end
      daemonize true
    end

    queues_info.each do |queue|
      queue[:workers].times do |w|
        process "#{queue[:name]}_#{w+1}" do
            depend_on :server
            working_dir "/home/vagrant/server/"
            pid_file "#{queue[:name]}_#{w+1}.pid"
            stderr "/home/vagrant/eye/#{queue[:name]}_#{w+1}_error.log"
            stdout "/home/vagrant/eye/#{queue[:name]}_#{w+1}.log"
            start_command "/usr/bin/env node #{queue[:name]}.js"
            stop_signals [:QUIT, 15.seconds, :TERM, 15.seconds, :KILL]
            daemonize true
        end
      end
    end

    singleton_tasks.each do |queue|
      queue[:workers].times do |w|
        process "#{queue[:name]}_#{w+1}" do
            depend_on :server
            working_dir "/home/vagrant/server/"
            pid_file "#{queue[:name]}_#{w+1}.pid"
            stderr "/home/vagrant/eye/#{queue[:name]}_#{w+1}_error.log"
            stdout "/home/vagrant/eye/#{queue[:name]}_#{w+1}.log"
            start_command "/usr/bin/env node singletonScripts/#{queue[:name]}.js"
            stop_signals [:QUIT, 15.seconds, :TERM, 15.seconds, :KILL]
            daemonize true
        end
      end
    end

  end
end

kostya commented 9 years ago

you should not use shell script and daemonize true. because it is not properly monitored. shell script produce tree of processes, ./startVagrantServer.sh -> node server.js, and eye whould monitor only ./startVagrantServer.sh, and send kill to it, which is not correct. (because after kill, node server.js still be alive).

Solutions:

not use daemonize true and do daemonization for node server.js by hands.
try option use_leaf_child true, this should solve situation with shell fork tree, but it is experimental.
use trigger transition, on before start, and do all things here.

innovia commented 9 years ago

there's actually a trigger for before start? can i use back ticks there like so?

trigger :transition, :to => :before_start, :do => -> {
         debug "creating sqs queues"
        `/usr/local/bin/node scripts/createQueues.js Vagrant`

        debug "listing sqs queues"
        `/usr/local/bin/node scripts/listQueues.js`

        debug "creatign dynamodb tables"
        `/usr/local/bin/node scripts/createDynamoDBTable.js`

        `/usr/bin/mysql -u root -h localhost --password=123456 < scripts/createMysqlTables.sql`
        `/usr/local/bin/node scripts/initializeDynamoBot.js`
      }

kostya commented 9 years ago

this should work:

      trigger :transition, to: :starting, do: -> {
        cmd = "touch /tmp/111"
        res = process.execute(cmd, process.config.merge(:timeout => 10.seconds))
        info "executing `#{cmd}` with result: #{res}"
      }

kostya commented 9 years ago

backticks is not good, because it can block the process actor.

innovia commented 9 years ago

Thank you so much for always being prompt and helpful I really appreciate your help and I think eye is an excellent product!

Thanks for the info I'll give it a try tomorrow

innovia commented 9 years ago

Thank you that resolved the issue!

kostya commented 9 years ago

in master i add helpers execute_async, execute_sync:

      trigger :transition, to: :starting, do: -> {
        process.execute_sync "touch /tmp/111"
        process.execute_sync "sleep 2", :timeout => 1
        process.execute_async "sleep 2"        
      }

innovia commented 9 years ago

There's an issue when you pass more than one param to cmd, like so

cmd = "/usr/bin/mysql -uroot -h127.0.0.1 -p123456 < scripts/createMysqlTables.sql"

only mysql triggered

or for example

cmd = "mv /a/b/c/file1 /d/"

mv fails, again more than one param

kostya commented 9 years ago

for me this is ok

+
+      trigger :transition, to: :starting, do: -> {
+        process.execute_sync "cp trash.log /tmp/"
+      }

12.11.2014 17:14:53 INFO -- [test:samples:sample1] execute_synccp trash.log /tmp/with res: {:pid=>5137, :exitstatus=>0}

file copied

you also can move all your commands into shell script, and run only it.

kostya / eye

How to add pre-requisites for a process (commands the must be run once every restart) #88