search

kostya / eye

Process monitoring tool. Inspired from Bluepill and God.
MIT License
1.19k stars 89 forks source link

[celluloid] Actor crashed! ArgumentError: argument out of range /usr/lib/ruby/2.4.0/time.rb:306:in `local' #207

Closed rgaufman closed 6 years ago

rgaufman commented 6 years ago

I have a long running eye process:

deployer@host1:~$ ps aux | grep -i eye
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
deployer 30667  0.5  3.7 2564172 146152 ?      SNl   2017 512:39 eye monitoring v0.9.2.nosigar [recorder] (in /home/deployer)

However, one of the processes has crashed and eye did not automatically restart it. I'm trying to troubleshoot and I'm getting this:

$ bundle exec eye i
socket(/home/deployer/.eye/sock) not found, did you run `eye load`?

Eye does seem to be running however and when I do tail eye.log, I see output being generated:

deployer@host1:~/timeagent/log$ tail -f eye.log
[2018-01-06 23:21:55 +0000] [recorder:58ad5f763d1f9264de80eff1:proxy] check:rtsp Checking Camera 58ad5f763d1f9264de80eff1 with: openRTSP -v -d 0.01 -O -b 1000000 -u 'admin' 'admin' rtsp://127.0.0.1:10119/proxyStream
[2018-01-06 23:21:55 +0000] [recorder:58889d603d1f9262d09063d0:proxy] check:rtsp Checking Camera 58889d603d1f9262d09063d0 with: openRTSP -v -d 0.01 -O -b 1000000 -u 'admin' 'admin' rtsp://127.0.0.1:10116/proxyStream
...

Any ideas what would cause sock to go missing and is there any advice how to troubleshoot further?

kostya commented 6 years ago

something happen with this process, need to find problem in log. Try to kill eye kill -9 or eye quit and eye load

rgaufman commented 6 years ago

Thank you for the super fast reply!

Ah yes, when I look more carefully at the logs, I see this (this is the first error it shows):

[2018-01-04 13:46:14 +0000] [recorder:58889d603d1f9262b7906576:recorder] NOTIFY: Bounded ctime: [*13:44, *13:44] send to [:restart]
[2018-01-04 13:46:14 +0000] [recorder:58889d603d1f9262b7906576:recorder] trigger(starting_guard) Exception: attempted to call a dead actor: process_by_full_name ["/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/proxy/sync.rb:9:in `method_missing'", "/data/deployer/timeagent/eye/helpers.rb:145:in `process_started?'", "/data/deployer/timeagent/eye/helpers.rb:151:in `block in wait_for_process'", "/data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/trigger.rb:103:in `instance_exec'", "/data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/trigger.rb:103:in `exec_proc'", "/data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/trigger/starting_guard.rb:28:in `block in check_start'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/calls.rb:28:in `public_send'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/calls.rb:28:in `dispatch'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/call/sync.rb:16:in `dispatch'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/future.rb:18:in `block in new'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-essentials-0.20.5/lib/celluloid/internals/thread_handle.rb:14:in `block in initialize'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/actor/system.rb:78:in `block in get_thread'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/group/spawner.rb:50:in `block in instantiate'", "(celluloid):0:in `remote procedure call'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/future.rb:142:in `value'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/future.rb:96:in `value'", "/data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/trigger.rb:112:in `defer'", "/data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/trigger/starting_guard.rb:28:in `check_start'", "/data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/trigger/starting_guard.rb:23:in `check'", "/data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/trigger.rb:73:in `notify'", "/data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/process/trigger.rb:12:in `block in check_triggers'", "/data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/process/trigger.rb:12:in `each'", "/data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/process/trigger.rb:12:in `check_triggers'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/state_machines-0.5.0/lib/state_machines/eval_helpers.rb:58:in `evaluate_method'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/state_machines-0.5.0/lib/state_machines/callback.rb:192:in `block in run_methods'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/state_machines-0.5.0/lib/state_machines/callback.rb:191:in `each'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/state_machines-0.5.0/lib/state_machines/callback.rb:191:in `run_methods'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/state_machines-0.5.0/lib/state_machines/callback.rb:159:in `call'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/state_machines-0.5.0/lib/state_machines/transition.rb:394:in `block (2 levels) in after'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/state_machines-0.5.0/lib/state_machines/transition.rb:394:in `each'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/state_machines-0.5.0/lib/state_machines/transition.rb:394:in `block in after'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/state_machines-0.5.0/lib/state_machines/transition.rb:392:in `catch'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/state_machines-0.5.0/lib/state_machines/transition.rb:392:in `after'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/state_machines-0.5.0/lib/state_machines/transition.rb:195:in `run_callbacks'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:127:in `run_callbacks'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:63:in `block (2 levels) in perform'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:63:in `catch'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:63:in `block in perform'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:182:in `block in within_transaction'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/state_machines-0.5.0/lib/state_machines/transition.rb:171:in `block in within_transaction'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/state_machines-0.5.0/lib/state_machines/machine.rb:1868:in `block in within_transaction'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/state_machines-0.5.0/lib/state_machines/machine.rb:2164:in `transaction'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/state_machines-0.5.0/lib/state_machines/machine.rb:1868:in `within_transaction'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/state_machines-0.5.0/lib/state_machines/transition.rb:170:in `within_transaction'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:181:in `within_transaction'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:62:in `perform'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/state_machines-0.5.0/lib/state_machines/transition.rb:163:in `perform'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/state_machines-0.5.0/lib/state_machines/event.rb:155:in `fire'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/state_machines-0.5.0/lib/state_machines/event.rb:219:in `block in add_actions'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/state_machines-0.5.0/lib/state_machines/machine.rb:729:in `block (2 levels) in define_helper'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/state_machines-0.5.0/lib/state_machines/event.rb:224:in `block in add_actions'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/state_machines-0.5.0/lib/state_machines/machine.rb:729:in `block (2 levels) in define_helper'", "/data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/process/states.rb:11:in `switch'", "/data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/process/commands.rb:6:in `start_process'", "/data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/process/commands.rb:81:in `restart_process'", "/data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/process/controller.rb:22:in `restart'", "/data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/process/scheduler.rb:74:in `scheduler_consume'", "/data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/process/scheduler.rb:126:in `scheduler_run'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/calls.rb:28:in `public_send'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/calls.rb:28:in `dispatch'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/call/async.rb:7:in `dispatch'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/cell.rb:50:in `block in dispatch'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/cell.rb:76:in `block in task'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/actor.rb:339:in `block in task'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/task.rb:44:in `block in initialize'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/task/fibered.rb:14:in `block in create'"]

Any ideas what happened there?

The helper method it is referring to is this:

def process_started?(name)
  p = Eye::Control.process_by_full_name(name)
  p && p.state_name == :up
end

The specific line it's failing on is:

p = Eye::Control.process_by_full_name(name)

Also, it is very strange this is the first error it shows, because the process crashed on Jan 1st at 10:30am and eye is not reporting anything about it in the logs or showing any errors, hmm.

kostya commented 6 years ago

something crashed before this in log, in your line already dead actor. may be before 10:30am

rgaufman commented 6 years ago

I had the same experience on a few servers recently, strangely all was OK until the new year, but the past 6 days, random units exhibit a similar behaviour (maybe 10-15% of of 300 or so servers exhibited this).

What's the best way to troubleshoot what is going wrong, to prevent it in the future?

kostya commented 6 years ago

i don't know, need to find when problem in log appeared first. if your restart eye it helps?

rgaufman commented 6 years ago

I looked at all logs going back to 2017-12-12 06:00 and there are no other errors in the logs :(

Yes, restarting eye brought it back to normal, but doesn't help troubleshoot the issue.

After restarting:

$ bundle exec eye i
recorder
  588897a83d1f9262b7905c27
    detector ...................... up  (00:01, 18%, 80Mb, <11180>)
    proxy ......................... up  (00:01, 0%, 2Mb, <10450>)
    proxy2 ........................ up  (00:01, 2%, 3Mb, <10782>)
    recorder ...................... up  (00:01, 1%, 23Mb, <11339>)
  58889d5f3d1f9262c8905fb6
...
rgaufman commented 6 years ago

Here are all the logs unedited for your reference: https://www.dropbox.com/sh/gbatr3kst7yx04f/AAB8wlav2YMqOXMnGdQj3po5a?dl=0

EDIT: Is there any way to tell eye to just die completely if an actor is dead? - I use systemd to start eye, so that would at least auto restart it when this rare event happens.

rgaufman commented 6 years ago

Ooooh, I just managed to get some log output from another server that has this problem:

    def start_time(pid) # unixtime
      if st = cache.proc_start_time(pid)
        Time.parse(st).to_i
      end
    end

It seems whatever cache.proc_start_time(pid) reports made Time.parse(st).to_i throw an exception at midnight Jan 1 2018:

[2017-12-28 10:51:59 +0000] [recorder:594ff91c3d1f921c5ac6fa31:detector] check crashed: process is down
[2017-12-28 10:52:48 +0000] [recorder:594ff91c3d1f921c5ac6fa31:recorder] NOTIFY: Bounded ctime: [*10:50, *10:50] send to [:restart]
[2017-12-28 10:55:51 +0000] [recorder:594ff91c3d1f921c5ac6fa31:recorder] NOTIFY: Bounded ctime: [*10:50, *10:50] send to [:restart]
[2018-01-01 00:00:09 +0000] [celluloid] Actor crashed!
ArgumentError: argument out of range
    /usr/lib/ruby/2.4.0/time.rb:306:in `local'
    /usr/lib/ruby/2.4.0/time.rb:306:in `make_time'
    /usr/lib/ruby/2.4.0/time.rb:370:in `parse'
    /data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/system_resources.rb:24:in `start_time'
    /data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/system_resources.rb:63:in `resources'
    /data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/process/data.rb:40:in `self_status_data'
    /data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/process/data.rb:24:in `status_data'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/calls.rb:28:in `public_send'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/calls.rb:28:in `dispatch'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/call/sync.rb:16:in `dispatch'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/cell.rb:50:in `block in dispatch'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/cell.rb:76:in `block in task'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/actor.rb:339:in `block in task'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/task.rb:44:in `block in initialize'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/task/fibered.rb:14:in `block in create'
[2018-01-01 00:00:09 +0000] [celluloid] Actor crashed!
ArgumentError: argument out of range
    /usr/lib/ruby/2.4.0/time.rb:306:in `local'
    /usr/lib/ruby/2.4.0/time.rb:306:in `make_time'
    /usr/lib/ruby/2.4.0/time.rb:370:in `parse'
    /data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/system_resources.rb:24:in `start_time'
    /data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/system_resources.rb:63:in `resources'
    /data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/process/data.rb:40:in `self_status_data'
    /data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/process/data.rb:24:in `status_data'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/calls.rb:28:in `public_send'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/calls.rb:28:in `dispatch'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/call/sync.rb:16:in `dispatch'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/cell.rb:50:in `block in dispatch'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/cell.rb:76:in `block in task'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/actor.rb:339:in `block in task'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/task.rb:44:in `block in initialize'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/task/fibered.rb:14:in `block in create'
    (celluloid):0:in `remote procedure call'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/call/sync.rb:45:in `value'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/proxy/sync.rb:22:in `method_missing'
    /data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/group/data.rb:4:in `block in status_data'
    /data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/utils/alive_array.rb:14:in `block in each'
    /data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/utils/alive_array.rb:14:in `each'
    /data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/utils/alive_array.rb:14:in `each'
    /data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/group/data.rb:4:in `map'
    /data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/group/data.rb:4:in `status_data'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/calls.rb:28:in `public_send'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/calls.rb:28:in `dispatch'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/call/sync.rb:16:in `dispatch'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/cell.rb:50:in `block in dispatch'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/cell.rb:76:in `block in task'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/actor.rb:339:in `block in task'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/task.rb:44:in `block in initialize'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/task/fibered.rb:14:in `block in create'
[2018-01-01 00:00:09 +0000] [celluloid] Actor crashed!
ArgumentError: argument out of range
    /usr/lib/ruby/2.4.0/time.rb:306:in `local'
    /usr/lib/ruby/2.4.0/time.rb:306:in `make_time'
    /usr/lib/ruby/2.4.0/time.rb:370:in `parse'
    /data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/system_resources.rb:24:in `start_time'
    /data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/system_resources.rb:63:in `resources'
    /data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/process/data.rb:40:in `self_status_data'
    /data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/process/data.rb:24:in `status_data'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/calls.rb:28:in `public_send'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/calls.rb:28:in `dispatch'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/call/sync.rb:16:in `dispatch'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/cell.rb:50:in `block in dispatch'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/cell.rb:76:in `block in task'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/actor.rb:339:in `block in task'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/task.rb:44:in `block in initialize'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/task/fibered.rb:14:in `block in create'
    (celluloid):0:in `remote procedure call'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/call/sync.rb:45:in `value'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/proxy/sync.rb:22:in `method_missing'
    /data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/group/data.rb:4:in `block in status_data'
    /data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/utils/alive_array.rb:14:in `block in each'
    /data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/utils/alive_array.rb:14:in `each'
    /data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/utils/alive_array.rb:14:in `each'
    /data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/group/data.rb:4:in `map'
    /data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/group/data.rb:4:in `status_data'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/calls.rb:28:in `public_send'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/calls.rb:28:in `dispatch'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/call/sync.rb:16:in `dispatch'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/cell.rb:50:in `block in dispatch'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/cell.rb:76:in `block in task'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/actor.rb:339:in `block in task'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/task.rb:44:in `block in initialize'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/task/fibered.rb:14:in `block in create'
    (celluloid):0:in `remote procedure call'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/call/sync.rb:45:in `value'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/proxy/sync.rb:22:in `method_missing'
    /data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/application.rb:29:in `block in status_data'
    /data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/utils/alive_array.rb:14:in `block in each'
    /data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/utils/alive_array.rb:14:in `each'
    /data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/utils/alive_array.rb:14:in `each'
    /data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/application.rb:29:in `map'
    /data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/application.rb:29:in `status_data'
    /data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/controller/status.rb:28:in `block in info_data'
    /data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/controller/status.rb:28:in `map'
    /data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/controller/status.rb:28:in `info_data'
    /data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/controller/commands.rb:47:in `command'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/calls.rb:28:in `public_send'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/calls.rb:28:in `dispatch'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/call/sync.rb:16:in `dispatch'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/cell.rb:50:in `block in dispatch'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/cell.rb:76:in `block in task'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/actor.rb:339:in `block in task'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/task.rb:44:in `block in initialize'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/task/fibered.rb:14:in `block in create'
    (celluloid):0:in `remote procedure call'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/call/sync.rb:45:in `value'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/proxy/sync.rb:22:in `method_missing'
    /data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/server.rb:51:in `handle_connection'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/calls.rb:28:in `public_send'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/calls.rb:28:in `dispatch'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/call/async.rb:7:in `dispatch'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/cell.rb:50:in `block in dispatch'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/cell.rb:76:in `block in task'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/actor.rb:339:in `block in task'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/task.rb:44:in `block in initialize'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/task/fibered.rb:14:in `block in create'
[2018-01-01 00:00:09 +0000] [celluloid] Actor crashed!
ArgumentError: argument out of range
    /usr/lib/ruby/2.4.0/time.rb:306:in `local'
    /usr/lib/ruby/2.4.0/time.rb:306:in `make_time'
    /usr/lib/ruby/2.4.0/time.rb:370:in `parse'
    /data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/system_resources.rb:24:in `start_time'
    /data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/system_resources.rb:63:in `resources'
    /data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/process/data.rb:40:in `self_status_data'
    /data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/process/data.rb:24:in `status_data'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/calls.rb:28:in `public_send'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/calls.rb:28:in `dispatch'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/call/sync.rb:16:in `dispatch'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/cell.rb:50:in `block in dispatch'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/cell.rb:76:in `block in task'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/actor.rb:339:in `block in task'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/task.rb:44:in `block in initialize'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/task/fibered.rb:14:in `block in create'
    (celluloid):0:in `remote procedure call'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/call/sync.rb:45:in `value'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/proxy/sync.rb:22:in `method_missing'
    /data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/group/data.rb:4:in `block in status_data'
    /data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/utils/alive_array.rb:14:in `block in each'
    /data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/utils/alive_array.rb:14:in `each'
    /data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/utils/alive_array.rb:14:in `each'
    /data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/group/data.rb:4:in `map'
    /data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/group/data.rb:4:in `status_data'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/calls.rb:28:in `public_send'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/calls.rb:28:in `dispatch'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/call/sync.rb:16:in `dispatch'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/cell.rb:50:in `block in dispatch'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/cell.rb:76:in `block in task'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/actor.rb:339:in `block in task'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/task.rb:44:in `block in initialize'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/task/fibered.rb:14:in `block in create'
    (celluloid):0:in `remote procedure call'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/call/sync.rb:45:in `value'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/proxy/sync.rb:22:in `method_missing'
    /data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/application.rb:29:in `block in status_data'
    /data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/utils/alive_array.rb:14:in `block in each'
    /data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/utils/alive_array.rb:14:in `each'
    /data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/utils/alive_array.rb:14:in `each'
    /data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/application.rb:29:in `map'
    /data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/application.rb:29:in `status_data'
    /data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/controller/status.rb:28:in `block in info_data'
    /data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/controller/status.rb:28:in `map'
    /data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/controller/status.rb:28:in `info_data'
    /data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/controller/commands.rb:47:in `command'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/calls.rb:28:in `public_send'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/calls.rb:28:in `dispatch'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/call/sync.rb:16:in `dispatch'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/cell.rb:50:in `block in dispatch'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/cell.rb:76:in `block in task'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/actor.rb:339:in `block in task'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/task.rb:44:in `block in initialize'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/task/fibered.rb:14:in `block in create'
    (celluloid):0:in `remote procedure call'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/call/sync.rb:45:in `value'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/proxy/sync.rb:22:in `method_missing'
    /data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/server.rb:51:in `handle_connection'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/calls.rb:28:in `public_send'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/calls.rb:28:in `dispatch'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/call/async.rb:7:in `dispatch'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/cell.rb:50:in `block in dispatch'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/cell.rb:76:in `block in task'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/actor.rb:339:in `block in task'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/task.rb:44:in `block in initialize'
    /data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/task/fibered.rb:14:in `block in create'
[2018-01-04 10:51:55 +0000] [recorder:594ff91c3d1f921c5ac6fa31:recorder] NOTIFY: Bounded ctime: [*10:49, *10:49] send to [:restart]
[2018-01-04 10:51:55 +0000] [recorder:594ff91c3d1f921c5ac6fa31:recorder] trigger(starting_guard) Exception: attempted to call a dead actor: process_by_full_name ["/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/proxy/sync.rb:9:in `method_missing'", "/data/deployer/timeagent/eye/helpers.rb:145:in `process_started?'", "/data/deployer/timeagent/eye/helpers.rb:151:in `block in wait_for_process'", "/data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/trigger.rb:103:in `instance_exec'", "/data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/trigger.rb:103:in `exec_proc'", "/data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/trigger/starting_guard.rb:28:in `block in check_start'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/calls.rb:28:in `public_send'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/calls.rb:28:in `dispatch'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/call/sync.rb:16:in `dispatch'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/future.rb:18:in `block in new'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-essentials-0.20.5/lib/celluloid/internals/thread_handle.rb:14:in `block in initialize'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/actor/system.rb:78:in `block in get_thread'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/group/spawner.rb:50:in `block in instantiate'", "(celluloid):0:in `remote procedure call'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/future.rb:142:in `value'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/future.rb:96:in `value'", "/data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/trigger.rb:112:in `defer'", "/data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/trigger/starting_guard.rb:28:in `check_start'", "/data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/trigger/starting_guard.rb:23:in `check'", "/data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/trigger.rb:73:in `notify'", "/data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/process/trigger.rb:12:in `block in check_triggers'", "/data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/process/trigger.rb:12:in `each'", "/data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/process/trigger.rb:12:in `check_triggers'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/state_machines-0.5.0/lib/state_machines/eval_helpers.rb:58:in `evaluate_method'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/state_machines-0.5.0/lib/state_machines/callback.rb:192:in `block in run_methods'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/state_machines-0.5.0/lib/state_machines/callback.rb:191:in `each'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/state_machines-0.5.0/lib/state_machines/callback.rb:191:in `run_methods'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/state_machines-0.5.0/lib/state_machines/callback.rb:159:in `call'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/state_machines-0.5.0/lib/state_machines/transition.rb:394:in `block (2 levels) in after'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/state_machines-0.5.0/lib/state_machines/transition.rb:394:in `each'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/state_machines-0.5.0/lib/state_machines/transition.rb:394:in `block in after'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/state_machines-0.5.0/lib/state_machines/transition.rb:392:in `catch'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/state_machines-0.5.0/lib/state_machines/transition.rb:392:in `after'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/state_machines-0.5.0/lib/state_machines/transition.rb:195:in `run_callbacks'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:127:in `run_callbacks'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:63:in `block (2 levels) in perform'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:63:in `catch'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:63:in `block in perform'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:182:in `block in within_transaction'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/state_machines-0.5.0/lib/state_machines/transition.rb:171:in `block in within_transaction'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/state_machines-0.5.0/lib/state_machines/machine.rb:1868:in `block in within_transaction'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/state_machines-0.5.0/lib/state_machines/machine.rb:2164:in `transaction'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/state_machines-0.5.0/lib/state_machines/machine.rb:1868:in `within_transaction'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/state_machines-0.5.0/lib/state_machines/transition.rb:170:in `within_transaction'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:181:in `within_transaction'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:62:in `perform'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/state_machines-0.5.0/lib/state_machines/transition.rb:163:in `perform'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/state_machines-0.5.0/lib/state_machines/event.rb:155:in `fire'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/state_machines-0.5.0/lib/state_machines/event.rb:219:in `block in add_actions'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/state_machines-0.5.0/lib/state_machines/machine.rb:729:in `block (2 levels) in define_helper'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/state_machines-0.5.0/lib/state_machines/event.rb:224:in `block in add_actions'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/state_machines-0.5.0/lib/state_machines/machine.rb:729:in `block (2 levels) in define_helper'", "/data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/process/states.rb:11:in `switch'", "/data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/process/commands.rb:6:in `start_process'", "/data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/process/commands.rb:81:in `restart_process'", "/data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/process/controller.rb:22:in `restart'", "/data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/process/scheduler.rb:74:in `scheduler_consume'", "/data/deployer/timeagent/vendor/cache/eye-fd6c0a79f201/lib/eye/process/scheduler.rb:126:in `scheduler_run'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/calls.rb:28:in `public_send'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/calls.rb:28:in `dispatch'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/call/async.rb:7:in `dispatch'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/cell.rb:50:in `block in dispatch'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/cell.rb:76:in `block in task'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/actor.rb:339:in `block in task'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/task.rb:44:in `block in initialize'", "/data/deployer/timeagent/vendor/bundle/ruby/2.4.0/gems/celluloid-0.17.3/lib/celluloid/task/fibered.rb:14:in `block in create'"]
[2018-01-04 16:01:30 +0000] [recorder:594ff91c3d1f921c5ac6fa31:proxy] check:rtsp Camera 594ff91c3d1f921c5ac6fa31 error: 404 Bad path (rtsp://127.0.0.1:10101/proxyStream)
[2018-01-04 16:39:30 +0000] [recorder:594ff91c3d1f921c5ac6fa31:proxy] check:rtsp Camera 594ff91c3d1f921c5ac6fa31 error: 404 Bad path (rtsp://127.0.0.1:10101/proxyStream)
[2018-01-04 17:03:30 +0000] [recorder:594ff91c3d1f921c5ac6fa31:proxy] check:rtsp Camera 594ff91c3d1f921c5ac6fa31 error: 404 Bad path (rtsp://127.0.0.1:10101/proxyStream)
[2018-01-04 18:03:30 +0000] [recorder:594ff91c3d1f921c5ac6fa31:proxy] check:rtsp Camera 594ff91c3d1f921c5ac6fa31 error: 404 Bad path (rtsp://127.0.0.1:10101/proxyStream)
[2018-01-04 18:07:30 +0000] [recorder:594ff91c3d1f921c5ac6fa31:proxy] check:rtsp Camera 594ff91c3d1f921c5ac6fa31 error: 404 Bad path (rtsp://127.0.0.1:10101/proxyStream)
[2018-01-04 18:33:30 +0000] [recorder:594ff91c3d1f921c5ac6fa31:proxy] check:rtsp Camera 594ff91c3d1f921c5ac6fa31 error: 404 Bad path (rtsp://127.0.0.1:10101/proxyStream)
[2018-01-04 18:50:30 +0000] [recorder:594ff91c3d1f921c5ac6fa31:proxy] check:rtsp Camera 594ff91c3d1f921c5ac6fa31 error: 404 Bad path (rtsp://127.0.0.1:10101/proxyStream)
[2018-01-04 18:55:30 +0000] [recorder:594ff91c3d1f921c5ac6fa31:proxy] check:rtsp Camera 594ff91c3d1f921c5ac6fa31 error: 404 Bad path (rtsp://127.0.0.1:10101/proxyStream)
[2018-01-04 19:30:30 +0000] [recorder:594ff91c3d1f921c5ac6fa31:proxy] check:rtsp Camera 594ff91c3d1f921c5ac6fa31 error: 404 Bad path (rtsp://127.0.0.1:10101/proxyStream)
[2018-01-04 20:33:30 +0000] [recorder:594ff91c3d1f921c5ac6fa31:proxy] check:rtsp Camera 594ff91c3d1f921c5ac6fa31 error: 404 Bad path (rtsp://127.0.0.1:10101/proxyStream)
[2018-01-04 20:51:30 +0000] [recorder:594ff91c3d1f921c5ac6fa31:proxy] check:rtsp Camera 594ff91c3d1f921c5ac6fa31 error: 404 Bad path (rtsp://127.0.0.1:10101/proxyStream)
[2018-01-04 21:37:30 +0000] [recorder:594ff91c3d1f921c5ac6fa31:proxy] check:rtsp Camera 594ff91c3d1f921c5ac6fa31 error: 404 Bad path (rtsp://127.0.0.1:10101/proxyStream)
[2018-01-04 21:53:30 +0000] [recorder:594ff91c3d1f921c5ac6fa31:proxy] check:rtsp Camera 594ff91c3d1f921c5ac6fa31 error: 404 Bad path (rtsp://127.0.0.1:10101/proxyStream)
[2018-01-04 22:00:30 +0000] [recorder:594ff91c3d1f921c5ac6fa31:proxy] check:rtsp Camera 594ff91c3d1f921c5ac6fa31 error: 404 Bad path (rtsp://127.0.0.1:10101/proxyStream)
[2018-01-05 03:30:30 +0000] [recorder:594ff91c3d1f921c5ac6fa31:proxy] check:rtsp Camera 594ff91c3d1f921c5ac6fa31 error: 404 Bad path (rtsp://127.0.0.1:10101/proxyStream)
[2018-01-05 04:00:30 +0000] [recorder:594ff91c3d1f921c5ac6fa31:proxy] check:rtsp Camera 594ff91c3d1f921c5ac6fa31 error: 404 Bad path (rtsp://127.0.0.1:10101/proxyStream)
[2018-01-05 07:40:30 +0000] [recorder:594ff91c3d1f921c5ac6fa31:proxy] check:rtsp Camera 594ff91c3d1f921c5ac6fa31 error: 404 Bad path (rtsp://127.0.0.1:10101/proxyStream)
[2018-01-05 12:25:30 +0000] [recorder:594ff91c3d1f921c5ac6fa31:proxy] check:rtsp Camera 594ff91c3d1f921c5ac6fa31 error: 404 Bad path (rtsp://127.0.0.1:10101/proxyStream)
[2018-01-05 17:10:31 +0000] [recorder:594ff91c3d1f921c5ac6fa31:proxy] check:rtsp Camera 594ff91c3d1f921c5ac6fa31 error: 404 Bad path (rtsp://127.0.0.1:10101/proxyStream)
[2018-01-05 17:46:31 +0000] [recorder:594ff91c3d1f921c5ac6fa31:proxy] check:rtsp Camera 594ff91c3d1f921c5ac6fa31 error: 404 Bad path (rtsp://127.0.0.1:10101/proxyStream)
[2018-01-05 18:25:31 +0000] [recorder:594ff91c3d1f921c5ac6fa31:proxy] check:rtsp Camera 594ff91c3d1f921c5ac6fa31 error: 404 Bad path (rtsp://127.0.0.1:10101/proxyStream)
[2018-01-05 19:15:31 +0000] [recorder:594ff91c3d1f921c5ac6fa31:proxy] check:rtsp Camera 594ff91c3d1f921c5ac6fa31 error: 404 Bad path (rtsp://127.0.0.1:10101/proxyStream)
[2018-01-05 21:06:31 +0000] [recorder:594ff91c3d1f921c5ac6fa31:proxy] check:rtsp Camera 594ff91c3d1f921c5ac6fa31 error: 404 Bad path (rtsp://127.0.0.1:10101/proxyStream)
[2018-01-05 21:11:31 +0000] [recorder:594ff91c3d1f921c5ac6fa31:proxy] check:rtsp Camera 594ff91c3d1f921c5ac6fa31 error: 404 Bad path (rtsp://127.0.0.1:10101/proxyStream)
[2018-01-05 21:31:31 +0000] [recorder:594ff91c3d1f921c5ac6fa31:proxy] check:rtsp Camera 594ff91c3d1f921c5ac6fa31 error: 404 Bad path (rtsp://127.0.0.1:10101/proxyStream)
[2018-01-06 05:34:31 +0000] [recorder:594ff91c3d1f921c5ac6fa31:proxy] check:rtsp Camera 594ff91c3d1f921c5ac6fa31 error: 404 Bad path (rtsp://127.0.0.1:10101/proxyStream)
[2018-01-06 07:57:31 +0000] [recorder:594ff91c3d1f921c5ac6fa31:proxy] check:rtsp Camera 594ff91c3d1f921c5ac6fa31 error: 404 Bad path (rtsp://127.0.0.1:10101/proxyStream)
[2018-01-06 08:13:31 +0000] [recorder:594ff91c3d1f921c5ac6fa31:proxy] check:rtsp Camera 594ff91c3d1f921c5ac6fa31 error: 404 Bad path (rtsp://127.0.0.1:10101/proxyStream)
[2018-01-06 16:39:31 +0000] [recorder:594ff91c3d1f921c5ac6fa31:proxy] check:rtsp Camera 594ff91c3d1f921c5ac6fa31 error: 404 Bad path (rtsp://127.0.0.1:10101/proxyStream)
[2018-01-06 16:43:31 +0000] [recorder:594ff91c3d1f921c5ac6fa31:proxy] check:rtsp Camera 594ff91c3d1f921c5ac6fa31 error: 404 Bad path (rtsp://127.0.0.1:10101/proxyStream)
[2018-01-06 16:49:31 +0000] [recorder:594ff91c3d1f921c5ac6fa31:proxy] check:rtsp Camera 594ff91c3d1f921c5ac6fa31 error: 404 Bad path (rtsp://127.0.0.1:10101/proxyStream)
[2018-01-06 17:46:31 +0000] [recorder:594ff91c3d1f921c5ac6fa31:proxy] check:rtsp Camera 594ff91c3d1f921c5ac6fa31 error: 404 Bad path (rtsp://127.0.0.1:10101/proxyStream)
[2018-01-06 17:57:31 +0000] [recorder:594ff91c3d1f921c5ac6fa31:proxy] check:rtsp Camera 594ff91c3d1f921c5ac6fa31 error: 404 Bad path (rtsp://127.0.0.1:10101/proxyStream)
[2018-01-06 18:02:31 +0000] [recorder:594ff91c3d1f921c5ac6fa31:proxy] check:rtsp Camera 594ff91c3d1f921c5ac6fa31 error: 404 Bad path (rtsp://127.0.0.1:10101/proxyStream)
[2018-01-06 18:17:31 +0000] [recorder:594ff91c3d1f921c5ac6fa31:proxy] check:rtsp Camera 594ff91c3d1f921c5ac6fa31 error: 404 Bad path (rtsp://127.0.0.1:10101/proxyStream)
[2018-01-06 18:21:31 +0000] [recorder:594ff91c3d1f921c5ac6fa31:proxy] check:rtsp Camera 594ff91c3d1f921c5ac6fa31 error: 404 Bad path (rtsp://127.0.0.1:10101/proxyStream)
[2018-01-06 18:33:31 +0000] [recorder:594ff91c3d1f921c5ac6fa31:proxy] check:rtsp Camera 594ff91c3d1f921c5ac6fa31 error: 404 Bad path (rtsp://127.0.0.1:10101/proxyStream)
[2018-01-06 18:59:31 +0000] [recorder:594ff91c3d1f921c5ac6fa31:proxy] check:rtsp Camera 594ff91c3d1f921c5ac6fa31 error: 404 Bad path (rtsp://127.0.0.1:10101/proxyStream)
[2018-01-06 20:34:31 +0000] [recorder:594ff91c3d1f921c5ac6fa31:proxy] check:rtsp Camera 594ff91c3d1f921c5ac6fa31 error: 404 Bad path (rtsp://127.0.0.1:10101/proxyStream)
[2018-01-06 20:53:31 +0000] [recorder:594ff91c3d1f921c5ac6fa31:proxy] check:rtsp Camera 594ff91c3d1f921c5ac6fa31 error: 404 Bad path (rtsp://127.0.0.1:10101/proxyStream)
[2018-01-06 21:47:31 +0000] [recorder:594ff91c3d1f921c5ac6fa31:proxy] check:rtsp Camera 594ff91c3d1f921c5ac6fa31 error: 404 Bad path (rtsp://127.0.0.1:10101/proxyStream)
[2018-01-06 23:09:31 +0000] [recorder:594ff91c3d1f921c5ac6fa31:proxy] check:rtsp Camera 594ff91c3d1f921c5ac6fa31 error: 404 Bad path (rtsp://127.0.0.1:10101/proxyStream)
kostya commented 6 years ago

just pushed 0.9.4.nosigar gem, this should not crash anymore (whatever it was parsing ruby time bug). Actors should not die at all, only in some very very strange and rare situations like this. But maybe i add some quit for eye in this situation.

rgaufman commented 6 years ago

Thank you very much for fixing this so quickly. Having an option to quit eye if an actor crashes would also be very much appreciated as it's difficult to predict every possible eventuality :)

kostya commented 6 years ago

btw, eye 0.10.0 fix sigar installation, so nosigar version not needed anymore.

rgaufman commented 6 years ago

Ah, brilliant, thank you!