As discussed in #84 (see this comment), Moonfire NVR doesn't shut down properly on SIGTERM or SIGINT (as when pressing ctrl-C) if it's retrying writes to recording files. This can happen when out of space or due to filesystem corruption. The symptom is that you see a bunch of log lines like this:
W20210401 11:21:02.362 s-driveway-main moonfire_base::clock] sleeping for Duration { secs: 1, nanos: 0 } after error: No space left on device (os error 28)
(set environment variable RUST_BACKTRACE=1 to see backtraces)
W20210401 11:21:03.362 s-driveway-main moonfire_base::clock] sleeping for Duration { secs: 1, nanos: 0 } after error: No space left on device (os error 28)
(set environment variable RUST_BACKTRACE=1 to see backtraces)
W20210401 11:21:04.363 s-driveway-main moonfire_base::clock] sleeping for Duration { secs: 1, nanos: 0 } after error: No space left on device (os error 28)
(set environment variable RUST_BACKTRACE=1 to see backtraces)
W20210401 11:21:05.363 s-driveway-main moonfire_base::clock] sleeping for Duration { secs: 1, nanos: 0 } after error: No space left on device (os error 28)
(set environment variable RUST_BACKTRACE=1 to see backtraces)
W20210401 11:21:06.364 s-driveway-main moonfire_base::clock] sleeping for Duration { secs: 1, nanos: 0 } after error: No space left on device (os error 28)
...
so you tell it to shut down. It logs this:
I20210401 11:21:07.542 main moonfire_nvr::cmds::run] Shutting down streamers.
but never actually shuts down, and keeps logging the sleeping message. Sending another SIGINT or SIGTERM does nothing.
The problem is the streamer threads and sample file syncer threads get into a "retry forever" loop, and they really mean forever. Instead, they should give up after shut down is requested.
Workaround: send it a SIGQUIT (ctrl-\ if Moonfire NVR is running from a terminal) or SIGKILL. These will kill it immediately.
It'd also be nice if it spammed the logs less during the retry loop. It's pretty verbose now, especially if you have backtraces enabled. Maybe only log once per minute instead of on every single attempt.
As discussed in #84 (see this comment), Moonfire NVR doesn't shut down properly on
SIGTERM
orSIGINT
(as when pressing ctrl-C) if it's retrying writes to recording files. This can happen when out of space or due to filesystem corruption. The symptom is that you see a bunch of log lines like this:so you tell it to shut down. It logs this:
but never actually shuts down, and keeps logging the
sleeping
message. Sending anotherSIGINT
orSIGTERM
does nothing.The problem is the streamer threads and sample file syncer threads get into a "retry forever" loop, and they really mean forever. Instead, they should give up after shut down is requested.
Workaround: send it a
SIGQUIT
(ctrl-\ if Moonfire NVR is running from a terminal) orSIGKILL
. These will kill it immediately.