support "monitor mode" / live view of unrecorded streams

[IronOxidizer]

Describe the bug Live view produces a black screen when record is disabled in config. This can be worked around by enabling record, but I would prefer that the data never touches the drives in consideration of drive longevity. No errors are reported on client and server, websocket is connected with response 101, and pings are sent regularly to keep the connection alive.

To Reproduce Steps to reproduce the behavior:

1. enable sub stream in config
2. disable record
3. start server
4. go to live view, black screen

Expected behavior Live stream working without recording. Resembles a "monitoring" mode in other NVRs

Server (please complete the following information):

• master based on 98d106553aa2fb648413c6d0f7b8a592b788d2ee
• manually compiled

Platform:

• Linux, Windows, Android
• Chrome, Firefox

Additional context Seems to be originating from here as a result of the Receiver never yielding a LiveSegment

https://github.com/scottlamb/moonfire-nvr/blob/master/server/src/web.rs#L495

[scottlamb]

I'm going to call this enhancement rather than bug because this is an expected part of the "just record everything" design today.

This can be worked around by enabling record, but I would prefer that the data never touches the drives in consideration of drive longevity.

Do you have any evidence that lowering the write rate significantly improves the reliability of classic (PMR, perpendicular magnetic recording) disks? SMR ones (shingled magnetic recording) have a significantly lower rated workload, but the hardware guide already notes that and cautions against them. I think this rating is because they may have an onboard SSD that everything gets staged through, or if they have a PMR staging area, then they do a lot of extra seeking and rewriting.

My assumption to date has been that SMR drive failures don't happen due to write throughput (predominately or maybe at all). They're because of other things like manufacturing defect (defect in the platter or seal), catastrophic events (mishandling), bearings wearing out (sometimes they just don't spin up after a power outage), and maybe to a lesser extent high seek rate (head movement). I just skimmed the wikipedia article on hard drive failures and don't see any mention of writes accelerating failure.

Starting from this assumption, I decided to just record everything, and that shows up several times in the current design:

• Streamer is designed to just grab video and write it to storage (SQLite database + sample files). It doesn't keep video in RAM or distribute it to subscribers. A Streamer is never instantiated if record is false on the stream row in the database.
• mp4::File is based on mp4::Segment which is based on db::recording::Segment which is instantiated from the database, and pulls in video exclusively from disk.
• LiveSegment is kind of a misnomer. It's not actually a segment of live video; it's a reference to a recording::Segment. It's basically saying there is a frame of video. When the WebSocket code in web.rs sees that, it uses it to create a segment. For the very first one, it's everything from the previous key frame to this frame (so the video can start quicker than most live stream software allows). For others, it's just that frame.
• even the headers sent over the WebSocket assume it comes from a recording. It deals with a recording id (assigned by the database layer) and "prev media time" tracked by the database layer.

I've broadened my use cases a bit though since originally designing this and would like to eventually support monitor mode. A couple reasons for the change of heart:

• It's sometimes useful to have cameras that you don't want recorded for privacy reasons. Eg baby cameras. I don't like the idea of recording video inside my house, but when my kids were younger I checked on them with live view of RTSP baby cameras (not through Moonfire NVR; it didn't support live view at all then).
• Many cameras support three streams. It might be useful to have separate streams for high-quality and low-quality playback streams and a dedicated video analytics stream (medium resolution, low frame rate) to have good video detection with low H.264 decoding and scaling overhead, without having to spend extra disk space on the analytics stream.

Anyway, off-hand to support monitor mode, these changes need to happen:

• We need some way of indicating we want "monitor" mode. Probably a small schema change from just having a record boolean to having an enum or flags. Or maybe as temporary measure (to allow work to proceed before creating a new-schema branch), a commandline argument of stream ids to monitor.
• Streamer needs to be the thing that increments the "prev media time" (even when writing to the database isn't happening). The persistence layer (the moonfire-db crate) should enforce that the prev media time makes sense (doesn't go backward) but should otherwise accept what Streamer gives it, and we update the comments in the sql file to indicate there can be gaps.
• Websockets' subscription shouldn't go through the db crate but stay within the moonfire_nvr crate, and they need a record to a moonfire-nvr-layer streamer/stream object. That should keep in-RAM the state from the previous key frame to now, so they can send it immediately on a new WebSocket connection. Then each frame should be sent to all existing Websocket subscriptions.
• mp4::Segment needs to allow this in-memory segment as an alternative to a recording::Segment. Probably a Box<dyn base::Recording> (where base::Recording is a new trait implemented by both db::recording::Segment and moonfire_nvr::InMemorySegment).

An alternative design would be to write the sample files to tmpfs and perhaps change less of the code. Never flush those recordings to the SQLite database either; keep them pending until they're dropped. But that has its own compromises; it'd either use a lot of RAM or would need shorter recordings which might cause some complications like "anchoring" the stream's wall time sooner (thus having the time further from correct). (or maybe they'd never be anchored. maybe live view doesn't really need anchoring anyway.)

I don't think this refactoring is in my idea of a "minimum viable product" feature set, so I don't plan to work on it for a while. If you want to work on it, I'd happily help you through it. It's a significant project though so it might be wise to tackle a couple small bugs or something to acclimate more to the codebase before getting into this.

[IronOxidizer]

Do you have any evidence that lowering the write rate significantly improves the reliability of classic (PMR, perpendicular magnetic recording) disks? SMR ones (shingled magnetic recording) have a significantly lower rated workload, but the hardware guide already notes that and cautions against them. I think this rating is because they may have an onboard SSD that everything gets staged through, or if they have a PMR staging area, then they do a lot of extra seeking and rewriting.

I didn't have any evidence, mostly went on the notion that some drives are rated to a maximum amount of writes per day. Did some further research and found this paper from micron comparing the endurance of SSD vs HDD.

https://www.micron.com/-/media/client/global/documents/products/white-paper/5210_ssd_vs_hdd_endurance_white_paper.pdf

They don't specifically say that HDDs have a rated DRWPD, but they do use DRWPD as a measure of hard drive endurance.

I don't think this refactoring is in my idea of a "minimum viable product" feature set, so I don't plan to work on it for a while. If you want to work on it, I'd happily help you through it. It's a significant project though so it might be wise to tackle a couple small bugs or something to acclimate more to the codebase before getting into this.

Seems like quite the large undertaking, might revisit this in the future. For now I'll see if I can help with things more relevant to the 1.0 release.

[scottlamb]

Interesting paper. They say "as their capacities increased, some HDD designs began to adopt a 'workload limit' rating as part of their standard specifications. " I think the HDD designs in question are all SMR, but I'm not certain. Another reason SMR drives may have a workload rating (in addition to what I said above) is that I think they need some slack spindle time for rebalancing stuff behind the scenes. They don't need it at any particular instant, but it needs to happen sometime, thus the talk of workload over long timescales rather than being limited to x MB/s.

[scottlamb]

fwiw, I've been running Moonfire NVR on a WD Purple for five years, a WD Green for five years, and a WD Purple for two years, without problems. I also tried it on a couple Seagate SMR drives, less successfully. In one case, I exceeded the rated workload, and the drive just couldn't keep up (writes just took so long that the rtsp connections would time out), so I gave up on it within days. In another case, I ran it for a couple 3 years, keeping it under the rated workload, but the drive failed twice (once under warranty, once not) by rapid buildup of bad sectors. I recently replaced it with a cheap refurbished WP Arsenal 14TB PMR drive. Small sample size, but the continuous recording with Moonfire NVR doesn't seem too harmful to PMR drives. It might kill SMR drives, or maybe they just suck no matter how you use them.

[scottlamb]

I might implement this while adding audio support (#34). I'll likely store a GOP of video frames in a row, then the audio that happened while they were received, and repeat. That will require building them up in a buffer and being able to continue doing live view from that in-memory buffer. That's essentially the same work I described doing for monitor mode in #120.

[scottlamb]

I'm dropping the "schema change" label for this. That part was done with schema version 7. The stream config is now a more flexible JSON object with a string mode (rather than a boolean record). We can claim the mode monitor for this feature.