[Enhancement]: Review Storage mechanics PLUS!

[KeithHanson]

Hello!

Firstly, I would like to commend you on the work you've done here. It's precisely what I need for a very important project for my city as well.

I am creating a fork of the repository now to begin diving into the codebase and attempt to determine what may be happening on our end, so please know I plan on rolling up my sleeves and helping contribute to a solution, NOT just log the bug :P

Anyhow, please see below:

ISSUE: Many hours after starting PM2 service for NVRJS, the timeline does not show video segments on the timeline. Context: We are looking to use NVRJS for the camera systems we've built utilizing Raspberry Pi's, PoE cams + switch, and USB harddrives.

We are so very close thanks to your work here. But after testing for roughly a week, we see timeline issues.

We DO see that it is properly restarting the ffmpeg processes and the files are logging to disk.

So everything seems to be working (ffmpeg, UI), except for some reason the segments stored to disk.

Also, NVRJS runs rock solid (haven't seen it rebooting over and over or anything for days at a time).

Once I DO end up restarting NVRJS, the timeline begins working normally, though is missing files that are definitely on disk.

I'll log here if I make progress on it!

Thank you for any insight you can help with though :)

[marcus-j-davies]

Hi @KeithHanson,

It's way passed my bed time where I am (2AM) but wanted to touch base before I hit sleep.

Thanks for the kind words btw :)

I must admit, this was a DIY project, but would welcome any help in making it more accessible.

On to the issue.

My initial thoughts are the way it stores the clip metadata.

It uses SQLITE which I'm not overly a fan of - I wonder if a different storage mechanism can be used?

To stop multiple writes happening, all writes are queued (FIFO), and an interval is happening to check for anything in the queue with the Commit() function (every 10s).

it's based on setTimeout, and if a write happens to error out, it may not reach calling setTimeout again for the next 10s, meaning there is lots in queue, but the Commit function is not being executed, as it crashed before setting the next 10s - the crash may have been due to some SQLITE lock issue, or file IO error, hence why i am not overly happy with my choice in using SQLite.

Another potential is the IO buffer with the pipes between the FFMPEG processs and it's self, if that gets full, it's self will stop receiving FFMPEG output, but I'm sure I ignore stdout...

[KeithHanson]

Thank you for responding so quickly! DIY is my approach to handling a fleet of pole cameras for our municipality. We have a decentralized, edge-based, DIY approach. So far, our Real Time Crime Center analysts like the interface :) And it fits my approach (if it can run decentralized straight from the pole, that's the best outcome).

Your response above all makes sense and what I was suspecting - likely some weird failure on the HD that gets taken care of by our other services on the Pi (decrypts and re-mounts the volume).

I will make a branch littered with debug statements to try to catch this (though, as you know, we will need to wait some time for whatever failure to occur).

Have you considered ditching the SQLite DB entirely? It's almost perfect as a use case to just look at the file system and draw the timeline since I only care about showing the user what's on disk. But you may have explored this already, which might lead me down the wrong track if you've already tried it.

If not, I may take a crack at a lightweight solution. It would be great if there were one less moving part to worry about (and no SQLite DB means not having to import gaps in the data, easy recovery, and other icing-on-the-cake so to speak :)

[KeithHanson]

Also, one last thought for now :P Converting from SQLite to something like PostgreSQL or MySQL might be pretty straightforward. I'm trying to keep as few moving parts on my setups as possible, though (since there's already so many), and keep processor utilization down as much as possible, so I liked the SQLite DB at first sight.

I think I will spend some time on the error not re-queueing the timeout. I'm not the best nodejs dev but I'm sure we're not the only ones who have had this issue with critical functions not being called / queued back up.

[marcus-j-davies]

To draw clips on the timeline, one could look to use ffinfo, along with the timestamp of the file being created/last modified, to accurately place them in time, the missing piece will be to associate any event data, and associate them and store the event data somewhere (other than SQLite :))

[marcus-j-davies]

I think I will spend some time on the error

Makes perfect sense at this stage, might be a simple overlook on my part 👍

[KeithHanson]

To draw clips on the timeline, one could look to use ffinfo, along with the timestamp of the file being created/last modified, to accurately place them in time, the missing piece will be to associate any event data, and associate them and store the event data somewhere (other than SQLite :))

Yes - that is pretty much the implementation I was thinking.

Ah - true true. I forgot about the events timeline (which was one of the reasons I chose this over others - that is very interesting down the road to me, and a simple API for it makes a lot of sense).

Perhaps it might be worth exploring an option for that. Since the timeline of segments is critical for our uses, it would make sense to store event data in an optional database. If you're not using that feature (like us), then you can disable it.

If you DO want that feature, event data is likely considered as non-critical compared to the video on disk itself.

So using a more "fail-proof" mechanism ("this is what is on disk and that's all I know") makes sense to me if it does to you.

[KeithHanson]

I think I will spend some time on the error

Makes perfect sense at this stage, might be a simple overlook on my part +1

Quick research suggests the "retry" module might be the ticket to patch it quickly :) Will give it a go in a branch.

I also think I will hit this issue again pretty quickly - I'm testing this on several Pi's on actual poles right now so... one of them is bound to trip up (something is causing problems regularly on our side - which is good for you and I :D )

[KeithHanson]

Progress: I've got debug statements running on a camera system right now via this fork/branch: https://github.com/city-of-shreveport/nvr-js.git/(debug-statements)

Working on setting up a way to cause failure for the SQLite DB and test the retry module in another branch.

[KeithHanson]

Ok! I found it. Some error catching and logs went a long way. I am sorry I had to dirty up your clean code :D

But the problem, I think, is more basic than I initially thought. Since the FIFO dequeue wasn't checking for an error on the run, it just drops those segments. I added an enqueue and reset the timeout. I tested this by simply mving the .db file and moving it back.

I'll submit a pull request once I do more live testing on our setup. I've pushed my changes here and re-deployed on two test and live systems. Will report back with any findings :)

This is the deployed codebase: https://github.com/city-of-shreveport/nvr-js/tree/retry-for-segment-writes

[marcus-j-davies]

Wow! You don't waste time 😅

Would you like to test MKV?

I did take a small look at the other PR - but your environment might be a more substantial test.

• can you view live footage
• can you view recorded segments
• all in the browser.

All whilst using mkv.

swap out the mp4 extension.

Here https://github.com/marcus-j-davies/nvr-js/blob/ab9e40289bb3ce9845cd637a72eec83e58b7c41a/NVRJS.js#L449

And here https://github.com/marcus-j-davies/nvr-js/blob/ab9e40289bb3ce9845cd637a72eec83e58b7c41a/NVRJS.js#L508

[KeithHanson]

Certainly! That one is very selfishly interesting to me, ha!

One adjustment I'll need make to this issue's branch prior to that is to either revert to console.log or dig into making debug output to stdout instead of stderr - PM2 thought it was simply erroring out due to this but the logs themselves seem to indicate everything is fine.

Happy to say using 3 minute chunks on 3 connected cameras to the pi, it didn't hiccup once with the patch.

Oddly, at about 6:30am my time though (CST here), the process died in an errored state and didn't restart because of it, so I'll dig into that today.

Once I verify we are good to go with logging, I'll push the mkv patch too.

The MKV format is interesting to me because our analysts typically want to roll back the footage within a very short window (they pull footage and monitor on every 911 call).

And we've seen all kinds of things go wrong on a pole (power sag/power loss/cabling coming loose inside our box, disk space, mounted drive unmounted, etc). So, lots of opportunity for improperly closing the mp4. And MKV will allow us to view partially saved footage.

I've gotta go wear the suit and tie for a little while but once I can get some time to code I'll knock those two adjustments out (stdout and MKV testing).

[marcus-j-davies]

Ok - you have twisted my arm 😅 I need a break from other OSS projects

I have created a branch (2.1.0) - I will use this to merge any enhancements being worked on - so any PR's, address them to this branch for the time being.

I think we can use the video file(s) as a way to provide the timeline content, and do away from any DB based index of segments. this way even MKV files still being written to will be in the timeline.

I'll wait to hear back from the MKV tests, before I venture down that road, but if the MKV files prove to work in the browser, this should be a smooth change - even event metadata could be made extremely simple also - a flat JSON file per event (that contains the linked segment file name) 😎

Quick Example (bf9d2a78-7425-4249-a559-39bc9ff85f6c-1660052517.json)

{
  "event": "Building Car Park Gates Opened",
  "cameraId": "52e5b562-1a1c-4ae8-88a9-c92dc94b497b",
  "linkedSegment": "yyyy-mm-dd-hh:mm.mkv",
  "sensorId": "bf9d2a78-7425-4249-a559-39bc9ff85f6c",
  "timestamp": 1660052517
}

[KeithHanson]

Awesome! I love the idea of a flat file metadata setup.

One thing I will need to is to checksum the video upon creation, for example, so as to provide proof that the file was not tampered with (if used in court in the U.S., chain of custody is a big deal when you can't prove authenticity of a file; with a checksum on creation, none of that matters much).

So that change could be interesting. Perhaps a video file + a metadata/events file along with each file on disk? IE: for any events coming in, they'd get appended to a file named the same as the video, but with a json extension.

This way, handing all details related to a clip is just two file downloads.

Re: 2.1.0 branch - will do!

[marcus-j-davies]

So that change could be interesting. Perhaps a video file + a metadata/events file along with each file on disk? IE: for any events coming in, they'd get appended to a file named the same as the video, but with a json extension.

That's the ticket!

A metadata file per segment. The metadata file will contain the checksum +basic info + any events that occurred in that segment

52e5b562-1a1c-4ae8-88a9-c92dc94b497b.1660054102.json

{
  "segment": {
    "cameraId": "52e5b562-1a1c-4ae8-88a9-c92dc94b497b",
    "fileName": "52e5b562-1a1c-4ae8-88a9-c92dc94b497b.1660054102.mkv",
    "startTime": 1660054102,
    "endTime": 1660098736,
    "checksum": "sha256:xxxxxxxxxxxxxx"
  },
  "events": [
    {
      "event": "Building Car Park Gates Opened",
      "sensorId": "bf9d2a78-7425-4249-a559-39bc9ff85f6c",
      "timestamp": 1660052517
    },
    {
      "event": "Building Car Park Gates Opened",
      "sensorId": "bf9d2a78-7425-4249-a559-39bc9ff85f6c",
      "timestamp": 1660052517
    }
  ]
}

These files then drive the timeline UI. let me know how the MKV use holds up. 👍

[KeithHanson]

PERFECT (for my use cases at least)!

I am happy to let you know that everything has been working perfectly with my branch's patch to re-enqueuing the writes to the SQLDB for over 12 hours now. I'll let it keep running for another 12 before I update that box to using MKV's.

I'm going to deploy this to another system now after making the MKV adjustment.

Question for my current branch, though - I do not submit pull requests often (ever), so I'm not all that versed on how best to handle this.

I have a pending branch that re-queues the SQLDB write. I actually need this in production atm, so am curious if you would:

1) Like me to submit a pull request for just that patch now that it's been fairly well tested in the wild? 2) Would you like another branch modifying the MP4 output to MKV from current main? Or should I spin up another branch based off of my current branch?

Thanks for the guidance!

[marcus-j-davies]

Hey @KeithHanson,

Believe it or not. - I have made massive progress, in removing the need for SQLite altogether (I also don't waste time 😄)

This is what will be coming (have written the logic for).....

• When a new video file is about to be created

  • A new metafile for it will be created (as above - give or take)
    • events created during this segment will be added to that file also.
      • at the end of the segment, its checksum is added to that metafile
        • a new video file is created + a new metafile
          • Rinse and repeat

  File name examples:

  {CamerID}.{StartUnixTimstamp}.mkv
  {CamerID}.{StartUnixTimstamp}.json

  These files will be used to drive the UI.

This should remove a truck load of surface area for problems to occur. I think at this stage

• Stick to your "remix of the logic" - for lack of a better term
  • Once you patch the files to use mkv (in place of mp4)
  • let me know of the experience
• A PR is not likely worth it at this stage

Once I am happy, you can migrate to the latest version. does that sounds like a plan?

I have not touched this project in a few weeks - so thanks for getting me excited about it again 😄

Importantly, no difference in the UI, its all backend changes. - but the UI should benefit by being able to view recorded footage that is still being written to (well on the basis MKV works 😅 )

[KeithHanson]

Absolutely! And I'm excited you're excited :)

Agreed about the PR! Sounds like my changes won't be worth much since this will now be flat-file based.

Patching in and deploying the MKV now - will let you know after a couple hours of footage :)

[marcus-j-davies]

By all means, once the new (better) enhancements are in place, feel free to rebrand it - Its OSS after all 😉

Renamed to [Enhancement]: Review Storage mechanics PLUS! 😅

[KeithHanson]

I do think making that configurable would be a great update :) But I also don't mind proudly using open-source and the branding that comes with :)

So! I just patched in MKV files, deployed them to another system, and watched two segments hit the timeline.

1. Live Streaming Works (Chrome/Firefox)

2. I can view recorded segments on the timeline (Chrome/Firefox) but can only view the video from the timeline in Chrome

3. All works in Chrome (linux), but playback of a recorded segment DOES NOT work in Firefox with the following error: Uncaught (in promise) DOMException: The media resource indicated by the src attribute or assigned media provider object was not suitable.

A quick google seems to indicate that MKV support is just not in Firefox and from what I can see, not coming :/

Thoughts? This isn't a deal breaker for us here since we use Chrome anyways.

[marcus-j-davies]

Live Streaming Works (Chrome/Firefox)

That make sense, since live streaming uses the MP4 container (well fMP4 - with trickery using websockets) One of the benefits with NVRJS - is that one FFMPEG instance caters for both the segment and live streaming.

See, I split the one input (camera) to 2 outputs [Live, File] - using one FFMPEG instance per camera,

IMO - The benefit to be had with MKV outweighs the use of Firefox, so at this point I can look past that 😅 Firefox prob doesn't work as it doesnt like MKV - but it works for Live as that is using the MP4 container, where as the segments use MKV.

I could test Fragmented MKV - but will likely be pushing my luck 😄

The live stream is passed via web sockets to a custom media file handler - the browser thinks its a single file (but really its live) it uses the MediaSource Bowser API

[marcus-j-davies]

My side of the world will be sleeping soon. update here (for my Morning), to ensure MKV is still a go ahead, and I'll continue to work on the enhancements 👍

[KeithHanson]

Ok. I can say that: it works reliably on both test machines. BUT...

For whatever reason, loading MP4 segments are significantly faster than loading MKV segments.

I have both in my timeline and recorded a test for you to see the dramatic difference. https://cityofshreveport.app.box.com/s/pjjd2y3chp8iepxloy3qn4rh8zw8hyh4 (1 minute .webm file)

Basically, MKV took about 4-5 seconds to load PER SEGMENT (with 3-minute segments - very small compared to regular operation of 15-minute segments), and MP4 took about half a second, reliably.

I have tested it during multiple periods today, and the behavior has been the same. MKV is significantly slower to load than MP4. I have no idea why, though :/

[marcus-j-davies]

Ok,

I have identified why MKV is slower.

As part of the recording chain, I move each files internal metadata to the start of the file (its usually at the end by default) MP4 allows you to move this metadata to the start (and I do so)

This allows quicker loading of the file into the Browser (by only loading a small portion to get to the metadata and start playing whilst it still downloading), and guess what, MKV does not allow this, so even though I add the flag to the ffmpeg chain, it has no affect to MKV files.

Meaning the browser has to grab the entier file (to get to the meta info at the end of the file, before it can start playing)

Turns out also, that MKV is really only supported by Chrome, I'm going to opt out of using MKV, but if one wanted to, there is a value you can change in code.

I'm almost done with using flat files - so far its working great! and the UI feels snappier with it

[KeithHanson]

Ahhh that makes sense!

I think I am with you. Snappier is better for realtime/in-the-moment lookups. I set the segments small enough that I don't think we will need to worry about that.

That is looking really good! I'm excited to begin testing it 😁

One thing I've noticed on our machines, but PM2 registers the NVRJS.js as errored after some time. When I check PM2 logs NVRJS, I don't see anything that would indicate it errored.

I'll try to dig in when I can today, but if you have some insight there as well, we'd appreciate it 😁

I can also open another issue on it to track if you'd like. But that is really the only problem I'm bumping into with my requeue patch in place while waiting for the flat file updates 😁

[marcus-j-davies]

Mmmm - I'm not sure how PM2 identifies something as errored, it might be picking up some FFMPEG debug output, and freaking out maybe.

I don't use PM2 myself these days (well... I did when I created this, hence the example start up 😅) I mainly use systemd now.

One thing to note about the upcoming version (3.0.0) Since it now uses a different storage mechanism, it won't pick up on previous recordings, as the DB has gone.

Unless you manually create the JSON files for each segment 😬 I should be able to push a test version out a bit later.

but I will also push it to the branch I created (2.1.0 - which I'll rename to 3.0.0 once uploaded)

3.0.0 also allows you to add events to the live view. i.e a live stream is being viewed, you can tag anytime an event to it - which will then appear on the timeline view.

[marcus-j-davies]

@KeithHanson

Overhaul complete. https://github.com/marcus-j-davies/nvr-js/tree/v3.0.0 https://github.com/marcus-j-davies/nvr-js/blob/v3.0.0/CHANGELOG.md

Its not published yet to NPM - but you should be able to pull down the branch and use it.

[KeithHanson]

Going to test this immediately :) THANK YOU! :)

I will keep digging into the PM2 issue. I love it due to PM2.io's interface, custom actions, metrics, etc. It's pretty nice when managing our 20+ camera systems that will eventually be 100+.

One question:

Since it now uses a different storage mechanism, it won't pick up on previous recordings, as the DB has gone. Can you tell me more here? Is it because we have to generate the metadata file to know it exists? Perhaps we could create an "import" or a "backlog" sort of script that does that work with what's on disk?

Our storage is pretty much temp storage anyways - the important parts are the live monitoring and clip retrieval in the moment. So not having it show up in the timeline for previous isn't a huge deal breaker, though I may just blow it away and start fresh.

Not a huge deal, but could become important in the future to have some sort of "Catch up" script. Most important thing is the video is on disk and files get deleted over time when space or retention dictates.

[marcus-j-davies]

Yup.

Recordings were catalogued in the DB, but since its now scrapped in favour of JSON, it won't use or load the DB. in theory, one could query the SQL file to write a bunch of JSON files - so whilst possible, just needs someone to do it 😅

Im hoping the prospect of losing previous recordings is extremely rare when updating. providing this new way is successful, I cant see it changing anytime soon. 👍

PS: Try the event system, events can now be created from live view and has the API of course 🤓

[marcus-j-davies]

One thing I should change.

The timestamp of an event being created from live view is captured, ONLY after the Prompt. so if a user takes 5 minutes to title the event - the timestamp is 5 minutes out.

easy change - Capture timestamp, then ask for event title.

https://github.com/marcus-j-davies/nvr-js/blob/a11239feb418ad94d3228c3057777d7e9630eea3/web/static/js/scripts.js#L218

[KeithHanson]

in theory, one could query the SQL file to write a bunch of JSON files - so whilst possible, just needs someone to do it

I was thinking that the UI would simply pull from the filesystem - are you saying that if there's a file with no JSON next to it, it won't load in the UI?

This is a very edge edge case, so no worries IMO. Just want to make sure I understand the intended behavior.

[marcus-j-davies]

Correct....

NVRJS now has an index of JSON files (lives in memory during runtime) - it scans the NVRJS_SYSTEM folder for the json files at start up, then updates it during the process lifetime.

such index looks like this.

Contract (timestamp = start time of segment)

{
  "cameraID": {
    "timestamp": "timestamp.json"
  }
}

Example

{
  "055d5242-b403-42fb-acfc-8b478ae5cebd": {
    "454354353534": "454354353534.json",
    "545656464566": "545656464566.json"
  },
  "055d5242-b403-42fb-acfc-8b478ae5cebd": {
    "454354353534": "454354353534.json",
    "545656464566": "545656464566.json"
  }
}

When the timeline in the UI is scrolled, it queries this index, pulling all JSON files between timeline start and timeline end that is currently in view providing the keys for the files falls within that period of time.

These JSON files, contain the meta - that is used to load clips and events in the UI.

{
  "segment": {
    "metaFileName": "1660154880.json",
    "cameraId": "66e39d21-72c4-405c-a838-05a8e8fe0742",
    "fileName": "1660154880.mp4",
    "startTime": 1660154880,
    "endTime": 1660157880,
    "checksum": "sha256:xxxxxxxxxxxxxxxxxxxxxx",
    "segmentId": "ef633a07-b720-4fd0-a15a-33ef0b8a346f"
  },
  "events": [
    {
      "eventId": "8d720aad-f7e6-49a4-824a-4b0289731a58",
      "event": "Agent 463 - Car theft Witnessed",
      "sensorId": "LIVE-VIEW-EVENT",
      "timestamp": 1660154975
    }
  ]
}

10 minutes off video, where you set a 5 minute segment time = 2 json files (each having there own mp4) each JSON file will have the unix/epoch start time and end time, and contain any events that occurred during that segment.

During segment record, endTime is set to 0, the JSON file is created the moment a new segment has been started. once the segment has ended, the endTime is set and checksum is calculated, and the file being saved again of course due to it being modified

[KeithHanson]

Brilliant :) If you don't mind, I'd like to keep this open for the next 24 hours while I have my systems testing it?

Edit: I am deploying it to a dozen or so systems now for a bigger test.

But this is fantastic. Tested the event capture too. FANTASTIC.

One tip I'd like to share - I am telling my team to generally only use the live view sparingly because the live view is the full quality stream - which will chew up bandwidth (not a huge worry, but once we enable "buddy system backups" where a pole backs up to another pole in the city, we'll be using the bandwidth at full tilt all the time).

It would be amazing if I could config the low quality stream for live view, and limit the HQ viewing to the timeline.

I also have a few other additions I'd like to make - a "maximize" button on the floating dialog boxes, a config option to always show full controls on the video embed tag (we really need this - being able to go full screen and back quickly, hopping around in the video, etc), and a config option to disable login (we have device based access over an SDWAN called ZeroTier - if someone is on that, it's because I personally clicked the button to allow them to be) - we are not concerned with logins so it would be nice to skip it.

Would you like me to make separate enhancement issues like this one for each of those "wants"?

[KeithHanson]

One more question - apologies.

When cleaning out old files, is it ONLY date based? Or does low disk space also trigger the clean as well?

[marcus-j-davies]

It would be amazing if I could config the low quality stream for live view, and limit the HQ viewing to the timeline.

Already Possible, by updating the stream config section 😎

https://github.com/marcus-j-davies/nvr-js/blob/ab9e40289bb3ce9845cd637a72eec83e58b7c41a/nvrjs.config.example.js#L48

By default it uses copy as the encoder = no transcoding (next to no CPU usage), will use what ever bitrate is offered by the incoming stream.

if you wanted to reduce it down, it needs to be transcoded (where you have control of encoding) but that will use CPU.

Example of lowering the quality for live stream (using libx264)

streamConfig: {
  an: '',
  vcodec: 'libx264',
  'b:v': '256k',
  f: 'mp4',
  movflags: '+frag_keyframe+empty_moov+default_base_moof',
  reset_timestamps: '1'
}

There are various encoders available (some hardware ones also - I think it depends on the system that it is running on) above is using a software renderer (libx264) at a bitrate of 256k

I haven't played with the options much (they are all ffmpeg options) - so tread carefully 😬 these settings are per camera

When cleaning out old files, is it ONLY date based? Or does low disk space also trigger the clean as well?

Its purely based on the number of days old the footage is continuousDays (14 days by default)

[KeithHanson]

Ah - so, to clarify, I have 1 RTSP stream coming from the camera that is low quality, and another separate RTSP stream coming in for HQ.

I'd like the live view to use one RTSP stream, and the storage/timeline view to depend on the HQ RTSP Stream? Hope that makes sense.

Its purely based on the number of days old the footage is continuousDays (14 days by default) Eek. Ok. I'm not 100% on the number of days so I'll play with it. It's another edge case, but bad things will likely happen if we hit 100% :P I'd feel much better if it took that into account.

If I stub my toe on this enough, I'll work on a small patch to keep a buffer of space available on top of days of retention.

Thank you! For everything :)

PS - how would you like me to handle my wishlist above? Enhancement issue + pull request?

[KeithHanson]

Just want to report in - so far so good. I've deployed this to several systems (~10 atm) and have recorded roughly 6 hours of footage.

Analysts are kicking the tires on it right now as well :) Several excited reactions when I demo'd creating an event from the live view :D

[marcus-j-davies]

Ah - so, to clarify, I have 1 RTSP stream coming from the camera that is low quality, and another separate RTSP stream coming in for HQ. I'd like the live view to use one RTSP stream, and the storage/timeline view to depend on the HQ RTSP Stream? Hope that makes sense.

NVRJS uses 1 FFMPEG Process per Camera, it then conjures 2 output streams from the one input. I sketched the below, this is per camera.

To pull in 2 streams from each camera, will require 2 FFMPEG instances. so having just 3 cameras, will require 6 FFMPEG processes, which could start to see performance issues.

As below, the live stream can be tailored to suite, i.e to reduce bandwidth, but it will require transcoding (i.e using libx264, instead of copy for vcodec) - nothing stopping you using a HW encoder i.e h264_v4l2m2m (if your system has one, I know most recent RPIs do) but the HW encoder h264_v4l2m2m has a bug, in that it produces a green tint, so you need to build FFMPEG from source, to get the latest build (unless its been fixed already)

moving away from copy for the live stream, means you can set a bitrate for the video (b:v), but at the cost of having to transcode (using a HW encoder will save on CPU tax)

If I stub my toe on this enough, I'll work on a small patch to keep a buffer of space available on top of days of retention.

The way it works in NVRJS is a combination of continuousPurgeIntervalHours and continuousDays Given continuousDays set to 7 and continuousPurgeIntervalHours set to 12.

it will purge any footage that is 7 days old every 12hours.

Maybe a version of this can live along side the purge schedule, with the ability to pass in the number of days, and triggered, based on a drive space poll. but the drive space poll, needs to be OS independent.

https://github.com/marcus-j-davies/nvr-js/blob/a11239feb418ad94d3228c3057777d7e9630eea3/NVRJS.js#L647

Just want to report in - so far so good. I've deployed this to several systems (~10 atm) and have recorded roughly 6 hours of footage. Analysts are kicking the tires on it right now as well :) Several excited reactions when I demo'd creating an event from the live view :D

That's encouraging to hear, the project was a hobbyist project, so nice to see it used in this way 😄

PS - how would you like me to handle my wishlist above? Enhancement issue + pull request?

I would say a PR, raising issues is fine, but may take me longer to get time to work on them 😅

[KeithHanson]

Hm. I just want to utilize the already encoded low-quality stream. But I will handle it another way (like spin up a second nvr-js to a backup drive) as I don't want to handle transcoding.

I do have an issue I've identified on two of the ten systems.

I don't know what causes it (logs look good), but I suspect it may be similar to before.

As you can see, we're doing great up to 1660218300.mp4, then the json files stop being created.

Any ideas? Assume all things can happen on the pole, because we've seen it plenty - power sag drops USB disk and things have to recover, for instance.

[KeithHanson]

As a quick fix, I'm going to hack together an "import" job that will generate the missing json files.

Other than these two systems' and this one issue, the rest have been rock solid.

[marcus-j-davies]

Mmm, nothing obvious is coming to mind - was it one camera or a whole unit? I wonder if a small IO error caused the NodeJS FileSystem Watcher to halt.

See, I use Nodes FS Watcher API to monitor the directory, for new MP4s - as that is the trigger to create a new JSON file.

[marcus-j-davies]

I could move that logic to listen for FFMPEG instead. - as I receive the segment entry name at the end of each segment. but it currently does nothing

https://github.com/marcus-j-davies/nvr-js/blob/03cb3e53857676cebdb5db0d9c5b918c3b25a8cb/NVRJS.js#L620

The above will receive the file name after each segment - so In theory, I could use it instead of monitoring for files (if IO errors can stop the watcher)

The only slight draw back - no JSON file will be available, only after the segment has finished - but during record, the JSON does not really serve a purpose, plus I need to see how I can detect the start of a new segment, if doing that 🤔

[KeithHanson]

I would much prefer attaching to ffmpeg since that is working so well. We def have IO errors for about 10 seconds sometimes. Everything recovers within about that timeframe but the json unfortunately.

Waiting until the segment is finished is pretty typical of most NVRs so.... Not a critical need (to me).

[marcus-j-davies]

Ok,

I do believe this is the FS Watcher not recovering from an IO error in that case. I'll attach it directly to FFMPEG.

I'll patch, and push the code - this will also include the new Ability to disable login (I added it today 🤫 )

There is one thing I need to fix on it - but will do when I patch the FS Watcher

https://github.com/marcus-j-davies/nvr-js/blob/03cb3e53857676cebdb5db0d9c5b918c3b25a8cb/nvrjs.config.example.js#L5

[KeithHanson]

Fantastic! I'll tune in here and redeploy everywhere once ready!

(Analysts will love disabling user/pass 😁)

[marcus-j-davies]

Ok, Have done some research....

Moving to FFMPEG to monitor the start of segments isn't straight forward currently, for a few reasons.

• There is no event to tell us (only on completetion of a segment)
• Having it based on the start up of an FFMPEG process, does not guarantee the start timestamp, is the true timestamp recording started (it can be different over several seconds - Hey I'm a perfectionist 😅 )

• Events occurring during a segment cant be written to disc (no segment meta file will exist, to write to and there is quite a lot of logic to rewrite to work around it)

  With that said, the watcher does have an error event.

  

  With this, I can restart the watcher, if it had errored due to an IO error (like a power drop say), what I will do is, wait 10s before restarting it, if I encounter this event.

  I think that will be my best approach avoiding the potential to introduce problems with the logic rewrite. and should, in theory allow the watcher to recover.

[KeithHanson]

Ah, that makes a lot of sense why that's not as straightforward as I was hoping.

Yes - that would be fine - as long as we're able to recover automatically we are happy :)

[marcus-j-davies]

Just an update - I am moving the Meta Creation Logic over to listen for FFMPEG directly after all. its WIP - so don't pull down any changes just yet.

[KeithHanson]

Roger that! Thanks for the update!

[KeithHanson]

@marcus-j-davies Any updates? :) :) :)

Current metadata bug is affecting us - I'm solving it with a cron-based restart of the service for now.

If you think it will be more than a few more days (no judgement! promise! just planning next steps), I'll go ahead and knock out the backfill task for us and have that on a periodic run & reboot service along with that every hour and I don't think we'll see any further metadata writing issues.

If I can help in any way further than that, please let me know! I don't want to step on your toes, though as I'm certain you'll produce a better outcome than I could :P

Again, many thanks for such a great tool and all the collaboration with us!

[marcus-j-davies]

Hi @KeithHanson ,

Only a couple of lines left really. I'll see if I can knock it out this evening.

Will ping you shortly