marcus-j-davies / nvr-js

A simple, lightweight, but very functional NVR aimed at 24/7 recording using nodejs.
MIT License
25 stars 13 forks source link

Avoid crashing if an unlink inside purge fails. #12

Closed drsvpla closed 2 years ago

drsvpla commented 2 years ago

We were seeing an issue where sometimes a call to fs.unlinkSync was failing when nvr-js was running a purge, which caused nvr-js to crash (often losing at least part of the current video files and associated metadata). In the worst case scenario, the problem persists on disk and after a restart, it crashes again next purge on the same file, etc.

The unlink could fail for any number of reasons -- permissions changing, a flaky disk, file inadvertently moved (such as to try to archive it), etc. In any event, the root cause isn't really fixable for nvr-js, but crashing is the worst case scenario.

This code "fixes" the issue by wrapping both unlinks in (separate) try/catch blocks and ignoring any errors. This way if either errors, the other is still attempted, and the Index itself is cleaned up, and nvr-js is able to continue working. If the file is still actually present on the disk, it'll effectively be ignored until the next time nvr-js is restarted, at which point the initial sync will find it, and it will probably get another chance at being deleted (or skipped again if the problem still hasn't been resolved).

I'm not thrilled with just ignoring the errors, but again, any error at this point is not really something nvr-js can fix, and we'd prefer uptime and continuous new footage to crashing in this case.

There is another unlinkSync call elsewhere in the code, but we've never seen any errors from that one (yet?), so I opted not to touch it.

marcus-j-davies commented 2 years ago

Thanks @drsvpla I have just got back for a vacation.

This seems really good. - thanks ❤️

I have a few other projects I have responsibility for - Node RED Smart Home automation (that I have already delayed 😅 ) - but hey! I'm on was on hols right 😆 .

I'll aim to get this merged into v3 soon.

I should get back into the swing of things in next few days.

In the mean time, let me know if your first patch (https://github.com/marcus-j-davies/nvr-js/issues/11#issuecomment-1239706203) is still running well.

Again thanks ❤️