schema version 7

[scottlamb]

Some easy stuff I definitely want to include:

• [x] flexible per-user UI settings (JSON-encoded string), as requested by @clydebarrow in https://github.com/scottlamb/moonfire-nvr/issues/153#issuecomment-907699710
• [x] flexible per-camera and per-stream settings (probably also a JSON-encoded string). Example things I want to include here: RTSP library to use (rather than a global flag), RTSP transport (TCP or UDP), frame timeout, audio mode (right now probably just none or use as a pacer for more accurate timestamps), force camera aspect ratio (rather than the hardcoded anamorphic sub stream sample aspect ratio overrides I have now), retry interval, any other special flags for per-camera workarounds, stream-but-don't-record mode for #120.
• [x] flexible settings for signals and signal types.
• [x] support a third per-camera stream. (I think "ext" is the common name?)
• [x] "drop reason" for recording runs (rather than just putting this in the logfile)
• label column in user_session, so they can give the name of the device or client agent or whatever Oh, I already added a description and forgot about it.
• [ ] better info for fixing timestamps (#9): in open boot id and monotonic time; likely a separate table with time steps observed while running. punted the latter.

Stretch goals (as in things I want but am unsure I want to hold version 7 up for their design and implementation), all punted:

• [ ] audio support (#34)
• [ ] "events" table, for analytics that give more detail than "signals". eg tags, thumbnails/key frames.
• [ ] object detection tracks
• [ ] checkpoint for analytics agent
• [ ] perfect per-frame sync between main and sub streams

My typical procedure is to start a branch, add everything in over several commits and tweak the upgrade script, and give it some more testing than usual. I want it to be pretty reliable given that the upgrade procedure can be intimidating for users and there's no downgrade procedure.

[clydebarrow]

Scott, I built the server from the new-schema branch and installed it. It would not run until I did nvr upgrade but then I was getting no data from ws streams (same in my UI and yours, recorded streams played ok.) Replaced the docker image with the distribution version but it fails because the database schema is too new, and nvr upgrade won't downgrade :-( I did not take a backup of the db. Do I have to start from scratch?

[clydebarrow]

Hmm, starting from scratch isn't so easy. I moved the db, did nvr init, now in nvr config trying to add a sample directory it claims ENOENT

Even though that dir does exist:

root@nvr:~# ls -ld /ssd/nvr/sample/
drwx------ 2 moonfire-nvr moonfire-nvr 7569408 Sep 28 10:01 /ssd/nvr/sample/

Same result with any other path on that drive.

[scottlamb]

Shoot. I wasn't ready for anyone to upgrade their main installation to schema v7. I haven't fully tested it or even finalized the schema, and as you've noticed there's no downgrade code. Starting from scratch is probably best.

Hmm, starting from scratch isn't so easy. I moved the db, did nvr init, now in nvr config trying to add a sample directory it claims ENOENT

I could give that error message better context, but it was probably on the mkdir syscall. The manpage says:

   ENOENT A directory component in pathname does not exist or is a dangling symbolic link.

so possibly /ssd/nvr doesn't exist inside the docker container, even though it does on the host. Is there a --mount for that in your wrapper script?

[clydebarrow]

Is there a --mount for that in your wrapper script?

Well, yes and no :-) I had added a mount line, but it was using a symbolic link that I must have created for reasons that I now have absolutely no recollection of. Since I'd lost the database I didn't realise I'd used the symbolic link instead of the actual mount point for the drive.

In my defence, I did re-read the installation instructions, but overlooked the note about the mount point. I assumed of course that since my script was already working it needed no changes :-(

Anyway, this has prompted me to reconfigure the hardware to swap out the SSD for a bigger high-endurance disk I'd since acquired.

I'm sure that nobody else will be stupid enough to make the same mistake - but just in case - perhaps the config ui could list the mount points available. From inside the container it would seem you can list these with mount |grep \^/dev | grep -v /etc - or maybe there is a cleaner way.

[clydebarrow]

Oh, and I wanted to use the new user preferences schema for testing the UI. I guess I need a disposable test server for that.

[scottlamb]

I'm sure that nobody else will be stupid enough to make the same mistake

Ha! No, very easy mistake to make, and I'm sure you won't be the last if we don't address it in some way.

• but just in case - perhaps the config ui could list the mount points available. From inside the container it would seem you can list these with mount |grep \^/dev | grep -v /etc - or maybe there is a cleaner way.

That's a good idea, but I think I can do even better by #160. Then there'd be no Docker mapping to worry about at all.

[edit: although, come to think of it, displaying the mount points in the UI is probably a good idea even without Docker's extra mapping layer involved...]

[scottlamb]

Oh, and I wanted to use the new user preferences schema for testing the UI. I guess I need a disposable test server for that.

That's what I do when messing with schema changes. I have a few installations:

• my main one that's exposed to the Internet running on an Intel NUC that talks to all my cameras with nice hard drives attached. It tends to run fairly close to HEAD.
• a backup on a Raspberry Pi 4 that also talks to all the cameras and just has one older smaller hard drive. It tends to run the latest released version or even older.
• a regularly-deleted installation on my workstation that just talks to some test camera or something as needed for whatever I'm debugging and only stores 1G or so of data. This is of course where I run code before it's committed. I do weird upgrades by hand and get it into strange states all the time.

[clydebarrow]

Ok, all back up and running with new drive. This install is literally built on a breadboard.