JSON API for configuration

[scottlamb]

Define and implement a JSON API that can do everything moonfire-nvr config currently does, so we can replace moonfire-nvr config with a web UI (#35):

• [ ] add/remove sample file directories
• [ ] configure retention, deleting old files as necessary to fit within lower limits.
• [ ] test out RTSP URLs (at least reporting if they work and pixel dimensions)
• [ ] add/edit/remove cameras
• [x] add/edit/remove users

Also, one thing that's not even possible through moonfire-nvr config now:

• [ ] add/remove/edit signal types and signals (as mentioned in #28)

One complication: Right now the code isn't really structured to allow changing these things while running. A few examples of what I mean:

• the sample file directory syncer thread isn't really set up to dynamically change the limits mid-run.
• the code which lowers retention will lock the database as long as it feels like (maybe a minute or more)
• we don't dynamically add/remove cameras

Maybe it's acceptable as a first pass to switch between "config mode" and "running mode" based on an API request; eventually we should just allow recording and reconfiguring at once.

@clydebarrow

[clydebarrow]

Yes, I figured that there would probably be more work in reworking the internals to be able to config on the fly, rather than the API itself. My thoughts:

1. Database - perhaps make the locking finer grained - is that possible with SQLite?
2. Reconfiguring other stuff - I'd suggest having an API call that restarts the server processes, so that the reconfiguration can be done while running, but does not take effect until the UI says so - probably linked to an "Apply" button.

[scottlamb]

Database - perhaps make the locking finer grained - is that possible with SQLite?

Sort of. In SQLite's WAL mode (which we use), "readers do not block writers and a writer does not block readers". SQLite doesn't have MVCC like some databases, so we can't do two write transactions in parallel, but I think we can get away without that.

The problem is probably less SQLite and more my own code. Which is good in the sense that I can fix it. My Database maintains a bunch of in-memory indexes and currently is just behind a mutex (no concurrency). And my lower_retention wasn't written to minimize impact on other things happening at the same time.

I just refreshed my memory, and lower_retention isn't as bad as I was thinking: it does one big database transaction to remark the recordings in question as garbage (holding the lock), then releases it and actually unlinks it all from disk (the really slow part), then locks again for a second database transaction to truly forget about them. I'm not sure how long the database transactions take, but maybe we're only locking for seconds each time rather than minutes.

• If so, we can probably just accept that the system's a little janky right when you kick off a massive delete, which is hopefully rare! lower_retention would still need to be refactored to work with a running Syncer as there'd be while the system is truly running, but this refactoring might not be a huge investment.
• If not, we'll have to either break the transactions apart into smaller ones (at the cost of flash write cycles but probably acceptable once in a while) or allow Database read operations to proceed in parallel with a write transaction (more complexity than I'd like but fundamentally possible). Other writes might be less of a problem than you'd think: many things are already queued and merged at query time to reduce the number of flash cycles, so (with exception I could fix) the serving code already doesn't wait for SQLite write transactions to finish.

Reconfiguring other stuff - I'd suggest having an API call that restarts the server processes, so that the reconfiguration can be done while running, but does not take effect until the UI says so - probably linked to an "Apply" button.

Neat idea. Kind of like what openwrt does.

Server-side, I think if we did that it'd be by basically constructing a pending reconfiguration operation, querying stuff through that view, and then applying it during the restart. It's possible. I might be convincing myself that refactoring to actually make the changes immediately is easier though. Which do you think is the better user experience?

[clydebarrow]

A seamless process would be a better user experience, but the "Apply" option would be quite acceptable. I'd recommend going for the least work server-side to start with, favouring rapid release over perfection.

Bear in mind that the admin interface is going to be used much less often than other parts of the UI so while I think a step up from the current system is worthwhile, it shouldn't take precedence over other features (like recording motion events, which is a Must Have.)

[scottlamb]

A seamless process would be a better user experience, but the "Apply" option would be quite acceptable. I'd recommend going for the least work server-side to start with, favouring rapid release over perfection.

I'll try seamless first then and re-evaluate if I decide it's significantly harder.

One thing we'll want is a way to report a long-running operation like deleting a bunch of things, rather than having a POST hang for so long that the user gives up and reloads the page, and then the config UI coming back up with no idea an operation is still running or when it will complete.

I've had bad experiences with long-running HTTP/1.1 requests like SSE or Content-Type: application/x-mixed-replace: the browser hits its limit of 4 HTTP connections to the same server and the UI basically freezes. So I think it's better to have a WebSocket event feed (see also #40). Maybe you'd do something like a POST operation to start an operation with a tag (returning immediately), and then the WebSocket feed would post events for "operation x is 39% done" and such. Or do you have a preference for how this works?

If you know of a some API that does this well, I'm happy to look at it. Google has a long-running operation API for its cloud APIs, but it's based on gRPC and probably not directly translatable to something for a web browser to use.

Bear in mind that the admin interface is going to be used much less often than other parts of the UI so while I think a step up from the current system is worthwhile, it shouldn't take precedence over other features (like recording motion events, which is a Must Have.)

Hmm. I also want motion support more, but I'm talking about this now because I'm eager to take advantage of your offer of UI help in #35. Two heads are better than one, even if it means not prioritizing quite how I would on my own.

If you'd prefer to first work on UI for motion/analytics, I'd be happy to work together on that first. Or if you'd prefer to work on UI for other places where the JSON API is already adequate or nearly so, I'd be happy to work on analytics in parallel (probably mostly building a solid server side, as that's where my expertise is).

[clydebarrow]

The POST/websocket should work fine. I have done very similar in another project, though I used a Pusher channel instead of a websocket, simply because the server was hosted on GAE which doesn't support websockets. Same principle though. The websocket just delivers JSON formatted status messages with some kind of ID for the operation, percent complete, a done flag and maybe other info like phase of the operation if relevant.

I'm happy to go with whatever you prefer server-side in terms of priorities. I can certainly handle the UI side and match whatever features get implemented on the server. I have started building the UI (from scratch rather than on top of what you have done) and should have that exceeding current functionality very shortly.

[scottlamb]

I'm happy to go with whatever you prefer server-side in terms of priorities.

In that case, maybe I'll pause on this, and we can move the conversation elsewhere:

• sounds like what we both want most is good analytics support.
  • Have you looked at the existing signals schema and API (#28)? I think it's at least a start—it allows eg basic support for motion detection (driveway camera in state unknown/still/motion) and security system zone state (front door unknown/closed/open). Definitely more than the current UI shows! and I'm happy to get feedback and refine as necessary.
  • I'd like to add more sophisticated stuff like "events" (with support for say tags and designated key frames to show thumbnails of) and "object detection tracks" (to have boxes, labels, and confidences overlayed on the video as it plays). I'm very interested in design feedback here also.
• the server is already ahead of the UI in various other ways, as you've probably noticed. Eg I think the API is in place (or nearly so) for a real scrub bar-based UI (#32) that fetches small chunks of video as needed, rather than the awkward and sometimes slow list UI I have now. Many of the live view's limitations (#115) require little if any change on the server side.
• and some pieces of config are easier than others. Eg user management operations would be easy to add, as they aren't as expensive/long-running or as intertwined with as much of the server state.

I have started building the UI (from scratch rather than on top of what you have done) and should have that exceeding current functionality very shortly.

Awesome! I'm excited to hear more.

[clydebarrow]

For user preferences there will be some data that can be saved in browser localstorage, but some, e.g. date formats, should be global for a user, and so should be saved in the server. So there should be an api call to save and retrieve user preferences. This could be simply one JSON-encoded string, so that it can be extended without database schema changes.

[scottlamb]

I'll create an issue for the next schema version's changes. User preferences storage is easy to implement, so I'll include it for sure.

[scottlamb]

User preferences are now on the new-schema branch. Let me know if the API works for you. I tried to come up with something that would be extensible to other user operations, although setting the current user's preferences is the only thing it's good for now.

[scottlamb]

I'm going to start on this API with signals. I think it'd be useful to support transactions with a variety of configuration changes. Eventually, most things supported here. The exception would be the long-running things mentioned in my Aug 25 comment, eg bulk deletes.

I'm thinking of doing POST /api/ with a list of operations to perform in the transaction. Perhaps types setSignalType and setSignal for INSERT/UPDATE/DELETE on those respective objects.