ZoneMinder / zoneminder

ZoneMinder is a free, open source Closed-circuit television software application developed for Linux which supports IP, USB and Analog cameras.
http://www.zoneminder.com/
GNU General Public License v2.0
4.9k stars 1.2k forks source link

Thoughts about the ZM API (for mobile consumption) #799

Closed pliablepixels closed 8 years ago

pliablepixels commented 9 years ago

I'm going to leave some more notes on how my (forthcoming) app can benefit more from the APIs. Not sure if this is the best way to do it - i.e. start an issue, so please advise if I need to present it any other way:

a) As I've already mentioned earlier, its critical the API ties into the same php auth ZM is using in opt_auth - an open API is unusable and insecure (I'm hoping you tell me I can fix it easily)

b) Mobile devices have memory restrictions, so http://zmdevapi/events.json can be a problem if the list is large. And this list can easily be very large. I'd recommend the following: b.1) Add ability to get length of events without fetching all of them. b.2) Add ability to specify a "from" and "to" so I can chunk get events in my code to preserve memory b.3) add ability to return events between a specified start date/time and stop date/time

c) There is no tie in with monitors and events. There should be. In addition to point b) above, I'd love to see events scoped inside monitors like so http://zmdevapi/monitors/1/events.json. All the operations of point b should work scoped inside a specific monitor too.

d) The monitor list should return a live view URL as something I can construct as an img-src - right now I am constructing it separately as img ng-src="{{loginData.url}}/cgi-bin/nph-zms?mode=jpeg&monitor={{monitor.ID}}&scale=100&maxfps=3&buffer=1000&user={{loginData.username}}&pass={{loginData.password}}&rand={{rand}}" - there is no tie in to the API

e) The detailed events list should also return an img-src url to view the event

f) In detailed events, please separate out time from date

Notes on your server side: a) As of today, the cmake install does not make the APIs active -- so if I am building an app that uses those APIs, pretty much no one can use it because even if they did do cmake, they need to add the API paths to apache configurations. I'd appreciate if you add that to your wiki. I added it via .htaccess - not sure if that is the right way (the APIs are unauthenticated - not sure if thats due to htaccess or not)

b) Consider adding Header set Access-Control-Allow-Origin "*" to your apache.conf to allow for desktop clients to be able to access your APIs

kylejohnson commented 9 years ago

Hey there. So -

a) As I've already mentioned earlier, its critical the API ties into the same php auth ZM is using in opt_auth - an open API is unusable and insecure (I'm hoping you tell me I can fix it easily)

Let us be careful with our words here. While the API does not have any authentication currently, it will. That said, it will probably, almost certainly, not be the "same php auth" that the current frontend is using.

b) Mobile devices have memory restrictions, so http://zmdevapi/events.json can be a problem if the list is large. And this list can easily be very large. I'd recommend the following:

Completely agree. In fact, this functionality is already there, but it is not documented nor intuitive. I'll need to document it - my bad.

c) There is no tie in with monitors and events. There should be.

Well, maybe there should, maybe there should not, at least in the way you're thinking (that is, a monitor recurses and finds all of its children)

You can currently find all Events which belong to a MonitorId, but you can not currently find all Events which are children of a Monitor.

Based on b) above, you can filter events by MonitorId (e.g. GET /api/events/MonitorId/3/MonitorId/4.json (I think)). Not sure if this is documented - probably not - my bad.

d) The monitor list should return a live view URL as something I can construct as an img-src

I've added that recently, or so I thought. If not, I agree. On second though, perhaps it was 'base event path' that I added.

e)

See above

f) As these fields are a single entity in the database, they're going to be a single entity in what is returned. It is the client's job to format the timestamp into a date + time, or whatever else they want. Not all clients may want this feature, so we remain neutral.

server notes

I'll need to document my experiences, but what you are experiencing is not what I was. I have, many times, reinstalled from CMAKE and then had a useable API immediately afterwards.

Part of the issue may be that you have your webroot under an alias (/zm) instead of under a virtual host (/). I access my server at http://server/, and the api is directly under that at http://server/api.

Also, I use nginx, so am hesitant to include web server configuration as part of the API. Just try hosting your site under a virtual host under /.

Hope this helps!

pliablepixels commented 9 years ago

Hello,

d) The monitor list should return a live view URL as something I can construct as an img-src

I've added that recently, or so I thought. If not, I agree. On second though, perhaps it was 'base event path' that I added.

Not from what I can see. I checked out the latest ZM trunk yesterday. Should I be using another branch? In monitors.json, you have a "Path" element, but that is really just part of the URL that one uses for receiving video streams. It differs depending on nature of protocol/type used with the camera. I am referring to the jpeg image stream that ZM normalizes everything into when you do a live view --> I am using that in my app so that irrespective of the protocol used between ZM and camera, my app can just rely on ZM providing a jpeg stream

Let us be careful with our words here. While the API does not have any authentication currently, it will. That said, it will probably, almost certainly, not be the "same php auth" that the current frontend is using.

That's fine. As long as it is authenticated that works. In retrospect you are correct - there is no reason to force the API to have the same auth as the front end as they are consumed for different reasons

b) Mobile devices have memory restrictions, so http://zmdevapi/events.json can be a problem if the list is large. And this list can easily be very large. I'd recommend the following:

Completely agree. In fact, this functionality is already there, but it is not documented nor intuitive. I'll need to document it - my bad.

Can you give me a hint on how this is structured? I can go exploring myself after that

You can currently find all Events which belong to a MonitorId, but you can not currently find all Events which are children of a Monitor.

Finding all events for a monitor is what I need. If you have it already, that's great (I think a common use -case is to navigate to a monitor and see all associated events). I am not sure I understood the difference you are making between events belonging to a monitor and children of a monitor - what do you mean by the latter?

Based on b) above, you can filter events by MonitorId (e.g. GET /api/events/MonitorId/3/MonitorId/4.json (I think)). Not sure if this is documented - probably not - my bad.

I tried http://myserver/zm/api/events/MonitorId/3.json but it returned Error: The action MonitorId is not defined in controller EventsController --> am I using it the right way?

Part of the issue may be that you have your webroot under an alias (/zm) instead of under a virtual host (/). I access my server at http://server/, and the api is directly under that at http://server/api.

Ah, ok. Possible. Let me go about exploring virtual servers then and see. Thanks.

pliablepixels commented 9 years ago

@kylejohnson I've started integrating the APIs into my app - so far so good. How do I keep letting you know additional thoughts observations? Should I keep posting them here ? BTW, would be great if you could tell me how to filter if the functionality is already there as you indicated above.

pliablepixels commented 9 years ago

@kylejohnson, Calling Elvis :-) Would help if you let me know how to navigate the APIs for drilling down as you said above - my app is sort of in 'hanging around waiting' stage.

kylejohnson commented 9 years ago

@arjunroychowdhury Sorry for the delay - I'm just getting back to work after the vacation.

Executing the following command should get you a paginated list of events for MonitorId's 7 and 25, between Sun Mar 15 2015 14:43:56 GMT-0400 (EDT) and Wed Apr 15 2015 14:43:56 GMT-0400 (EDT)

curl 'http://localhost:8081/api/events/index/MonitorId:7/MonitorId:25/StartTime >=:2015-03-15 18:43:56/EndTime <=:2015-04-15 18:43:56.json?page=1'

Sure, feel free to keep posting comments here.

pliablepixels commented 9 years ago

@kylejohnson, I don't believe this syntax is working: a) curl "http://myserver.net/zm/api/events/index/MonitorId:1.json" simply returns all events for all monitors

b) Your example curl "http://myserver.com/zm/api/events/index/MonitorId:7/MonitorId:25/StartTime >=:2015-03-15 18:43:56/EndTime <=:2015-04-15 18:43:56.json?page=1" returns a 500 with an HTML page saying The view for EventsController::index() was not found

I am running ZM 1.28.1 (checked out around 2 weeks ago, compiled from src/cmake)

pliablepixels commented 9 years ago

new API feature requests: a) ability to get version of API (that way I can protect my app from future API upgrades breaking logic) b) events.json returns an events list in ascending order (oldest first). Should it not return latest first as a design philosophy? I can always reverse it, but reversing requires two copies of the events array whereas it seems more logical to return it latest first c) The web view shows total number of events (last hour, last day, etc) along with monitors. This is very useful as a summary display. If I were to replicate that via API, I need to make one HTTP call for monitors, then grab list of events for that monitor and find its length (and parse times for equivalent functions). Given that an event count per monitor is very important, and you are already doing it in web view, consider adding this count to the monitors list

pliablepixels commented 9 years ago

@kylejohnson let me know if your examples are working for you. They don't work for me. I've posted the error above. Thx.

davidsainty commented 9 years ago

BTW I have (had for years) an Android app that monitors my ZoneMinder camera.

The first version polled for events. These days I have a separate java server that polls mysql tables directly and pushes events to the phones (as well as supporting other non-camera automation like controlling lights, doors, heaters).

On 07:40, Fri, 17/04/2015 ARC notifications@github.com wrote:

@kylejohnson https://github.com/kylejohnson let me know if your examples are working for you. They don't work for me. I've posted the error above. Thx.

— Reply to this email directly or view it on GitHub https://github.com/ZoneMinder/ZoneMinder/issues/799#issuecomment-93869021 .

pliablepixels commented 9 years ago

Thanks - I'm trying to avoid that - otherwise I'd be the only person who can use my mobile app, unless I also open source my server component which will likely not be used by anyone given ZM has its own API

kylejohnson commented 9 years ago

@arjunroychowdhury Looks like my latest code isn't in master. If you'd like to test out the 'angular-ui' branch, the code should be there. Give it a go. If my examples work for you there, I can try to cherry pick the commits into master that will give you what you need.

pliablepixels commented 9 years ago

@kylejohnson , would it work if I just merged your web/api directory from angular-ui with the master branch I have (changing Config/Database.php) or do I need to do a full checkout?

kylejohnson commented 9 years ago

That may work - give it a shot. Just back up your local repo first :)

kylejohnson commented 9 years ago

^ That said angular-ui will not be merged into master in the upstream repo any time soon.

pliablepixels commented 9 years ago

@kylejohnson - the new code uses CrudControllerTrait (the old one did not) --> I have php 5.5 and there is no file called 'CrudControllerTrait.php' anywhere on my system, nor is it part of your angular-js branch.

So even if I were to do a clean install of the angular-ui branch does not look like I would have it. The error I get when I try an API is: (I've currently replaced the angular api branch over my master api branch and updated database.php and bootstrap.php

I'm a little over my head with CakePhp - not quite sure how it works.

Error: Trait 'CrudControllerTrait' not found
File: /usr/local/share/zoneminder/www/api/app/Controller/AppController.php
Line: 34

kylejohnson commented 9 years ago

@arjunroychowdhury Aye, I started using the 'Crud' plugin to make the REST more like most people expect.

I'll need to cherry pick some commits and get things merged.

pliablepixels commented 9 years ago

@kylejohnson Yay - I got the new apis working -- seems I needed something called "composer.phrar" and then needed to " sudo php5enmod mcrypt" and then do ./composer.phar require friendsofcake/crud:~3.0 inside the app/ directory. My head hurts. Now my APIs are working. I'm going to test your additions now

kylejohnson commented 9 years ago

@arjunroychowdhury :+1:

pliablepixels commented 9 years ago

@kylejohnson -- so the filtering works :+1: But there's an odd problem, just with the events feed

When I do a "curl http://myserver/zm/api/events.json" I can see a backend apache error that says "PHP Fatal error: Call to undefined function Error() in /usr/local/share/zoneminder/www/api/app/Controller/Component/ImageComponent.php on line 64" and it returns a large amount of data with php stuff in it (including the data). The problem only exists with the events feed

Does curl with the events feed return a valid JSON object for you?

(Upon further investigation, its something to do with your image thumbnail creation -- not sure what)

pliablepixels commented 9 years ago

It looks like you have tied this into a web browser requesting the JSON object in some way. So when I use safari to query this JSON object, it works without an error, but if its curl, or my own app asking for the object, it throws an error. It seems to be related to generating thumbnail images which a browser handles but not a http client that is not a browser. Any insights?

pliablepixels commented 9 years ago

Update: I fixed the error by commenting out lines 52-68 that deals with image generation, i.e. the section dealing with if ( $overwrite || !file_exists( $thumbFile ) || !filesize( $thumbFile ) )

pliablepixels commented 9 years ago

@kylejohnson so far its great - its great that I can retrieve total event count by just getting a single page of events --> much better than getting all of them and summing up. (Please do consider my feature request to put this total event count inside monitors.json -- its such a common need)

a) The one thing I noticed is that there is no way to know via APIs whether a monitor is not working. The xml skin I think had a field called State --> the API does not . It would be useful to know if a monitor went down

b) In my app, I currently am asking the user for both the "API" path and "zm install path" as well as auth separately for both. The reason is I don't see any way to view live feed, or events playback with APIs. Is this planned?

pliablepixels commented 9 years ago

One more: I'd request you add a log layer for the API too - it would be important to know who is consuming the APIs, especially if people start deploying them today in an unauthenticated fashion (and useful in general nonetheless)

pliablepixels commented 9 years ago

@kylejohnson I am trying to implement retrieving events per monitor for the last hour. the start/end time format you specified above returns an error. Can you double check if the format is correct? (Retrieving list of events without specifying Start/Stop time works fine)

Command: curl "http://server/zm/api/events/index/MonitorId:1/StartTime >=:2015-04-24 08:00:00/EndTime <=:2015-04-24 09:00:00.json"

Error: The view for EventsController::index() was not found.

pliablepixels commented 9 years ago

@kylejohnson would you mind taking a peek at the StartTime filtering problem above?

pliablepixels commented 9 years ago

Looking at EventsController.php in apis - looks like we are limiting the number of events returned by matching it to ZM_WEB_EVENTS_PER_PAGE --> this should probably not be the case as the web needs and JSON object needs are different. I should be able to specify how many events I want via a GET parameter (including all). Looks like pagination is forced as of now - this will result in many http calls and each one is only returning 25 or so events

pliablepixels commented 9 years ago

@kylejohnson Is there a way to get the list of events starting with "latest" first, as I paginate through?

Here is the problem --> I can easily reverse a list, but combined with pagination (as you have currently implemented) its a bad UX for folks consuming the APIs

For example, I grab events for page 1 and display them. As the user keeps scrolling, I grab more pages (infinite-scrolling). Now each page lists events in ascending order (old - new). If I dynamically keep sorting, the UX is not optimal as the new events of page X+1 actually get added to the top of the list if I display it in descending order.

In other words, pagination+ ascending mode of events is a pain.

I could first do an http.get, find out # of pages and walk in reverse order of pages. A simple solution would be some means for me to retrieve the latest events via the JSON api - in other words page1 ==> latest events and so forth till pageN==>oldest events

I think most people would like to see latest events first and it makes sense the API offers a way for that

pliablepixels commented 9 years ago

After installing the Angular JS branch, I can confirm that date/time filtering works perfectly - thanks. The pending question now remains: How do I returns events in reverse order with pagination? (latest first)

pliablepixels commented 9 years ago

I am implementing PTZ right now and I noticed that https://github.com/ZoneMinder/ZoneMinder/blob/master/web/includes/control_functions.php uses various PTZ parameters stored in the monitor object to construct its url. Can you please expose various PTZ parameters (I only see isControllable)

pliablepixels commented 9 years ago
curl http://server/zm/api/monitors/daemonStatus/id:5/daemon:zmc.json

should report if zmc for that monitor is running. It returns false or true. Unfortunately, it also returns true if actual status is pending (which means something is wrong , like can't reach IP)

pliablepixels commented 8 years ago

@kylejohnson --- the events controller has a "search" URL built in - can you give me an example of how to use it?

pliablepixels commented 8 years ago

No need to keep this open I'd think.