search

openhab / openhab-addons

Add-ons for openHAB
https://www.openhab.org/
Eclipse Public License 2.0
1.9k stars 3.6k forks source link

[ipcamera] Add channel to show adv event and object detection data #11391

Closed jimtng closed 2 years ago

jimtng commented 3 years ago

@Skinah Continuing on #9129, it would be useful to get the raw data provided by the camera on smart motion detection events. I have Dahua cameras with SMD (Smart motion detection) and IVS (Intelligent Video Surveillance), Facial detection, etc.

Currently the ipcamera binding can receive events from SMD and IVS, but it only signals the start and end of such events. Dahua sends more details in the data field which is currently discarded / not used by the binding.

I propose adding a special channel called lastRawData that captures this extra data field. Dahua sends such data in a JSON format, but other brands might use a different format. It would be up to the rule writer to parse the json string and interpret this data.

Some examples of data sent by Dahua, retrieved using:

$ curl -s  --digest --user 'xxx:xxx' 'http://x.x.x.x:80/cgi-bin/eventManager.cgi?action=attach&codes=[All]'
Code=SmartMotionHuman;action=Start;index=0;data={
   "RegionName" : [ "Region1" ],
   "WindowId" : [ 0 ],
   "object" : [
      {
         "HumamID" : 542,
         "Rect" : [ 2072, 1808, 3440, 8184 ]
      }
   ]
}

Code=RtspSessionDisconnect;action=Start;index=0;data={
   "Device" : "192.168.x.x",
   "StreamType" : "Main",
   "UserAgent" : ""
}

Code=VideoMotion;action=Start;index=0;data={
   "Id" : [ 0 ],
   "RegionName" : [ "Region1" ],
   "SmartMotionEnable" : true
}

Code=CrossLineDetection;action=Start;index=0;data={
   "Class" : "Normal",
   "CountInGroup" : 1,
   "DetectLine" : [
      [ 2694, 2301 ],
      [ 8009, 6056 ]
   ],
   "Direction" : "RightToLeft",
   "EventID" : 10025,
   "GroupID" : 0,
   "Name" : "IVS-2",
   "Object" : {
      "Action" : "Appear",
      "BelongID" : 0,
      "BoundingBox" : [ 4592, 1280, 5552, 5648 ],
      "Center" : [ 5072, 3464 ],
      "Confidence" : 0,
      "ObjectID" : 542,
      "ObjectType" : "Unknown",
      "RelativeID" : 0,
      "Speed" : 0
   },
   "PTS" : 42955211710.0,
   "RuleID" : 5,
   "Track" : [],
   "UTC" : 1634296535,
   "UTCMS" : 557
}
Code=CrossRegionDetection;action=Start;index=0;data={
   "Action" : "Cross",
   "Class" : "Normal",
   "CountInGroup" : 1,
   "DetectRegion" : [
      [ 218, 7837 ],
      [ 8045, 8064 ],
      [ 8045, 5182 ],
      [ 2530, 2074 ],
      [ 2239, 99 ],
      [ 146, 196 ]
   ],
   "Direction" : "Enter",
   "EventID" : 10005,
   "GroupID" : 4,
   "Name" : "IVS-1",
   "Object" : {
      "Action" : "Appear",
      "BelongID" : 0,
      "BoundingBox" : [ 4312, 1320, 5256, 5736 ],
      "Center" : [ 4784, 3528 ],
      "Confidence" : 0,
      "LowerBodyColor" : [ 0, 0, 0, 0 ],
      "MainColor" : [ 0, 0, 0, 0 ],
      "ObjectID" : 270,
      "ObjectType" : "Human",
      "RelativeID" : 0,
      "Speed" : 0,
      "humanTripLineDirection" : 0
   },
   "PTS" : 42954293150.0,
   "RuleID" : 4,
   "Track" : [],
   "UTC" : 1634295616,
   "UTCMS" : 998
}

In the first example data above, the channel should extract the data field so it would contain

{
   "RegionName" : [ "Region1" ],
   "WindowId" : [ 0 ],
   "object" : [
      {
         "HumamID" : 542,
         "Rect" : [ 2072, 1808, 3440, 8184 ]
      }
   ]
}
Skinah commented 3 years ago

Thanks for the examples, I’ll take a look at adding it once I tidy up some changes I’m in the middle of.

Skinah commented 3 years ago

OK the changes are merged that I was working on, so happy to move forward on this... The main problem I see is the name of the channel and also the label needs to be descriptive enough. The label needs to be 2-3 words long and be clear enough that most people can understand what it is without looking at the docs....

Last AI Data Last Object Data Object Detection Details Last Smart Data

Needs to be non brand specific and as descriptive as it can be and I dont feel that 'Last Raw Data' is descriptive enough that someone that is interested in object detection will notice the channel and then look it up in the documentation or link it and try it out to see what it does. It is a tough one as it is not just for objects as it can be for 'RtspSessionDisconnect' events but since it is mostly for AI and object detection uses, it is probably best to name it in this direction to grab peoples attention.

Thoughts?

jimtng commented 3 years ago

I'm open to a better name. However, I'd say Last AI Data or Object Detection Details isn't quite accurate since it could also hold non AI related event such as what you mentioned 'RtspSessionDisconnect'.

Skinah commented 3 years ago

Maybe ‘lastEventDetails’ or lastEventData? Would a channel called that make you want to know more about it? Since they are all events perhaps this makes the most sense?

jimtng commented 3 years ago

lastEventData sounds good to me

Skinah commented 3 years ago

Ok, have a jar for you to try here called jsonbeta...

http://pcmus.com/openhab/IpCameraBinding/

You will need to delete and re-add the thing to be able to see the new channel.

jimtng commented 3 years ago

I've just tested it. I realised, that looking at the bare json data is a bit confusing, because I don't know exactly which event it belongs to and what it really means. What do you think if it would simply contain the whole thing, i.e. include the Code=..... all the way to data= as well as the data itself. 

Code=VideoMotion;action=Start;index=0;data={
   "Id" : [ 0 ],
   "RegionName" : [ "Region1" ],
   "SmartMotionEnable" : true
}
Skinah commented 3 years ago

It depends on what makes using the data the easiest in rules. My opinion is the heavy lifting or parsing the json out to its pure json form from a mixed string is best done in the binding. We have to think about those that use graphical ways to write rules and may not have the ability to parse it out easily as someone that uses Java/JS/Jython/ etc.. based rules. Looking for feedback on this, as it is not an area I know very well.

Some untested code to give the idea of how to use it....

rule "Backyard Line Crossed"
    when
    Item BackyardCam_LineCrossingAlarm changed to ON
    then
    var json = BackyardCam_LastEventData.state.toString
        if(json.contains("RightToLeft")){           
            logInfo("Alarm", "Line 1 Crossed")
        // the line is crossed in 1 direction from right to left. We know this person is walking towards the house
        }
end

Basically the event code gets split into a different channel and the data is sent to its own channel. Does this prove to be easier to use? The above would still work if it was in a mixed string and not pure json. So does keeping it as pure json give advantages or should it be mixed to make it easier to follow in logs?

openhab-bot commented 3 years ago

This issue has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/dahua-and-amcrest-adding-advanced-event-and-object-detection-abilities-to-the-binding/128089/1

jimtng commented 3 years ago

Combining lastMotionType and lastEventData like in your example works well. Thanks!

Skinah commented 3 years ago

Ok I will wait for others to comment whilst I look at hikvision and what they may need. Theirs will be in xml format most likely so making sure it is consistent across brands before I complete and make a PR to be merged.

jimtng commented 2 years ago

I hope this makes it into 3.2M5

Skinah commented 2 years ago

The next release will be 3.2 Stable and this will not make it in there in time. In fact I went to do some more work on this for Hikvision and discovered I had deleted the branch by mistake that held the changes so I'm back to square one on this so I need to find a day to redo the work over again and test with Hikvision so both brands can use the same channel. It is a busy time of year.

Skinah commented 2 years ago

Did some testing on Hikvision and the results are below. The 2 is which line number was crossed and triggered an alarm, so this data people will want and can be added to the same channels that Dahua can use.

--boundary
Content-Type: application/xml; charset="UTF-8"
Content-Length: 1253

<EventNotificationAlert version="2.0" xmlns="http://www.hikvision.com/ver20/XMLSchema">
<ipAddress>XXXXXXXXXX</ipAddress>
<portNo>80</portNo>
<protocol>HTTP</protocol>
<macAddress>XXXXXXXXXXX</macAddress>
<channelID>1</channelID>
<dateTime>2021-12-05T11:46:27+10:00</dateTime>
<activePostCount>93</activePostCount>
<eventType>linedetection</eventType>
<eventState>active</eventState>
<eventDescription>linedetection alarm</eventDescription>
<DetectionRegionList>
<DetectionRegionEntry>
<regionID>2</regionID>
<sensitivityLevel>40</sensitivityLevel>
<RegionCoordinatesList>
<RegionCoordinates>
<positionX>5</positionX>
<positionY>644</positionY>
</RegionCoordinates>
<RegionCoordinates>
<positionX>188</positionX>
<positionY>514</positionY>
</RegionCoordinates>
</RegionCoordinatesList>
</DetectionRegionEntry>
</DetectionRegionList>
<channelName></channelName>
<Extensions version="1.0" xmlns="urn:psialliance-org">
<serialNumber xmlns="urn:selfextension:psiaext-ver10-xsd">DX-2XDXXXXFWD-IXXXXXXC15XXXX71</serialNumber>
<eventPush xmlns="urn:selfextension:psiaext-ver10-xsd">IO&amp;&amp;DX-XXXXXXXFWD-IXXXXXXXXXX671,2021-12-05T11:46:27+10:00,1.0</eventPush>
</Extensions>
</EventNotificationAlert>
openhab-bot commented 2 years ago

This issue has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/hikvision-unlock-extra-line-crossing-alarms-in-your-camera-for-free/129625/2

Skinah commented 2 years ago

Ok I need some people to test it for both Dahua and Hikvision as both are now implemented in the jar that is found here http://pcmus.com/openhab/IpCameraBinding/ Source code is in my github branch/fork

jimtng commented 2 years ago

I've just tried it on 3.2M5. Got this error

2021-12-09 22:16:34.068 [ERROR] [mera.internal.IpCameraHandlerFactory] - bundle org.openhab.binding.ipcamera:3.2.0.202112091012 (293)[org.openhab.binding.ipcamera.internal.IpCameraHandlerFactory(338)] : Error during instantiation of the implementation object
java.lang.IllegalArgumentException: argument type mismatch
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
        at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:490) ~[?:?]
        at org.apache.felix.scr.impl.inject.internal.ComponentConstructorImpl.newInstance(ComponentConstructorImpl.java:316) ~[bundleFile:?]
        at org.apache.felix.scr.impl.manager.SingleComponentManager.createImplementationObject(SingleComponentManager.java:286) [bundleFile:?]
        at org.apache.felix.scr.impl.manager.SingleComponentManager.createComponent(SingleComponentManager.java:115) [bundleFile:?]
        at org.apache.felix.scr.impl.manager.SingleComponentManager.getService(SingleComponentManager.java:1000) [bundleFile:?]
        at org.apache.felix.scr.impl.manager.SingleComponentManager.getServiceInternal(SingleComponentManager.java:973) [bundleFile:?]
        at org.apache.felix.scr.impl.manager.SingleComponentManager.getService(SingleComponentManager.java:918) [bundleFile:?]
        at org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse$1.run(ServiceFactoryUse.java:216) [org.eclipse.osgi-3.16.300.jar:?]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
        at org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse.factoryGetService(ServiceFactoryUse.java:213) [org.eclipse.osgi-3.16.300.jar:?]
        at org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse.getService(ServiceFactoryUse.java:114) [org.eclipse.osgi-3.16.300.jar:?]
        at org.eclipse.osgi.internal.serviceregistry.ServiceConsumer$2.getService(ServiceConsumer.java:48) [org.eclipse.osgi-3.16.300.jar:?]
        at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.getService(ServiceRegistrationImpl.java:547) [org.eclipse.osgi-3.16.300.jar:?]
        at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.getService(ServiceRegistry.java:533) [org.eclipse.osgi-3.16.300.jar:?]
        at org.eclipse.osgi.internal.framework.BundleContextImpl.getService(BundleContextImpl.java:656) [org.eclipse.osgi-3.16.300.jar:?]
        at org.apache.felix.scr.impl.manager.SingleRefPair.getServiceObject(SingleRefPair.java:88) [bundleFile:?]
        at org.apache.felix.scr.impl.inject.methods.BindMethod.getServiceObject(BindMethod.java:675) [bundleFile:?]
        at org.apache.felix.scr.impl.manager.DependencyManager.getServiceObject(DependencyManager.java:2556) [bundleFile:?]
        at org.apache.felix.scr.impl.manager.DependencyManager.doInvokeBindMethod(DependencyManager.java:2075) [bundleFile:?]
        at org.apache.felix.scr.impl.manager.DependencyManager.invokeBindMethod(DependencyManager.java:2058) [bundleFile:?]
        at org.apache.felix.scr.impl.manager.SingleComponentManager.invokeBindMethod(SingleComponentManager.java:443) [bundleFile:?]
        at org.apache.felix.scr.impl.manager.DependencyManager$MultipleDynamicCustomizer.addedService(DependencyManager.java:333) [bundleFile:?]
        at org.apache.felix.scr.impl.manager.DependencyManager$MultipleDynamicCustomizer.addedService(DependencyManager.java:301) [bundleFile:?]
        at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1200) [bundleFile:?]
        at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1121) [bundleFile:?]
        at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.trackAdding(ServiceTracker.java:928) [bundleFile:?]
        at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.track(ServiceTracker.java:864) [bundleFile:?]
        at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:1152) [bundleFile:?]
        at org.apache.felix.scr.impl.BundleComponentActivator$ListenerInfo.serviceChanged(BundleComponentActivator.java:114) [bundleFile:?]
        at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:120) [org.eclipse.osgi-3.16.300.jar:?]
        at org.eclipse.osgi.internal.framework.BundleContextImpl.dispatchEvent(BundleContextImpl.java:957) [org.eclipse.osgi-3.16.300.jar:?]
        at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:234) [org.eclipse.osgi-3.16.300.jar:?]
        at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:151) [org.eclipse.osgi-3.16.300.jar:?]
        at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:936) [org.eclipse.osgi-3.16.300.jar:?]
        at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:873) [org.eclipse.osgi-3.16.300.jar:?]
        at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:141) [org.eclipse.osgi-3.16.300.jar:?]
        at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:261) [org.eclipse.osgi-3.16.300.jar:?]
        at org.eclipse.osgi.internal.framework.BundleContextImpl.registerService(BundleContextImpl.java:496) [org.eclipse.osgi-3.16.300.jar:?]
        at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:929) [bundleFile:?]
        at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:915) [bundleFile:?]
        at org.apache.felix.scr.impl.manager.RegistrationManager.changeRegistration(RegistrationManager.java:133) [bundleFile:?]
        at org.apache.felix.scr.impl.manager.AbstractComponentManager.registerService(AbstractComponentManager.java:984) [bundleFile:?]
        at org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:752) [bundleFile:?]
        at org.apache.felix.scr.impl.manager.AbstractComponentManager.enableInternal(AbstractComponentManager.java:674) [bundleFile:?]
        at org.apache.felix.scr.impl.manager.AbstractComponentManager.enable(AbstractComponentManager.java:437) [bundleFile:?]
        at org.apache.felix.scr.impl.manager.ConfigurableComponentHolder.enableComponents(ConfigurableComponentHolder.java:667) [bundleFile:?]
        at org.apache.felix.scr.impl.BundleComponentActivator.initialEnable(BundleComponentActivator.java:305) [bundleFile:?]
        at org.apache.felix.scr.impl.Activator.loadComponents(Activator.java:554) [bundleFile:?]
        at org.apache.felix.scr.impl.Activator.access$200(Activator.java:70) [bundleFile:?]
        at org.apache.felix.scr.impl.Activator$ScrExtension.start(Activator.java:421) [bundleFile:?]
        at org.apache.felix.scr.impl.AbstractExtender.createExtension(AbstractExtender.java:196) [bundleFile:?]
        at org.apache.felix.scr.impl.AbstractExtender.modifiedBundle(AbstractExtender.java:169) [bundleFile:?]
        at org.apache.felix.scr.impl.AbstractExtender.modifiedBundle(AbstractExtender.java:49) [bundleFile:?]

...
jimtng commented 2 years ago

Another one with a different line number

22:21:32.623 [ERROR] [amera.internal.IpCameraHandlerFactory] - bundle org.openhab.binding.ipcamera:3.2.0.202111030031 (237)[org.openhab.binding.ipcamera.internal.IpCameraHandlerFactory(271)] : Error during instantiation of the implementation object
java.lang.IllegalArgumentException: argument type mismatch
    at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
    at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
    at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
    at java.lang.reflect.Constructor.newInstance(Constructor.java:490) ~[?:?]
jimtng commented 2 years ago

Dahua: the data includes ;data={ - this appears to be a parsing error. The data needs to be in json format, so it should start with { and ends with }

Skinah commented 2 years ago

I have fixed the parsing bug and new Jar is uploaded ready for testing. It is now fully tested here and works for both Dahua and Hikvision. I suspect the other errors are from you needing to delete and re-add the thing as the channels have changed due to me loosing the code and having to re-write this from scratch. They may also be from changing the jar whilst the binding was running? Let me know if they continue to occur after removing and re-adding the thing.

Example rule for hik cameras

rule "Backyard Line Crossed"
    when
    Item BackyardCam_LineCrossingAlarm changed to ON
    then
    var xml = BackyardCam_LastEventData.state.toString
        if(xml.contains("<regionID>1</regionID>")){         
            logInfo("Alarm", "Line 1 Crossed")
        }else if(xml.contains("<regionID>2</regionID>")){
            logInfo("Alarm", "Line 2 Crossed")
        }   
end
jimtng commented 2 years ago

I've always seen this error, even in the previous versions.. I'm running 3.2M5:

08:31:55.727 [WARN ] [org.apache.felix.fileinstall         ] - Error while starting bundle: file:/openhab/addons/org.openhab.binding.ipcamera-3.2.0-SNAPSHOT.jar.new-notworking
org.osgi.framework.BundleException: Could not resolve module: org.openhab.binding.ipcamera [238]
  Unresolved requirement: Import-Package: io.netty.bootstrap; version="[4.1.0,5.0.0)"

    at org.eclipse.osgi.container.Module.start(Module.java:463) ~[org.eclipse.osgi-3.16.300.jar:?]
    at org.eclipse.osgi.internal.framework.EquinoxBundle.start(EquinoxBundle.java:445) ~[org.eclipse.osgi-3.16.300.jar:?]
    at org.apache.felix.fileinstall.internal.DirectoryWatcher.startBundle(DirectoryWatcher.java:1260) [bundleFile:3.7.0]
    at org.apache.felix.fileinstall.internal.DirectoryWatcher.startBundles(DirectoryWatcher.java:1233) [bundleFile:3.7.0]
    at org.apache.felix.fileinstall.internal.DirectoryWatcher.doProcess(DirectoryWatcher.java:520) [bundleFile:3.7.0]
    at org.apache.felix.fileinstall.internal.DirectoryWatcher.process(DirectoryWatcher.java:365) [bundleFile:3.7.0]
    at org.apache.felix.fileinstall.internal.DirectoryWatcher.run(DirectoryWatcher.java:316) [bundleFile:3.7.0]
jimtng commented 2 years ago

Thanks for the update. The Dahua data is correct now.

Skinah commented 2 years ago

That warn is from a missing netty dependency. How to fix it is found in this link below, but it wont happen when you use the merged binding after the changes get added..

https://community.openhab.org/t/ipcamera-binding-breaking-changes-and-new-features-in-3-2-milestone-3-and-newer/126735

jimtng commented 2 years ago

Sometimes I came across this in the data:

{
   "Id" : [ 0 ],
   "RegionName" : [ "Region1" ],
   "SmartMotionEnable" : true
}

--myboundary
Content-Type: text/plain
Content-Length:41

Code=VideoMotionInfo;action=State;index=0

--myboundary
Content-Type: text/plain
Content-Length:126

Code=VideoMotion;action=Stop;index=0;data={
   "Id" : [ 0 ],
   "RegionName" : [ "Region1" ],
   "SmartMotionEnable" : true
}

It appears that the parser missed the boundary?

openhab-bot commented 2 years ago

This issue has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/ipcamera-new-ip-camera-binding/42771/2721

openhab-bot commented 1 year ago

This issue has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/hikvision-acusens-nvr-face-recognition-integration-for-people-anouncement-who-is-at-the-door/145775/2