search

homebridge / HAP-NodeJS

Node.js implementation of the HomeKit Accessory Protocol (HAP)
Apache License 2.0
2.68k stars 630 forks source link

Discussion: Current state of IP Cameras (Secure Video & custom RTP implementation) #741

Closed Supereg closed 2 years ago

Supereg commented 4 years ago

I wanted to start a discussion about the current state of IP cameras and some crazy ideas from my side 🙃

At the time of writing our implementation looks like the following

Generally the RTP implementation is a key point for a reliable implementation of IP cameras. I though of starting a project trying to create a custom RTP implementation as required by the HAP spec. Meaning adhering to RFC 3550, RFC 3551, RFC 4585, RFC 5104, multiplexing RTP and RTCP defined in RFC 5761, payload formats for H.264 (RFC 6184) and Opus voice (RFC 7587) and securing the connection according to RFC 3711 and RFC 6188. Then implementing a video/audio pipeline using gstreamer for better bitrate control. I would focus my work mainly on a Raspberry Pi and it's PiCam though keeping the API as abstract as possible.

I wanted to start this discussion to get some feedback from people with more experience, who maybe dissuade me and prevent me wasting my time lol 😅 I currently have a basic understanding of the fundamental RTP RFCs, though no concrete understanding for stuff like SRTP. I also don't know how time intensive this would be, how motivated I can stay and even wether I can manage to invest that much time into this. I just wanted to get my crazy idea out and start a conversation with people, collecting some knowledge and/or experience.

netRunner0 commented 4 years ago

@Supereg Any news on tvl8 format for routers?

Supereg commented 4 years ago

Nope. But weren't really able to do much research. I got a rough understanding but nothing more. Would be interesting to play with some existing hardware like the eeros (though seeing some of the setup videos, I'm unsure if I could extract the pairing code and intercept the communications), but didn't want to spend the money for it. Secure Video also has a higher priority right know.

adriancable commented 4 years ago

@Supereg - quick question about HSV if I may. From screenshots I have seen, it seems that HSV cameras, in addition to the uploading of MP4 fragments when events happen, also may support random access seek in addition to live view. See for example this image

The write-up you've done so far on HSV (which is excellent) seems to address the event fragment side, but not the random access part. Do we have any idea right now how in HAP to make the random access UI available for a camera, how HomeKit sends seek events to the accessory, and whether that's actually part of HSV or something that can be enabled separately for cameras independently of the event recording component?

It seems that supporting the random access part, if it's known how it works, could be easier than a practical implementation of the MP4 event fragment part. Thoughts?

Supereg commented 4 years ago

You mean random access of already recorded clips? Recorded clips are stored in iCloud and referenced with the accessory. So randomly accessing already recorded clips will just ping iCloud for the specific recording and will play it back. Nothing is save on the camera. The camera only knows two states, sending mp4 fragments or not sending them. The whole thing what clips will be dropped and what clips will be tagged as motion detected and which clips are get put together is all decided on your HomeHub and then e2e encrypted in iCloud.

adriancable commented 4 years ago

If I understand right, that's a shame!

Suppose as an example I have a camera that records continuously to some local storage. I can send a command to the camera to start playing live video, or another command to seek to some point in the past, and play the historical stream from that point.

Is there any way (using HSV or otherwise) to add the seek capability to the HomeKit camera UI that you can see in my image I posted above? To be clear, I don't want or need to upload all the video to iCloud, since I have it locally on the camera and can instruct the camera to start playing live, or from any point in the past.

Supereg commented 4 years ago

I assume your recording isn't available instantly?

You have two options:

The second one will probably be more efficient and will, depending on the camera and live streaming protocol used, produce the better video quality thus being a bit out of date regarding motion notifications (you could still expose a separate motion sensor which is not linked to the camera recording service so you can have live motion sensing maybe?). In any way data needs to be transmitted over HDS in fmp4 format.

tokamac commented 4 years ago

Are you seriously asking to use Apple HomeKit Secure Video and at the same time to use you own local storage for recorded movies? Sorry it won't be possible to have one's cake and eat it… you are searching for a local 3rd-party solution that is simply not iCloud-based HSV.

adriancable commented 4 years ago

@tokamac - actually I think I am saying the opposite. I am saying that there is a good use case for being able to seek in HomeKit cameras without having to send any video to iCloud.

The Home app for HSV cameras does have a "seek" UI and I am wondering if it is possible to enable just the seek UI without having to send any video to iCloud, which seems quite unnecessary if I want something like the following UX:

But from @Supereg it seems this isn't possible. I can't offer a simple 'seek UI' in the Home app without sending all my video to iCloud. Seems strange (but this is Apple).

adriancable commented 4 years ago

@Supereg - the recording is available instantly, in the sense that (let's assume) the camera can seek to any point in time, from now (i.e. live) to X days ago, and start streaming H264 video from that point.

tokamac commented 4 years ago

This is exactly what I understood. HSV sends the recorded videos to iCloud, period. There is no local storage. You want at the same time the HSV interface with seek functionality, but you want to seek through some local storage only, which is not a possibility proposed by Apple in the Home app. So you want another solution that is local, and not CLOUD-based HSV. Such feature (point to past movies stored in iCloud) IMHO is not handled by the accessory code but by the HSV framework underneath, controlled by Apple. I may be wrong of course, and it would be great to have a local storage feature, but I feel it doesn't work that way and it's not possible.

adriancable commented 4 years ago

@tokamac - yes, I think that's accurate. I want the seek UI in the Home app, but not HSV. I guess Apple doesn't offer the seek UI without HSV.

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity, and will be closed if no further activity occurs. If this issue was overlooked, forgotten, or should remain open for any other reason, please reply here to call attention to it and remove the stale status. Thank you for your contributions.

bdruth commented 4 years ago

I haven't seen this mentioned anywhere, so I'll ask it here. Has anyone noticed that rich notifications from HSV cameras indicate what was detected in the notification? Person, animal, person and animal, etc.? I'm looking over the existing MotionSensor service type and there's no characteristics I can see that would allow this additional detail to be provided.

image

Also, just an aside from my observations - this stuff is not bug-free on Apple's part. I only have person & animal activity enabled, and all other motion sensing disabled, and I still get generic camera has detected motion notifications - without any associated clips/thumbnails/etc. It's more flaky currently than my Wyze + camera-ffmpeg + Frigate AI person detection setup. Also - there doesn't appear to be any way on HSV cameras to configure notifications to be delivered on "this device" like there is for normal cameras - so, there doesn't appear to be any way to get notifications of any kind on from the HSV on macOS Home, for example.

tokamac commented 4 years ago

@bdruth Do you have iOS 13 or iOS 14 beta on your iPhone? I found that if notifications are enabled for the Eufy app also, and that it is configured differently than the settings in the Home app, then it will result in conflicting informations. So pay attention to disable notifications for the Eufy app (at the system-level or in the app itself) or configure the settings exactly like HomeKit for every single detail.

adriancable commented 4 years ago

@bdruth - I don't know, but HomeKit Secure Video fundamentally, architecturally, doesn't make any sense to me.

Why is it, if I have a local camera, which stores video locally, and I want to watch live video, locally, I can do that just fine in the Home app? But if I want to watch video from yesterday (which the camera has available locally, and can send to me locally just like it can send me live video locally) the only way HomeKit lets me do that is if I upload absolutely everything to iCloud, and then the Home app pulls it all back down again?

Makes no sense at all.

bdruth commented 4 years ago

@bdruth Do you have iOS 13 or iOS 14 beta on your iPhone? I found that if notifications are enabled for the Eufy app also, and that it is configured differently than the settings in the Home app, then it will result in conflicting informations. So pay attention to disable notifications for the Eufy app (at the system-level or in the app itself) or configure the settings exactly like HomeKit for every single detail.

iOS 14 beta, and yeah - I've disabled everything I can in Eufy Security so that (hopefully) only HomeKit is doing anything.

bdruth commented 4 years ago

@bdruth - I don't know, but HomeKit Secure Video fundamentally, architecturally, doesn't make any sense to me.

Why is it, if I have a local camera, which stores video locally, and I want to watch live video, locally, I can do that just fine in the Home app? But if I want to watch video from yesterday (which the camera has available locally, and can send to me locally just like it can send me live video locally) the only way HomeKit lets me do that is if I upload absolutely everything to iCloud, and then the Home app pulls it all back down again?

Makes no sense at all.

Yeah, I'm definitely scratching my head on that somewhat, too. My uneducated guess is it has something to do with security and further isolating the device from anything upstream (manufacturer's cloud, etc.). Basically - watching live video locally is just a video stream, no video is or needs to be stored, on the camera, hub, or iDevice. I think the notion of "locally stored video" is basically non-existant in HomeKit, right? Whether it's from yesterday or from 5 minutes ago, doesn't matter - the only "storage" of video is in iCloud. If it supported accessing locally stored video, it would have to go well above and beyond the confines of what it supports today for the live stream, and then it would have to secure that mechanism, and what about security for the data at rest? The HSV in iCloud is e2e encrypted, right? So, then you need to manage that wherever the video is stored locally, which for something like a NAS or similar device would be no problem, but for a Wyze cam with limited resources - might be more of a challenge. This is all just conjecture, obviously :)

adriancable commented 4 years ago

@bdruth - the notion of "locally stored video" is non-existent in HomeKit - but HomeKit (with or without HSV) could easily add a notion of 'seeking' which it doesn't have. So 'seek' to now gives live video. 'Seek' to 5 days ago gives past video. HomeKit would just need to issue the request ... the device itself (or something upstream) could then decide how to best service that. My surprise was that this isn't there, with or without HSV. It means that cameras that do 24/7 recording cannot expose this through HomeKit. Even with HSV, you can only 'seek back' to specific events (motion, person, etc.). You can't rewind to an arbitrary point in time in the past, even if the camera itself supports doing this. This, to me, is an amazing gap which really limits its utility.

(incorrect statement on HSV E2E removed)

Sunoo commented 4 years ago

@adriancable If I remember correctly that analysis is handled on your personal iOS devices, not Apple’s servers.

adriancable commented 4 years ago

@Sunoo - it turns out you are absolutely correct. From Apple's web site:

Data related to your home is encrypted and stored in the iCloud Keychain of your device. It’s also encrypted in transit between your Apple device and the devices you’re controlling in your home, even when you control your accessories from a remote location. Recordings from security cameras that use HomeKit Secure Video are analyzed privately on your Apple devices at home and then sent securely to iCloud through end-to-end encryption.

bdruth commented 4 years ago

@adriancable - reading through the above, aside from HSV, is it possible to do video previews in rich notifications today? If there was a mechanism to send the fmp4's? Or is anything other than a still image only a feature for HSV?

adriancable commented 4 years ago

@bdruth - I believe it isn't possible, but I'm not 100% sure. The way the 'rich notifications' work, is that they include the last snapshot requested by HomeKit in the notification. So, what I have tried is supplying an animated GIF (instead of a JPG) to handleSnapshotRequest, but this unfortunately does not work (the notification ends up without any image at all). I haven't tried supplying an MP4.

So, there may be another way, but I'm not aware of it.

bdruth commented 4 years ago

@adriancable - would Charles Proxy let me see the request that's going to the Eufy for that? Curious what mime-type it's returning.

adriancable commented 4 years ago

@bdruth - it's worth a try! Are you saying that the Eufy shows videos inside rich notifications for things, without HSV? (Sorry, not familiar with Eufy.)

bdruth commented 4 years ago

@adriancable Yep, exactly.

bdruth commented 4 years ago

@adriancable sorry, I responded too fast. Eufy is an HSV camera, so ... yes, it does that, but not sure if it's because of HSV.

Brandawg93 commented 4 years ago

So, what I have tried is supplying an animated GIF (instead of a JPG) to handleSnapshotRequest, but this unfortunately does not work (the notification ends up without any image at all). I haven't tried supplying an MP4.

From the HAP specification from Apple's website:

The IP camera must set the MIMEType in the HTTP response to image and must provide the snapshot in JPEG format.

The rich notifications that Eufy is using is most likely an HSV feature. That means that Eufy has nothing to do with it. The recording is analyzed on Apple's servers and sent back to the client via push notification. I'm guessing that any camera that implements HSV in the future will need to remove motion accessories and let Apple do all the work. That is Apple's intent on HSV.

Supereg commented 4 years ago

The recording is analyzed on Apple's servers

This is wrong. The recording is analyzed on your Home Hub inside your home network. It is stored e2e encrypted in iCloud. I assume a client device will just pull the latest recording from iCloud and show it in the notification view.

I'm guessing that any camera that implements HSV in the future will need to remove motion accessories and let Apple do all the work.

Not sure what you wanna say with that, but HSV cameras require a Motion sensor to signal an "recording event".

Supereg commented 4 years ago

That’s what I said(?). HSV recording is what gets displayed as „video notification“.

Brandawg93 commented 4 years ago

Ok. I see what you're saying. For homebridge-nest-cam, in order to use HSV features such as rich notifications, I would have to implement HSV into the plugin, but this would mean that the client's hub would be what is analyzing video and sending notifications, and I would no longer be able to let Google decide motion events. Does that sound correct?

Supereg commented 4 years ago

Well yes. HSV protocol would need to be implemented.

I would no longer be able to let Google decide motion events

Well, the accessory still decides when motion occurs. Video analysis (on HomePod or Apple TV) is only done when the accessory reports motion (or a doorbell press; depending what is configured as motion trigger). A more or less complete flow of events can be seen here.

Brandawg93 commented 4 years ago

A more or less complete flow of events can be seen here.

I didn't know that resource existed. Thanks for the info. It sounds like Google will still trigger the original motion event, but it would be up to HSV to determine if it were motion, person, or sound.

bdruth commented 4 years ago

This isn't entirely correct, I don't think. On the Eufy, all motion detection is disabled and HomeKit is doing all of it. The set of things HSV can detect != the set of things Eufy can natively detect.

My understanding us that HomeKit is using ML models running on the hub (which is why the camera is always streaming/ recording) - which to me means the camera isn't contributing anything but a video feed. I did test on the Eufy, the rich video notifications only seem to happen if you're in recording mode, if you're in streaming mode, you get a snapshot, but that's it. Beyond that, you get no activity notifications in any other mode.

image

tokamac commented 4 years ago

Is this integrated motion sensor exposed by the camera in HomeKit an actual physical motion sensor (separate from the lens), or is it a picture analysis done in software by the camera from the captured feed (frame difference comparison)?

bdruth commented 4 years ago

@tokamac I expect there are probably differences between cameras, but generally it's picture analysis.

Brandawg93 commented 4 years ago

@bdruth So you are saying that motion would be triggered from picture analysis and not a motion sensor accessory on the camera? If that is the case then my previous statement would be incorrect...

It sounds like Google will still trigger the original motion event, but it would be up to HSV to determine if it were motion, person, or sound.

...and my original statement would be valid?

I'm guessing that any camera that implements HSV in the future will need to remove motion accessories and let Apple your hub do all the work.

Supereg commented 4 years ago

Motion is signaled via the Motion Sensor Service included by every HSV camera. When setting motion detected characteristic to true, the camera signals "motion detected" and false for "no motion detected".

How exactly the camera does motion analysis is not part of the HAP specification (afaik). Pretty much all cameras I know do a picture analysis on the camera itself. I assume one could also include a hardware motion sensor, though I think doing picture analysis give the best results. But you can also set the motion detected characteristic based on any other motion source.

The decision if motion is detected is NOT done by the Home Hub or Apple in any way. It's always the accessories responsibility to report about motion.

bdruth commented 4 years ago

@Brandawg93 basically what @Supereg said, but I'm not sure the last bit is correct. The camera has a motion accessory, it's likely not a physical motion sensor, but I'm nearly positive that HSV is doing activity detection using CoreML running on the hub. That's why iOS 14 is able to add activity zones where iOS 13 didn't have them, IMO.

Supereg commented 4 years ago

but I'm nearly positive that HSV is doing activity detection using CoreML running on the hub.

iOS 14 activity zones do not change anything in parts of the HSV protocol. It's identical to the flow of events described in my specification writeup.

  1. Camera will signal motion detected
  2. Camera will send all video recordings which happened while motion was detected

Activity Zones only change how the Home Hub handles those recordings and only flags them for activity if the detected motion happened inside the activity zone.

bdruth commented 4 years ago

@Supereg - I was just reading your specification writeup - I think I get what you're saying now ... maybe. Basically, the camera has to do a basic, very basic "motion detected" - this can be anything. When it signals this, then the Home hub will grab the fmp4's and run them through it's own ML model (I'm guessing) to determine what kind of motion it was. That is to say - the camera itself doesn't need to support the activities that Home can detect, just the ability to let Home know that motion was detected.

Is that right?

Supereg commented 4 years ago

@bdruth yes pretty much

n0rt0nthec4t commented 3 years ago

I'm wondering if its possible to return an animated GIF/MJPEG/Short Mp4 in the buffer as part of the handleSnapshotRequest???? Over a still image

Supereg commented 3 years ago

Nope. Must have jpg format. Unsure if it also accepts some other "still" formats, but definitely no animation or video feeds.

n0rt0nthec4t commented 3 years ago

Nope. Must have jpg format. Unsure if it also accepts some other "still" formats, but definitely no animation or video feeds.

I've gotten PNG files to display, atleast in iOS/iPad 14 Betas

tokamac commented 3 years ago

I've gotten PNG files to display, atleast in iOS/iPad 14 Betas

Then, have you tried animated PNG?

hendrik42 commented 3 years ago

I'd love to be able to use HomeKit Secure Video with Homebridge cameras. What's missing and what's the next step to get there?

hjdhjd commented 3 years ago

My understanding from talking to folks earlier in the year is that basic motion detection is done on-camera. Whether it’s worthy of alerting / saving is figured out by HKSV. HKSV serves two purposes:

  1. Complex objection detection and alerting.
  2. Handles the details of storage in iCloud for the stream in question.

The question that’s unclear to me then / now...so are we talking about two motion detection events / alerts ultimately? One from the camera itself, and another by HKSV when it’s something to pay attention to rather than the wind blowing a tree?

Edit: answered below. :smile:

hjdhjd commented 3 years ago

When a motion event occurs, you'll receive a notification from the Home app that includes the type of activity seen, such as a "The Front Door Camera has detected a person", along with a thumbnail image. Tapping on the notification will take you directly to your camera's live feed, and you'll also be presented with toggles for devices that are in the same "room" as your camera, such as lights that you can turn on with just a tap.

Notification controls are available as well, giving you the ability to filter out nuisance alerts, such as a dog running by your camera. This also applies to recording video, as you can set the Home app to only record people, or cars if you wish. You can also control who has access to your cameras, and when and what others can see.

https://www.imore.com/homekit-secure-video-everything-you-need-know

More detail.

hjdhjd commented 3 years ago

And some pics of the UI for how to control notifications:

It seems you can select to either get a notification when any motion is detected (aka what we have today), or when HKSV thinks there’s something worth looking at.

n0rt0nthec4t commented 3 years ago

I notice that the latest firmware update to Starling home hub notes that you can now set activity zones on Nest cameras from Home app.. wondering how they might have implemented that or would they needed to get HSV going first?