HomeKit Secure Video Unofficial Specification - R3

1. Overview

Service Definitions

Cameras

HomeKit Secure-Video cameras need to expose the same services as normal cameras with the following changes:

• Every RTPStreamManagement service must add the Active characteristic (This is used to indicate that the camera is fully turned off)
• The required amount of RTPStreamManagement services was dropped from two to one.
• The MotionSensor service is required (to indicate movement and thus a start and stop of a recording)
• The CameraOperatingMode service is required
• The DataStreamManagement service is required (to initiate HomeKit Data Stream communication)
• The CameraEventRecordingManagement service is required. It needs to link to the MotionSensor and DataStreamManagement service

If MotionSensor or OccupancySensor are added, they must expose the Active characteristic.

Doorbells

HomeKit Secure-Video doorbells need to expose the same services as both doorbells and secure-video cameras.

Active states

Every Secure-Video enabled camera can be set to four different states: Off, Detect Activity, Stream and Stream & Allow Recording.
Depending on the state the following Active characteristics for the given services are set.

Recording requirements

• The camera needs so support basic motion detection
• The camera needs to support encoding, buffering and transmitting of fragmented MP4 - Pretty similar to Section 3.3 RFC 8216 (HLS). The RFC refers to ISO/IEC 14496-12 ISO base media file format.
  How exactly the data is transmitted can be read in the HDS Packet Formats section and the Flow of events section.
• Supported video codecs are: h.264, h.265 (possibly)
• Supported audio codecs are: AAC-LC, AAC-ELD
• Typical resolutions one might support (a camera might support more):
  • 640x480
  • 1024x768
  • 1280x960
  • 1600x1200
  • 2048x1536
  • 640x360
  • 1280x720 (Mandatory)
  • 1920x1080 (Mandatory, default at 24 or 30 fps)
  • 3840x2160
• Frame rates:
  • 15 fps (Mandatory)
  • 24 fps
  • 30 fps (One of 24 fps or 30 fps is mandatory)

2. Services

2.1 CameraOperatingMode

2.2 CameraEventRecordingManagement

3. Characteristics

3.1 CameraOperatingModeIndicator

This characteristic indicates if the camera LED, which shows the current state of the camera (see states), should be turned on. Controlled by "Camera status light" setting in the Home App.

3.2 EventSnapshotsActive

This characteristic indicates if the option "Camera Settings" -> Notifications -> "Allow Snapshots in Notifications" is turned on. If this option is turned on, a notification from this camera sent to anyone in this home, will include a snapshot of the motion or activity.

3.3 HomeKitCameraActive

This characteristic indicates if the camera should detect activity (Unsure if activity just means motion detection or also button presses for doorbell accessories).

3.4 ManuallyDisabled

This characteristic indicates if the camera was manually turned off, for example using a physical switch on the camera.

3.5 NightVision

This characteristic is already present in the current HAP spec This characteristic indicates if automatic night vision should be turned on.

3.6 PeriodicSnapshotsActive

Exact behaviour unclear. Seems to be always set to true regardless of anyone viewing periodic snapshots or not.

3.7 RecordingAudioActive

This characteristic indicates if recordings should include audio.

3.8 SupportedCameraRecordingConfiguration

A read request on this characteristic returns the following structure:

Supported Camera Recording Configuration:

Media Container Configuration:

Media Container Parameters:

3.9 SupportedVideoRecordingConfiguration

The value of this characteristic is a TLV8-encoded list of supported video codecs:

Supported Video Recording Configuration:

Video Codec Configuration:

Video Codec Configuration TLV contains exact one tlv of 'Video Codec Parameters' and one entry of 'Video Attributes' per supported resolution/frame rate combination.

Video Codec Parameters:

Video Codec Attributes:

3.10 SupportedAudioRecordingConfiguration

The value of this characteristic is a TLV8-encoded list of supported audio codecs:

Supported Audio Recording Configuration:

Audio Codec Configuration:

Audio Codec Parameters:

3.11 SelectedCameraRecordingConfiguration

The structure of the write value looks like the following:

Selected Camera Recording Configuration:

3.12 ThirdPartyCameraActive

Usage and behaviour of this characteristic is currently pretty unclear.

3.13 DiagonalFieldOfView

4. HomeKit Data Stream Packet Formats

4.1 Start

When the camera detects motion it will send a hap event for the characteristic as usual. After that, one of the connected Home Hubs will send an open request.

The header should use dataSend as the protocol and open as the topic.
The request has the following message fields:

The response has the following message fields:

4.2 Binary Data

The header should use dataSend as the protocol and data as the topic. The event has the following message fields:

A packet dictionary looks like the following:

Metadata for recording chunks is defined as:

4.3 Close

This event closes the stream and is sent by the home hub once the motion sensor is set back to "No motion detected" (It seems that the home hub still waits for the last mp4 segment to be sent).
The header should use dataSend as the protocol and close as the topic.

The event has the following message fields:

The accessory can also send this event message to indicate that the session errored unexpectedly and should be aborted.

5. Image Snapshots

The POST body of the POST /resource request received a new optional property reason with number type.

With the reason property a controller indicates the reason for a snapshot request:

• 0: Request is the result of a periodic snapshot request.
• 1: Request is the result of an event snapshot (e.g. to display image for a motion event).

If the accessory has PeriodicSnapshotsActive turned off, any snapshot request without a reason property or the reason property set to 0 must be rejected.

If the accessory has EventSnapshotsActive turned off, any snapshot request without a reason property or the reason property set to 1 must be rejected.

When rejecting a snapshot request the accessory must return HTTP 207 Multi-Status and a HAP Status code of -70401 (INSUFFICIENT_PRIVILEGES; if it was rejected due to the missing reason property) or -70412 (NOT_ALLOWED_IN_CURRENT_STATE, if the reason doesn't match the current set rules).

6. Flow of events

In this section I will give a brief overlook on how an activity will be recorded using secure-video.

• The secure-video camera gets paired.
• The user sets the current camera state to Stream & Allow Recording
• The configuration of the camera will be set up:
  • The accessory will receive a write request on the SelectedCameraRecordingConfiguration characteristic
    • Important settings like the prebuffer length are configured
  • The Active characteristic of the CameraEventRecordingManagement service will be set to true (and all other active characteristics getting updated according to the camera state)
• If the camera is set to detect motion it will continuously check the video stream for any movement as usual. If recording is enabled, the camera will fill the pre buffer with mp4 fragments according to the First-In-Last-Out principle.
• When the motion service reports activity, all available Home Hubs will open an HomeKit Data Stream Connection to the accessory
• If the camera detects motion (analogous for doorbell button presses) if will set the Motion Detected characteristic of the Motion Sensor to true
  • After that, a home hub will initiate a bulk send session over HDS and sends a dataSend start request with a new streamId.
  • The camera will now send a mediaInitialization dataSend data event with the below listed metadata. The mp4 data contains a ftyp and moov box.
    • dataSequenceNumber: 1
    • dataChunkSequenceNumber: 1
    • isLastDataChunk: true
  • After that the actual mp4 fragments are getting sent using mediaFragment dataSend data events. The mp4 data contains a moof and a mdata box and must start with a keyframe. The accessory will begin immediately by sending the fragments currently contained in the prebuffer (typically 2x4 seconds in length). After that the accessory will send any newly recorded mp4 fragments (typically 4s in length) when they become available (any fragment will be sent, where the recording was started while motion was still detected).
    Every mp4 fragment gets a new incrementally assigned dataSequenceNumber (starting by 2 for the first segment in the preBuffer). If the size of one mp4 fragment is too big, it can be split into multiple chunks. Then every chunk is enumerated by the dataChunkSequenceNumber, while the last chunk must always be marked with isLastDataChunk equal to true. Current cameras seems to use a maximum chunk size of 262,144 bytes (or 0x40000 bytes).
• The HomeKit Home Hub receiving the mp4 fragments will analyze every mp4 fragment for moving objects, recognize faces, and decide, according to the configured motion settings, if a given fragment will be flagged for motion or not. The Home Hub will then assemble a recording from the flagged mp4 fragments and save it in iCloud. iCloud will then tell iPhones, iPads, HomePods and AppleTV to notify the user based on their notification settings.
• When motion stops the accessory will set the Motion Detected characteristic of the Motion Sensor to false. The camera will still send out the last mp4 fragment which is currently recorded (remember: typically fixed 4s fragment length). After a short time the Home Hub will send a dataSend close event to indicate that the given transmission for the streamId is closed.

Example fmp4 files of a transmitted recording can be found in the examples directory. Additionally, a full writeup of the transmitted HomeKit Data Stream payloads for the given example can be found here.