Add Camera Devices (Doorbells, Hubs, Cameras, etc.)

ErikSGross commented 2 years ago

Description

Though the Google SDM API allows access to the cameras integrated into the Nest Hello doorbell and Nest Hub Max products, it does not allow control of the devices. Additional control and information for these devices is located in the Nest mobile app, so perhaps these devices can be enhanced by this integration.

The doorbell and hub devices are available as quartz buckets like this:

        {
          "object_key": "buckets.xxxxxxx",
          "object_revision": -xxxx,
          "object_timestamp": xxxxxxxxxxxxx,
          "value": {
            "buckets": [
              "quartz.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
            ]
          }
        },

The object_key entry for a doorbell looks like this:

          "object_key": "quartz.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
          "object_revision": -xxxxx,
          "object_timestamp": xxxxxxxxxxxxx,
          "value": {
            "capabilities": [
              "image.properties",
              "force_idr",
              "streaming.data-usage-tier.120",
              "streaming.cameraprofile.VIDEO_H264_530KBIT_L31",
              "spoken_locale.da_DK",
              "aspect_ratio_4x3",
              "sharing",
              "audio.microphone",
              "spoken_locale.en_US",
              "indoor_chime",
              "spoken_locale.sv_SE",
              "streaming.params",
              "streaming.cameraprofile.VIDEO_H264_2MBIT_L40",
              "stranger_detection",
              "spoken_locale.es_US",
              "spoken_locale.de_DE",
              "statusled",
              "spoken_locale.it_IT",
              "streaming.data-usage-tier.300",
              "streaming.cameraprofile.VIDEO_H264_100KBIT_L30",
              "spoken_locale.fr_FR",
              "statusled.brightness.settable",
              "backfill",
              "watermark",
              "spoken_locale.nb_NO",
              "spoken_locale.es_ES",
              "labs",
              "spoken_locale.fr_CA",
              "audio.speaker",
              "spoken_locale.fi_FI",
              "dptz",
              "irled",
              "notify.email",
              "standby",
              "detectors.on_camera",
              "spoken_locale.en_GB",
              "streaming.start-stop",
              "spoken_locale.nl_NL",
              "spoken_locale",
              "dptz.8x",
              "streaming.data-usage-tier.30",
              "statusled.watching"
            ],
            "last_disconnect_time": xxxxxxxxxxxxx,
            "cvr_enrolled": "xxxxxxx",
            "last_connect_time": xxxxxxxxxxxxx,
            "activation_time": xxxxxxxxxxxxx,
            "fabric_id": "xxxxxxxxxxxxxxxx",
            "websocket_nexustalk_host": "xxxxxxxxxx-us1.dropcam.com:80",
            "model": "Nest Doorbell (wired)",
            "description": "",
            "direct_nexustalk_host": "xxxxxxxxxx-us1.dropcam.com",
            "audio_input_enabled": true,
            "download_host": "xxxxxxxxxx-us1.dropcam.com:80",
            "recorded_stream_host": "xxxxxxxxxx-us1.dropcam.com:1935",
            "mac_address": "xxxxxxxxxxxx",
            "structure_id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
            "last_disconnect_reason": "no_reason",
            "software_version": "4110054",
            "wwn_stream_host": "xxxxxx-ue1-delta.dropcam.com",
            "live_stream_host": "xxxxxxxxxx-us1.dropcam.com:1935",
            "ip_address": "xxxxxxxxxxxxx",
            "camera_type": 12,
            "streaming_state": "streaming-enabled",
            "nexus_api_http_server_url": "https://nexusapi-us1.dropcam.com",
            "where_id": "00000000-0000-0000-0000-xxxxxxxxxxxx",
            "serial_number": "xxxxxxxxxxxxxxxx",
            "preview_streaming_enabled": true,
            "snapshot_url": {
              "snapshot_url_prefix": "https://www.dropcam.com/api/wwn.get_snapshot",
              "snapshot_url_suffix": ""
            },
            "weave_device_id": "xxxxxxxxxxxxxxxx",
            "public_share_enabled": false
          }
        },

The object_key for the Hub Max looks like this:

        {
          "object_key": "quartz.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
          "object_revision": xxxxx,
          "object_timestamp": xxxxxxxxxxxxx,
          "value": {
            "capabilities": [
              "image.properties",
              "force_idr",
              "streaming.data-usage-tier.100",
              "streaming.data-usage-tier.200",
              "aspect_ratio_4x3",
              "streaming.start-stop",
              "audio.microphone",
              "background-update",
              "statusled",
              "log.level",
              "stranger_detection",
              "streaming.params",
              "dptz.12x",
              "streaming.data-usage-tier.300",
              "live_view.chime",
              "backfill",
              "watermark",
              "talkback.chime",
              "can-decode-unknown-protobuf-fields",
              "notify.email",
              "audio.speaker",
              "labs",
              "standby",
              "detectors.on_camera",
              "statusled.watching",
              "streaming.daily-data-usage-cap"
            ],
            "last_disconnect_time": xxxxxxxxxxxxx,
            "cvr_enrolled": "xxxxxxx",
            "last_connect_time": xxxxxxxxxxxxx,
            "activation_time": xxxxxxxxxxxxx,
            "fabric_id": "xxxxxxxxxxxxxxxx",
            "websocket_nexustalk_host": "xxxxxxxxxx-us1.dropcam.com:80",
            "model": "Google Nest Hub Max",
            "description": "",
            "direct_nexustalk_host": "xxxxxxxxxx-us1.dropcam.com",
            "audio_input_enabled": false,
            "download_host": "xxxxxxxxxx-us1.dropcam.com:80",
            "recorded_stream_host": "xxxxxxxxxx-us1.dropcam.com:1935",
            "mac_address": "xxxxxxxxxxxxxxxx",
            "structure_id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
            "last_disconnect_reason": "no_reason",
            "software_version": "290465-100000",
            "wwn_stream_host": "stream-uc2-bravo.dropcam.com",
            "live_stream_host": "xxxxxxxxxx-us1.dropcam.com:1935",
            "ip_address": "xxxxxxxxxxxxx",
            "camera_type": 14,
            "streaming_state": "online-disabled",
            "nexus_api_http_server_url": "https://nexusapi-us1.dropcam.com",
            "where_id": "00000000-0000-0000-0000-xxxxxxxxxxxx",
            "serial_number": "xxxxxxxxxxxxxx",
            "preview_streaming_enabled": true,
            "snapshot_url": {
              "snapshot_url_prefix": "https://www.dropcam.com/api/wwn.get_snapshot",
              "snapshot_url_suffix": ""
            },
            "weave_device_id": "xxxxxxxxxxxxxxxx",
            "public_share_enabled": false
          }
        },

Device (optional)

No response

Additional information

No response

iMicknl commented 2 years ago

Thanks! Would you know which values we need to expose (not supported in SDM only!) and which values we need to be able to change? (select or switch?).

(and can you change these settings via https://home.nest.com? In that case, you could have a look at the XHR request as well and share them here).

ErikSGross commented 2 years ago

As far as things to try to expose to Home Assistant:

Doorbells: binary_sensor for doorbell ring, motion/person/sound/package sensing; switches for camera on/off, status light on/off, indoor chime on/off, and motion/sound/person/package detection on/off
Hub Max and other cameras: motion/person/vehicle/package sensing; switch for camera on/off, status light on/off, and motion/sound/person/package detection on/off

Yes, you can control those things from home.nest.com. Not sure if sensing the doorbell ring or motion-related sensing is visible there, though.

Do you have any docs or a guide for how to look at the XHR request you mentioned? I'm not familiar with that, but I'm happy to try it out to get additional data.

iMicknl commented 2 years ago

@ErikSGross are you sure that doorbell ring etc. are not implemented yet in the SDM API?

Here you find an example of how an XHR request would look like: https://support.deskpro.com/en/kb/articles/how-to-inspect-ajax-requests-with-chrome-tools

ErikSGross commented 2 years ago

This is new to me, but poking around the XHR data, I see the following when I enabled a camera that was previously disabled:

There's a POST request to https://webapi.camera.home.nest.com/api/dropcams.set_properties with a payload of streaming.enabled=true&uuid=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

And then there's a response of:

{
  "status": 0,
  "items": [
    {
      "freetier.history.enabled": true,
      "alarms.streaming.enabled": true,
      "audio.enabled": true,
      "audio.inputgainlevel": 0.75,
      "streaming.cameraprofile": "AUTO",
      "doorbell.chime_assist.enabled": false,
      "notify.mobile_push.enabled": true,
      "protect.clips.enabled": false,
      "watermark.enabled": true,
      "low_motion_bitrate_ratio": 0,
      "audio.start-stop-sound": 1,
      "irled.state": "auto_on",
      "streaming.data-usage-tier": 300,
      "preview.streaming.enabled": true,
      "audio.recording.enabled": true,
      "statusled.brightness": 0,
      "streaming.enabled": true,
      "doorbell.indoor_chime.enabled": false,
      "notify.sound.enabled": false,
      "nest.away.streaming.enabled": true,
      "face_tracking.enabled": true,
      "video.flipped": false,
      "notify.offline.enabled": true,
      "audio.spoken_locale": "en_US",
      "notify.motion.enabled": true,
      "detectors.enabled": true,
      "log.level": "None",
      "low_motion_bitrate_ratio.enabled": false,
      "websocket_nexustalk_host": [
        "xxxxxxxxxxx-us1.dropcam.com:80"
      ],
      "nest.away.notify.enabled": false,
      "dptz.state": "1,1,0.5,0.5",
      "doorbell.theme": "default",
      "notify.email.enabled": false,
      "cvr.allowed": true,
      "doorbell.indoor_chime.duration": 300,
      "doorbell.indoor_chime.type": "mechanical",
      "doorbell.quiet_time.enabled_until": 0,
      "adaptive_bandwidth.enabled": true
    }
  ],
  "status_description": "ok",
  "status_detail": ""
}

I tried disabling the camera and I got basically the same thing, except streaming.enabled was false.

I'm not sure where the UUID comes from, but I'm assuming that's associated with the particular camera I toggled the enable for. Presumably, a call to that API would allow you to set/get any of those settings associated with the camera, right?

Is that what you're working with to observe/manipulate the topaz/kryptonite devices?

ErikSGross commented 2 years ago

Opening up the settings page of a doorbell results in this XHR entry:

GET request to: https://webapi.camera.home.nest.com/api/cameras.get_with_properties?uuid=xxxxxxxxxxxxxxxxxxxxxxxxx

Response is similar to above, but much more data proivided - the above entries are all contained within items->properties

iMicknl commented 2 years ago

Thanks @ErikSGross! It seems that cameras use another endpoint, which I didn't implement yet. I am quite busy the coming period, but let's keep these issues open to track this request.

Perhaps someone else with cameras is willing to implement this and otherwise we need to wait. The HomeBridge repository uses these endpoints as well, thus could be a good example.

ErikSGross commented 2 years ago

I wanted to see what functiuonality homebridge currently exposes, so I spun up an instance and added the homebridge-nest-cam plugin. For doorbells and Hub Max products, it exposes switches for streaming, chime, announcement, and audio. There's a sensor and a switch for the doorbell button. Also, there are sensors for:

motion
sound
person
package delivered
package retrieved
face - Unknown
face - [insert known name here] (there's a sensor like this for each known person)

The authentication method appears to be the same as what ha-nest-protect is using - namely log into home.nest.com with your Google account and get a refresh token. The same token works for both the cameras and for other nest products in the homebridge-nest plugin (smoke detectors, thermostats, temp sensors, locks, etc.)

The homebridge plugin has a configurable polling interval, so it appears this is cloud polling, not cloud push. Setting the polling to the default of 10s seems to get notifications reasonably quickly.

So it looks like it's definitely possible to get Google/Nest to cough up the data, and the method seems very similar to what's already present in ha-nest-protect.

iMicknl commented 2 years ago

They use the GRPC / protobuf API, however this works in another way and is hard to test and debug for me since I don't own any of these devices. When I have some more time in the future, I will perhaps give it another try, but for now this won't be implemented soon.

iMicknl / ha-nest-protect