Streaming

[mmccool]

• Streaming covers several protocols: RTSP, HLS, DASH, WebRTC, RTMP, etc...
• Streaming support (probably based on HLS/DASH) will also be included in Matter
• These do not cover all the same use cases: RTSP/RTMP common in IP cameras (simple, low-latency, supports multicast, UDP, suitable for LAN, etc), HLS/DASH tends to be for content delivery (supported by CDNs, higher latency but better compression...), WebRTC for videoconferencing (peer-to-peer, use in browser, low latency, etc)...
• We probably at least need to add a work-item for "streaming" to the detailed plan
• MAY not be as simple as (one or more) protocol bindings, not sure if we need another affordance type, but I THINK it can be handled with actions with some specific new "verbs" (e.g. streampublish, streamsubscribe, etc. subprotocols) coupled with protocol bindings mapping those to methods eg in RTSP
• Motivated by the fact that IP cameras using RTSP in particular are a commodity IoT device and can also be integrated into IoT systems (e.g. for a smart home or smart building, or even smart city use cases) using AI to recognize people, pets, packages, etc. Also, not a research project, see systems like Frigate (for example - I can even give a demo). AI systems like this in turn would be "Virtual Things", eg. services running on an edge computer (or in the cloud) that connect to cameras (for instance) and generate events, JSON data (e.g. describing the location of a recognized object), and so forth.
• Another motivation might be embedding of content, control of TVs, etc. For instance, specifying videos as documentation sources, starting and stopping playback of streams on TVs as part of a user interface, etc.

[benfrancis]

For the record, WebThings has a VideoProperty schema which exposes a video stream as a Property of a Thing.

For example, the ONVIF adapter for WebThings Gateway transcodes live video streams from IP cameras (using RTSP) into HLS and DASH video streams which can be rendered by web browsers and exposes them as a property which looks something like this in a Thing Description...

"properties": {
  "stream": {
    "type": null,
    "@type": "VideoProperty",
    "forms": [
      {
        "href": "/media/onvif/onvif-00626E925C7F/index.mpd",
        "contentType": "application/dash+xml"
      }
      {
        "href": "/media/onvif/onvif-00626E925C7F/master.m3u8",
        "contentType": "application/vnd.apple.mpegurl"
      }
    ]
  }
}

Server-Sent Events is arguably a streaming protocol over HTTP (using the text/event-stream content type), which we use for property and event affordances, e.g.

  "events": {
    "overheated": {
      "title": "Overheated",
      "@type": "OverheatedEvent",
      "data": {
        "type": "number",
        "unit": "degree celsius"
      },
      "description": "The lamp has exceeded its safe operating temperature",
      "forms": [{
        "href": "events/overheated",
        "subprotocol": "sse"
      }]
    }
  }

The ONVIF adapter also supports PTZ control of cameras and the Chromecast adapter supports remote control of TVs.

My point is that a lot may already be possible without necessarily needing to add new affordance or operation types.

Caveat: WebThings achieves a lot of this by bridging to web native protocols rather than attempting to directly describe non-web protocols. There may be additional requirements for describing certain protocols directly using protocol binding templates.

[mmccool]

I think this does not have to go into the charter it can go into the "details" document under new protocols. But it MAY also need some new TD features, e.g. a subprotocol.