search

WebThingsIO / gateway

WebThings Gateway - a self-hosted web application for monitoring and controlling a building over the web
http://webthings.io/gateway
Mozilla Public License 2.0
2.61k stars 339 forks source link

Camera Capability UI #1438

Closed benfrancis closed 5 years ago

benfrancis commented 6 years ago

As a user I want to access images or video from a camera so that I can check up on my home while I'm away.

benfrancis commented 6 years ago

See also: https://github.com/mozilla-iot/gateway/issues/712

mrstegeman commented 6 years ago

The properties I added to the ONVIF adapter can be seen here: https://github.com/mozilla-iot/onvif-adapter/blob/master/lib/onvif-device.js#L43

There are also motion (and other?) events in the spec, but my camera doesn't expose them, so I'm unsure what all is available.

The documentation for the library I'm using is here. It's got pretty good coverage of the spec, so you should be able to get a good idea of what's available.

drodbez commented 5 years ago

Adding onto the camera capability... Create custom event where RTSP h.264/h.265 video is sent to the desired chromecast. -optionally add feature to do this automatically after motion or other triggered events occur.

Because most all smart TV's have chromecast built in, chromecast is already working with iot gateway and chromecast supports RTSP this seems highly feasible?

Would suggest discouraging any mjpeg footage due to its extensive bandwidth.

Also, viewing RTSP footage with Mozilla stopped when NPAPI support was dropped. Another possible solution could be something similar to this embedded? https://stackoverflow.com/questions/2245040/how-can-i-display-an-rtsp-video-stream-in-a-web-page

I run a small security camera company, so I have a few different brand ONVIF cameras from fixed, motorized zoom, PTZ, fisheye at my disposal that I will gladly help test with.

mrstegeman commented 5 years ago

Using RTSP Directly

A Javascript-based player is available from Streamedian. However, in order to use that player, you have to license their websocket server.

Converting RTSP to WebRTC

WebRTC is a web standard, but not yet supported by all browsers -- notably absent is Safari, both on desktop and mobile.

In order to transcode to WebRTC, a server is needed. I have found 2 open source solutions, but both have extremely heavy dependencies:

I like the thought of using WebRTC, but Janus is super heavy, and I couldn't get WebRTC-streamer to work with my camera.

Converting RTSP to MPEG-DASH

MPEG-DASH is a web standard, but not yet supported by all browsers -- notably absent is Safari, both on desktop and mobile which seems to be supported by all popular browsers, either natively or with a library.

This standard seems to be a bit of a mess right now. Supported codecs have been shifting, and transcoding is quite ugly, as multiple output files have to be generated. Playback support has to be added via library.

Converting RTSP to HLS

HLS is a proprietary protocol created by Apple. However, the spec is freely available as RFC 8216. HLS is supported in most browsers, if not natively then with a library.

Transcoding with ffmpeg is actually very, very light on resources. For example:

ffmpeg \
    -i "rtsp://<stream_url>" \
    -fflags flush_packets \
    -max_delay 1 \
    -an \
    -flags \
    -global_header \
    -hls_time 1 \
    -hls_list_size 3 \
    -hls_wrap 3 \
    -vcodec copy \
    -y \
    /tmp/video.m3u8
<!doctype html>
<html>
  <head>
    <title>HTML5 Live Streaming Test</title>
  </head>
  <body>
    <script src="https://cdn.jsdelivr.net/npm/hls.js@latest"></script>
    <video id="video" width="640" height="400" controls="controls"></video>
    <script>
      const url = "/video.m3u8";
      const video = document.getElementById('video');

      if (Hls.isSupported()) {
        const hls = new Hls();
        hls.loadSource(url);
        hls.attachMedia(video);
      } else if (video.canPlayType('application/vnd.apple.mpegurl')) {
        video.src = url;
      }
    </script>
  </body>
</html>
mrstegeman commented 5 years ago

If we go the HLS or MPEG-DASH route, codecs are as follows:

HLS allows some additional audio codecs, i.e. AC-3 and EC-3, but neither of those has wide browser support.

MPEG-DASH is "codec-agnostic", but the codecs above are still the ones with the widest support.

benfrancis commented 5 years ago

I think based on the above my personal preference would be for RTSP converted to MPEG-DASH over HLS, if we can get it to work well, just because it's more of an open standard.

See https://github.com/mozilla-iot/schemas/issues/25 for proposed Camera and VideoCamera capabilities.

Here is a start on a UI design: camera_and_video_camera_capabilities

In this case I think we might need to break from the "splat" style design pattern in order to make more room for the video/image display and not require a second click to view them. However, some open questions:

mrstegeman commented 5 years ago
  • What media controls are necessary? Will the standard <video> playback controls suffice for videos? Do we need a reload button for an ImageProperty?

Standard controls should be fine, and could probably even be disabled. I'm not sure how well seeking will work in a live broadcast.

  • What happens if the Camera or VideoCamera capability are combined with another capability like MotionSensor or some kind of PTZ functionality? It may be difficult to display other properties with this design.

What if we still have a splat, but there is a generic video stream icon that can expand into the player? That would allow for other properties to be displayed as usual.

  • What happens if both an ImageProperty and VideoProperty are defined, or more than one of each?

The above comment applies here as well.

benfrancis commented 5 years ago

What if we still have a splat, but there is a generic video stream icon that can expand into the player? That would allow for other properties to be displayed as usual.

That was actually the design I came up with first. What I didn't like with that approach was that it would require two clicks from the Things screen to get the video to play and to see any actual useful information.

We may be forced to do something like that if we want to display many other properties though...

freaktechnik commented 5 years ago

What if instead of replacing the splat there's an alternative view, similar to how the external view a thing can specify as alternate works?

benfrancis commented 5 years ago

What if we still have a splat, but there is a generic video stream icon that can expand into the player? That would allow for other properties to be displayed as usual.

That could look something like this.

video_camera

I'm just not keen on requiring another click to get to the video...

mrstegeman commented 5 years ago

@benfrancis Yep, that's what I was thinking of. @freaktechnik's idea is interesting as well.

benfrancis commented 5 years ago

Here is what I propose:

camera_and_video_camera_capabilities [edited]

This uses the two stage approach Mike suggested. I explored having an alternative view, but with things already having the default system view and potentially a linked custom view, adding a third alternative seems really confusing.

Note the reload button on the expanded image property view so you can re-fetch the image. @mrstegeman Do you think this makes sense?

I've added specifications for a Camera Capability, VideoCamera Capability, Image Property and Video Property to the UX spec.

Let me know what you think and I'll start exporting all the icons.

mrstegeman commented 5 years ago

I like it. I updated the ONVIF adapter today to match the new schemas and did some prep work in the gateway, so I should be able to start implementing this. Icons would be great.

benfrancis commented 5 years ago

Icons:

benfrancis commented 5 years ago