search

WebThingsIO / addon-list

List of installable add-ons for WebThings Gateway
Mozilla Public License 2.0
77 stars 54 forks source link

RTSP support #1295

Open createcandle opened 3 years ago

createcandle commented 3 years ago

Currently the Gateway has ONVIF support, but no RTSP support. What would it take to create an RTSP adapter? Are there things to look out for, or things to be aware of when potentially creating such an addon?

benfrancis commented 3 years ago

Currently the Gateway has ONVIF support, but no RTSP support.

I don't think that's quite correct. As I understand it the ONVIF adapter should support any ONVIF Profile S compatible camera, and all Profile S devices and clients must support RTSP.

RTSP just provides the media transport, ONVIF provides a wrapper around the RTSP stream which does a lot of other things like device discovery, authentication, capability description, network configuration, PTZ controls etc.

So I guess what you're really asking is whether an RTSP stream could be directly turned into a web thing, without using something like PSIA or ONVIF as a wrapper. I would guess that is possible, if users manually enter an rtsp:// URL it could be possible to create a Thing Description which exposes that video stream as a property (trans-coding the video as necessary) and offers some basic actions like play and pause.

I guess the more important question is what are you trying to achieve? Is there a device you'd like to support which doesn't support ONVIF, but does expose an RTSP stream directly?

mrstegeman commented 3 years ago

As an outside example, @atirage managed to get the Raspberry Pi camera module running as a native web thing: https://github.com/atirage/picamera-webthing

createcandle commented 3 years ago

@benfrancis Interesting!

The main goal is to have a privacy-focussed doorbell that could also double as a security camera, where the shutter and microphone can be trusted to be off. A physical camera shutter is essential to really offer peace of mind.

Here's an early prototype: https://www.candlesmarthome.com/exploring-a-privacy-friendly-smart-doorbell

That one's based on an ESP32, so it's not very powerful. It only supports motion jpeg, altough RTSP theoretically supports both MJPEG and H26X. So it should work?

A big advantage would be that it's cheap, safe to unplug, easy to create open source privacy-focused software for, and could be pre-programmed with a wifi password when flashed from Candle Manager (increasing transparency and trust).

A more powerful option would be to base this on a Raspberry Pi Zero W. This is probably a good idea, since it might also make the return audio channel easier. Theoretically it could even just run the WebThings Gateway software again (a security camera running Voco might offer attractive convergence). It also has downsides though. Having a larger attack surface is one. A principle for Candle projects is to use the 'least viable hardware', so if an ESP32 can handle it, that would be the preference. It's all research in progress.

I was thinking of creating an addon that receives a stream from the doorbell/cameras. It could store streams to the SD card or (ideally) an externally plugged in drive. This would be done on a rotation basis, or when a face/movement is detected. The addon should allow a user to look at and listen to this stream live, which might be done through the existing support, if it's easy to then quickly press an "unlock the door" button.

In the case of the doorbell, it should ideally also return speech to a tiny speaker (bluetooth? ;-) ), or otherwise allow the user to trigger pre-recorded audio ("please wait a second", "not interested"). Does the current UI allow for streaming audio back from the Gateway UI to an endpoint?

Privacy is once again the USP here so:

To complete the picture: the idea is to also have the Raspberry Pi create its own wifi accesspoint, so as to enforce that these devices are on a separate WiFi network from the home network. Especially because a camera doorbell literally hangs outside your house. It's important that if that device gets stolen, only that secondary wifi network is compromised (if its wifi password is extracted). In this doom scenario the user would have to generate a new network password and reflash the devices that were connecting to it so that they all have the new password again. Not having smart home devices communicate over the local network wherever possible is another one of Candle's principles.

Sumarily put, the question this project asks is the same that all Candle experiments have asked: can we have some level of convenience (and security in this case) without blindly creating a surveillance culture in and around the home?

// end of plan / pitch :-)

benfrancis commented 3 years ago

Here's an early prototype: https://www.candlesmarthome.com/exploring-a-privacy-friendly-smart-doorbell

If you're building your own device, why not make it a native web thing which exposes the VideoCamera capability directly and therefore doesn't need an adapter? You could look at what the ONVIF adapter does to expose a web-viewable video stream via a Thing Description and ideally natively output a web-viewable video stream so that the gateway doesn't need to transcode it (which is very resource intensive).

Does the current UI allow for streaming audio back from the Gateway UI to an endpoint?

No I don't think so, but the proposed MediaPlayer capability might be able to handle that.

flatsiedatsie commented 3 years ago

That's a good idea, thanks, I'll look into it.

flatsiedatsie commented 3 years ago

This looks interesting for a return audio channel: recording audio in JS.

flatsiedatsie commented 3 years ago

I'm getting an error that I'm not sure how to fix:

2021-02-23 16:05:07.201 ERROR  : Invalid message received: {
  "messageType": 8192,
  "data": {
    "adapterId": "candlecam",
    "device": {
      "id": "candlecam",
      "title": "Candlecam",
      "@context": "https://webthings.io/schemas",
      "@type": [
        "VideoCamera"
      ],
      "description": "A privacy friendly smart doorbell or security camera",
      "properties": {
        "stream": {
          "name": "stream",
          "value": null,
          "visible": true,
          "type": null,
          "@type": "VideoProperty",
          "readOnly": true,
          "links": [
            {
              "rel": "alternative",
              "href": "http://192.168.2.167:8000/stream.mjpg",
              "mediaType": "x-motion-jpeg"
            }
          ]
        },
        "snapshot": {
          "name": "snapshot",
          "value": null,
          "visible": true,
          "type": null,
          "@type": "ImageProperty",
          "readOnly": true,
          "links": [
            {
              "rel": "alternative",
              "href": "/extensions/candlecam/photos/latest.jpg",
              "mediaType": "image/jpeg"
            }
          ]
        }
      },
      "actions": {},
      "events": {},
      "links": [],
      "baseHref": "",
      "pin": {
        "required": false,
        "pattern": ""
      },
      "credentialsRequired": false
    },
    "pluginId": "candlecam"
  }
}
2021-02-23 16:05:07.202 ERROR  : Validation error: [
  {
    "keyword": "type",
    "dataPath": ".data.device.properties['stream'].type",
    "schemaPath": "#/properties/type/type",
    "params": {
      "type": "string"
    },
    "message": "should be string"
  }
]

But shouldn't type be None if it's a video stream?

flatsiedatsie commented 3 years ago

Setting it to a string of 'null' solved the warning. I still don't see the media show up, but that's another issue.

flatsiedatsie commented 3 years ago

I've managed to get an image to show.

atirage commented 3 years ago

@flatsiedatsie, for the video streams you need to set up a Tornado handler to serve that URL, i.e. the stream is not a static resource like the snapshot. Have a look here for an example: https://github.com/atirage/picamera-webthing/blob/aa782fcc7723d636a2cab0c296aa3babfaa37a8a/picamera-webthing.py#L16

flatsiedatsie commented 3 years ago

Thanks @atirage. Your work has actually been the inspiration for what I tried to do, after it was recommended to me earlier in this thread.

I have used this example from picamera to create an mjpg server. It works great if I open the link directly. So my hope was that the Gateway would just set the mjpg stream URL as the src of the image object. But it seems that's not the case. https://picamera.readthedocs.io/en/release-1.13/recipes2.html#web-streaming

flatsiedatsie commented 3 years ago

I guess it would require the gateway to use an img element instead of a video element, since it seems the video element can't handle motion jpeg. Weird.

If I hack in an image element and set the mjpeg url as it's source... cam

So close :-)

flatsiedatsie commented 3 years ago

I've tried to create a hybrid ffmpeg command based on the settings in the onvif-adapter.

ffmpeg -input_format yuyv422 -fflags nobuffer -vsync 0 -f video4linux2 -s 1280x720 -r 10 -i /dev/video0 -f alsa -ac 1 -ar 44100 -i hw:1,0 -map 0:0 -map 1:0 -c:a aac -b:a 96k -c:v h264_omx -r 10 -b:v 800k -copyts -probesize 200000 -window_size 5 -extra_window_size 10 -use_timeline 1 -use_template 1 -hls_playlist 1 -format_options movflags=empty_moov+omit_tfhd_offset+frag_keyframe+default_base_moof -seg_duration 1 -dash_segment_type mp4 -f dash ~/.webthings/media/candlecam/stream/index.mpd -remove_at_exit 1 -loglevel quiet

This leads to the video element having some source, although something is clearly still wrong.

dash

FFMPEG seems to be chugging along:

2021-02-23 22:55:27.331 ERROR  : candlecam: frame= 6126 fps= 10 q=-0.0 size=N/A time=448365:45:14.77 bitrate=N/A speed=2.64e+06x    
2021-02-23 22:55:27.837 ERROR  : candlecam: frame= 6131 fps= 10 q=-0.0 size=N/A time=448365:45:14.77 bitrate=N/A speed=2.63e+06x    
2021-02-23 22:55:28.238 ERROR  : candlecam: [dash @ 0x165c2a0] Opening '/home/pi/.webthings/media/candlecam/stream/index.mpd.tmp' for writing
2021-02-23 22:55:28.240 ERROR  : candlecam: [dash @ 0x165c2a0] Opening '/home/pi/.webthings/media/candlecam/stream/media_0.m3u8.tmp' for writing
2021-02-23 22:55:28.240 ERROR  : candlecam: [dash @ 0x165c2a0] Opening '/home/pi/.webthings/media/candlecam/stream/chunk-stream0-00512.m4s.tmp' for writing

The MPD file looks like this:

<?xml version="1.0" encoding="utf-8"?>
<MPD xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xmlns="urn:mpeg:dash:schema:mpd:2011"
    xmlns:xlink="http://www.w3.org/1999/xlink"
    xsi:schemaLocation="urn:mpeg:DASH:schema:MPD:2011 http://standards.iso.org/ittf/PubliclyAvailableStandards/MPEG-DASH_schema_files/DASH-MPD.xsd"
    profiles="urn:mpeg:dash:profile:isoff-live:2011"
    type="dynamic"
    minimumUpdatePeriod="PT1S"
    suggestedPresentationDelay="PT1S"
    availabilityStartTime="2021-02-23T21:45:14Z"
    publishTime="2021-02-23T21:47:05Z"
    timeShiftBufferDepth="PT6.0S"
    minBufferTime="PT2.4S">
    <ProgramInformation>
    </ProgramInformation>
    <Period id="0" start="PT0.0S">
        <AdaptationSet id="0" contentType="video" segmentAlignment="true" bitstreamSwitching="true">
            <Representation id="0" mimeType="video/mp4" codecs="avc1.640028" bandwidth="2000000" width="1280" height="720" frameRate="10/1">
                <SegmentTemplate timescale="10240" initialization="init-stream$RepresentationID$.m4s" media="chunk-stream$RepresentationID$-$Number%05d$.m4s" startNumber="88">
                    <SegmentTimeline>
                        <S t="1069056" d="12288" r="4" />
                    </SegmentTimeline>
                </SegmentTemplate>
            </Representation>
        </AdaptationSet>
        <AdaptationSet id="1" contentType="audio" segmentAlignment="true" bitstreamSwitching="true">
            <Representation id="1" mimeType="audio/mp4" codecs="mp4a.40.2" bandwidth="96000" audioSamplingRate="48000">
                <AudioChannelConfiguration schemeIdUri="urn:mpeg:dash:23003:3:audio_channel_configuration:2011" value="1" />
                <SegmentTemplate timescale="48000" initialization="init-stream$RepresentationID$.m4s" media="chunk-stream$RepresentationID$-$Number%05d$.m4s" startNumber="1">
                    <SegmentTimeline>
                    </SegmentTimeline>
                </SegmentTemplate>
            </Representation>
        </AdaptationSet>
    </Period>
</MPD>
flatsiedatsie commented 3 years ago

I've managed to get it working. Kind of.

ffmpeg -y -f alsa -ac 1 -ar 44100 -i hw:1,0  -f v4l2 -fflags nobuffer -vsync 0 -video_size 640x480 -framerate 10 -i /dev/video0 -muxdelay 0 -vcodec h264_omx -keyint_min 0 -g 10 -map 0:a -c:a aac -b:a 96k  -map 1:v -b:v 400k -video_track_timescale 9000 -f dash -seg_duration 1 -use_template 1 -use_timeline 1 -remove_at_exit 1 -window_size 6 -extra_window_size 10 /home/pi/.webthings/addons/candlecam/stream/index.mpd

I can see that stream in Shaka player, which I'm using in a UI extension. My goal was to have the stream work both in the UI extension and in the thing itself. But it still doesn't work if I click on the stream property on the thing itself. It still stuck on the loading animation. Using the /media/candlecam/stream/index.mpd location didn't solve that.

The lowest delay I've been able to get is between 5 and 6 seconds (including telling Shaka player to buffer as little as possible).

flatsiedatsie commented 3 years ago

It's working! Man this was a tough one.

To answer one of my own questions for future reference:

Are there limitations to where the index.mpd file may be placed in the file system

No, but the limitation is that href in the thing's json can't be set to an absolute url. the first part of these URL's is always formed by the somewhat mysterious base variable, which provides the first part of the URL. Because of that, the thing json and the media files are expected to come from the same port. So you can't have a thing providing json on port 8888, and then refer to a stream file on port 8080. It's a package deal.

For me, the trick was to first stop using the addon route to create things, and instead use the thing url adapter route. Secondly I had to make the index.pdb file available on port 8888 as well. For that I had to add some additional routes to the python webthing's webserver. Luckily, this was easy to do.

        more_routes = [
            (r"/media/candlecam/(.*)", tornado.web.StaticFileHandler, {"path": "/home/pi/.webthings/media/candlecam"})
        ]

        thing_server = WebThingServer(SingleThing(thing), port=8888, additional_routes=more_routes)

Phew

LucasFerrazBR commented 2 years ago

Hello flatsiedatsie!

I'm trying to create an RTSP Add-on to stream my camera in the gateway, i've tried the Onvif Add-on but it does not work for my camera (other Onvif client softwares do work for her). I just found this thread and it's looks like that ou made an add-on for RTSP stream, is this correct?

I would like to use this add-on too, can you disponibilize it or tell how you made it, please?

Thank you!

It's working! Man this was a tough one.

To answer one of my own questions for future reference:

Are there limitations to where the index.mpd file may be placed in the file system

No, but the limitation is that href in the thing's json can't be set to an absolute url. the first part of these URL's is always formed by the somewhat mysterious base variable, which provides the first part of the URL. Because of that, the thing json and the media files are expected to come from the same port. So you can't have a thing providing json on port 8888, and then refer to a stream file on port 8080. It's a package deal.

For me, the trick was to first stop using the addon route to create things, and instead use the thing url adapter route. Secondly I had to make the index.pdb file available on port 8888 as well. For that I had to add some additional routes to the python webthing's webserver. Luckily, this was easy to do.

        more_routes = [
            (r"/media/candlecam/(.*)", tornado.web.StaticFileHandler, {"path": "/home/pi/.webthings/media/candlecam"})
        ]

        thing_server = WebThingServer(SingleThing(thing), port=8888, additional_routes=more_routes)

Phew

flatsiedatsie commented 2 years ago

Hey Lucas, I was unable to make such an addon. I have created / am creating a security camera /smart doorbell which works with the gateway, as part of a privacy research project. It can switch between fast-but-no-sound mjpeg output or delayed-but-with-sound Dash output.

The reason RTSP might not work as you hope is that the gateway has a built-in system where it wants to be able to forward the stream over it's 'tunnel' system, so you can watch your streams from outside of the house by logging into your gateway through that tunnel. RTSP could work around this (literally), but at the moment there is no built-in way to show the RTSP stream in the UI directly.