Initiating WebRTC stream without offer_file

[bankersheldrake]

Hi Allen, thankyou for all the work you have done on this, as a novice python coder I am overwhelmed by the complexity of this. This is my first attempt at engaging in a github project, so apologies if I get anything wrong in approaching you.

I have the nest camera battery. I have been able to install and execute the python code directly (raspberrypios) to pull structures, devices which return data as expected, and event subscriptions which trigger as expected. I have been also able to install the home assistant module to stream from my cameras, which works, just to ensure that it was possible on my setup.

I have not been able to pull an image directly from the cameras (which I think is not supported for the model camera anyway), and I have not been able to initiate a WebRTC stream. I think the issue relates to my offerSDP file, but I cannot see anywhere what format it needs to be, and what form to provide it to the SDM. It may be something else too.

I'm hoping you can direct me on what I am missing?

in response to this POST request[post json]=%s {'command': 'sdm.devices.commands.CameraLiveStream.GenerateWebRtcStream', 'params': {'offerSdp': '{'}}

I have tried various forms of the based on what I can find online, but none have led to a different response than :

ApiException: Error from API: 400: INVALID_ARGUMENT: Request contains an invalid argument.: Bad Request which is triggered by: File "xxxx/python3.7/site-packages/aiohttp/client_reqrep.py", line 1009, in raise_for_status headers=self.headers, aiohttp.client_exceptions.ClientResponseError: 400, message='Bad Request', url=URL('https://smartdevicemanagement.googleapis.com/v1/enterprises/xxxxxx/devices/xxxxxx:executeCommand')

[allenporter]

Hi, thanks for reaching out, you are totally welcome to engage here!

This is my first project working on WebRTC so I am not an expert and agree it's tough to get started. What we did in home assistant was use JavaScript to initiate the WebRTC offer. I suggest installing home assistant and getting it working there, then turning up debug logging in python so that you can view the webrtc offer sent to the nest API. Does that make sense? Happy to elaborate on the steps but thought a quick reply could help.

[allenporter]

The offer sdp is incredibly complicated, for what it's worth, so you likely need a client library capable of speaking it in order to get frames from the video. E.g. there are python libraries like aiortc but I have not tried them. Happy to share what I know though.

[bankersheldrake]

Thanks! I'll do some more digging into the debug option in homeassistant and report back. I'd love to get involved in the project you're working on, if I can add anything. In your code (google_nest.py, line 113) there is a reference to the commandline option of 'offer-file'. Is that a file that is expected to be provided by the caller, or created after the WebRTC connection is made?

[allenporter]

I added that when I had no idea how webrtc worked and was trying to debug it. I don't think you should use it at all really.

Happy to collaborate.

[bankersheldrake]

OK. I'll come back when I've got some progress. cheers

[bankersheldrake]

I'm doing more work on python code using the nest stream that is created to capture snapshots, saved videos etc, but I wanted to let you know that I have a working sdp offer which can be inserted into your code to make the WebRTC pathway functional in your library. I've not collaborated on github before so I'm not confident to fork and submit my code, but here is is in case it is useful to you , or to others who are searching for this methodology.

google-nest.py line 51- insert

 from aiortc import (
    RTCPeerConnection
)

google-nest.py line 325 - insert

remotePeerConnection = RTCPeerConnection()
remotePeerConnection.createDataChannel("dataSendChannel")
remotePeerConnection.addTransceiver("audio", direction="recvonly")  #audio transceiver must be created first for google (per https://stackoverflow.com/questions/70281052/yet-another-invalid-argument-error-on-nest-battery-cam-generatewebrtcstream-comm), and recvonly (per google spec)
remotePeerConnection.addTransceiver("video", direction="recvonly")

await remotePeerConnection.setLocalDescription(await remotePeerConnection.createOffer())   
offer_sdp = remotePeerConnection.localDescription.sdp

[allenporter]

Thanks, I've been meaning to play with aiortc so that's a nice code snippet to get started.

What should the tool do with the answer once you have it? Just print it for debugging/ example?

[bankersheldrake]

Currently I'm building a python function to capture the answerSDP from sdm and load a stream into an aiortc MediaRecorder. I'm still trying to make that work, but can capture an image from the frame stream, so am confident that I can make that work. I'm sourcing a lot from the aiortc code examples, and other webrtc examples that are doted around the net. I have already built a js code module linked to a simple HTML page that uses the generated RTC stream to show the cameras in a live feed and I might keep working on that (i expect that is what the homeassistant team have already built).

I would think your library would expose some "WebRTC to video" functions, which is the obvious (IMO) thing to do with the generate_webrtc_stream CLI action.

I have a couple of use cases. I'm doing this on an rPI4. 1) is to bypass the need for the google nest aware subscription (which is $10 month and gives 30 days retention and person detection events, the basic option is 3 hours retention and no person detection). So I am planning on using the subscriber events code you've already written to trigger video save of the event, which I'll sync to the cloud and deal with storage size etc myself. 2) I'll take an image snapshot every n seconds so that I can get an immediate preview of the current view for events (like doorbell, or door open), rather than waiting for the 2-3 seconds that the video load takes in google home app 3) I want to play with object detection to see if I can emulate person detection (and because I'm interested in it) 4) I will play with constant video capture to see if I can setup a proper home surveillance system rather than relying on google event detection for video capture.

I was planning on writing all of that myself, using your token create/refresh code for the google SDM, but I wonder if any would appeal to you for your CLI options?

[allenporter]

It might be cool to have the command line tool support a simple "grab snapshot" or "save video" for X second kind of functionality?

I would gladly accept a pull request I'd it's something you want to add.

BTW I only started learning GitHub, forking, PRs for this project and find it really worthwhile to be able to do so I encourage you to! (Having contributed small changes to many different projects out there, it's great to do!)

[bankersheldrake]

OK, I'll give it a try. cheers