search

humbertogontijo / homeassistant-roborock

Roborock integration for Home Assistant. This integration uses your devices from the Roborock App
GNU General Public License v3.0
649 stars 65 forks source link

Front camera #9

Open Horndrup opened 1 year ago

Horndrup commented 1 year ago

Thanks for all the great work. Everything is working fine.

I have a Roborock s6 maxV and would it be possible to see the stream from the front camera?

humbertogontijo commented 1 year ago

Hey. I couldn't find any information about the live camera protocol. I might need help getting this information

kristofferR commented 1 year ago

Yeah, the Mi Home app/API, that has been used until you and rovo89 managed to utilize the Roborock API, doesn't have camera support, so that's likely why no one has documented it.

ckdo8008 commented 1 year ago

I checked. The camera uses WebRTC. WebRTC must be available in Client Web. A connection is established only through a mutual handshake. Handshake is not public and can only be accessed by decompiling the app.

Brian-lub commented 1 year ago

I am so thrilled that you are making this integration, i have been using miio app but never satisfied with that, now I'm back with roborock app that is great, and i will also be thrilled to get access to the live cam, and the speak through robot function, if you are able to do it, happy to helt test it, i am runnig a S6 MavV.

licheng5625 commented 1 year ago

looking forward this feature

isdedina commented 1 year ago

looking forward for this feature too

benedikt-bartscher commented 1 year ago

@AlexxIT did some great work in https://github.com/AlexxIT/WebRTC and https://github.com/AlexxIT/go2rtc, if anyone tries integration in HA check out those repos

AlexxIT commented 1 year ago

Can someone provide how handshake process looks like? Maybe I can help. Ideally we need to make SDP exchange process.

Menelao147 commented 1 year ago

Can someone provide how handshake process looks like? Maybe I can help. Ideally we need to make SDP exchange process.

Hi, I decompiled the app and found this part where it talks about WebRTC and SDP, is that what you are looking for?

RRVideoModule.java.txt

AlexxIT commented 1 year ago

You are welcome https://github.com/AlexxIT/go2rtc/releases/tag/v1.3.0

Lash-L commented 1 year ago

Wow great work @AlexxIT! I don't have a Roborock with a camera but I'll spend some time implementing this into the integration. I'll need a few testers

@humbertogontijo had let me know he doesn't have much time for hobby projects at the moment, and I've been working on turning this into a core integration, so I have a pretty good understanding of the code base

benedikt-bartscher commented 1 year ago

Thanks @AlexxIT, i will test this with my S7 MAXV Ultra and report back with my results.

AlexxIT commented 1 year ago

I don't have a vacuum with a camera either. But I have thousands of users :) There is currently no support for reverse audio for the S6 MaxV model. I still plan to check what's wrong there.

copystring commented 1 year ago

Nice! I'll implement this in my roborock adapter for iobroker.

Lash-L commented 1 year ago

Looking at the go2rtc code, I don't see a straight forward way to add it into the roborock integration itself (that the core maintainers will accept when this part of the code gets added/ that I know of). + with me being unable to test anything I try, I'd rather leave this problem to someone else,.

I think for now it is definitely better for any user who wants access to the Roborock cameras to just download go2rtc, as it provides everything they need and can grab login from this integration. I'll create a PR to recommend as much in the Readme. And if anyone down the line has a better idea - they are more than welcome trying to implement it directly in the roborock integration - I believe the nest integration has done something similar in the past.

AlexxIT commented 1 year ago

As I know, Hass support WebRTC cameras like Nest. This is about direct connection between user browser and camera. Hass in this case must exchange SDP and candidates. I've never experienced this part of Hass, so I can't help. But in this case the camera only works for viewing. You cannot record video from the camera or send a stream to Frigate.

humbertogontijo commented 1 year ago

This is amazing! I just need to first run

service: roborock.vacuum_remote_control_start
data: {}
target:
  entity_id: vacuum.roborock_s7_maxv

So the vacuum leaves the dock, and it worked!

Great job @AlexxIT !

AlexxIT commented 1 year ago

You can also add vacuum as media player and stream TTS and mp3 to it :) https://github.com/AlexxIT/WebRTC#stream-to-camera

Works only with S7 MaxV for now. You need to select opus codec.

Important - current webrtc integration has old go2rtc inside. You need to use it with external go2rtc.

Lash-L commented 1 year ago

Yeah - I saw Nest's implementation @AlexxIT

Camera Entities have a function: async def async_handle_web_rtc_offer(self, offer_sdp: str) -> str | None: which nest returns their "answer_sdp" to

However, looking at nest, basically all of the webRTC logic seems to be internal in their pypi package and not built directly into hass.

Since I know basically nothing about webrtc and I don't have a camera to test it on, I wasn't going to attempt it. I'm sure someone with webrtc know-how/ a roborock with a camera to test on could get it done, but it is outside my knowledge and what I have time for.

Perhaps @humbertogontijo can directly integrate it if he has time/ sees the need.

JurajNyiri commented 1 year ago

I tested the camera working with the latest go2rtc, great job! When I open the stream, the vacuuming stops and vacuum announces that the video call has been started. Is there a way to have the stream running while the vacuum is vacuuming like in the app?

Should I open feature request @AlexxIT ?

AlexxIT commented 1 year ago

I think robot stops because audio. If you get rtsp://xxx:8554/robot1?video or http://xxx:1984/webrtc.html?src=robot1&media=video - go2rtc won't init audio stream.

I can add option to use only video for stream. So user can create two streams in go2rtc yaml. One with audio and one without.

JurajNyiri commented 1 year ago

The rtsp link worked flawlessly inside HA webrtc card, thank you very much!

Horndrup commented 1 year ago

What are the steps to see the frontcamera with go2rtc

polysa commented 1 year ago

What are the steps to see the frontcamera with go2rtc

  1. Add AlexxIT repo to Addons "https://github.com/AlexxIT/hassio-addons"
  2. Install & start go2rtc
  3. Open Web UI
  4. Go to Add --> Roborock
  5. Copy roborock://... if you have the Custom Roborock integration or add your Roborock cretential to generate the link.
  6. Go to Config and add the code:
streams:
    camera.vacuum:
      - roborock://..
  1. Click Save & Restart
  2. Go to Streams and click stream robot must be out of the dock in order for the stream to work.

If you want to add this to Lovelace install WebRTC Camera from HACS Add custom:webrtc-camera card add the rtsp url rtsp://x.x.x.x:8554/camera.vacuum?video <-- x.x.x.x Home Assistant IP

humbertogontijo commented 1 year ago

I think robot stops because audio. If you get rtsp://xxx:8554/robot1?video or http://xxx:1984/webrtc.html?src=robot1&media=video - go2rtc won't init audio stream.

I can add option to use only video for stream. So user can create two streams in go2rtc yaml. One with audio and one without.

Can you add the only audio option? I'm using a generic camera pointing to rtsp and shared to homekit. Now I can't use the vacuum because it always pauses the cleaning

Th3M1k3y commented 1 year ago

I think robot stops because audio. If you get rtsp://xxx:8554/robot1?video or http://xxx:1984/webrtc.html?src=robot1&media=video - go2rtc won't init audio stream.

I can add option to use only video for stream. So user can create two streams in go2rtc yaml. One with audio and one without.

I tried both links on my S7 MaxV, both gives me connection refused, do I need to change something else? I can view the camera from the roborock app on my phone.

blackax1 commented 1 year ago

What is the URL format. I do not use hassio so I can not install the addon. I do have my token for roborock I'm just not sure on the url format

acs-lux commented 10 months ago

I think robot stops because audio. If you get rtsp://xxx:8554/robot1?video or http://xxx:1984/webrtc.html?src=robot1&media=video - go2rtc won't init audio stream.

I can add option to use only video for stream. So user can create two streams in go2rtc yaml. One with audio and one without.

Can you add the only audio option? I'm using a generic camera pointing to rtsp and shared to homekit. Now I can't use the vacuum because it always pauses the cleaning

Same for me, i tried with ?video but it still makes a call and stops the vacuum. Frustrating, the roborock addon also becomes unavailable for a while.

acs-lux commented 10 months ago

I think robot stops because audio. If you get rtsp://xxx:8554/robot1?video or http://xxx:1984/webrtc.html?src=robot1&media=video - go2rtc won't init audio stream.

I can add option to use only video for stream. So user can create two streams in go2rtc yaml. One with audio and one without.

Can you add the only audio option? I'm using a generic camera pointing to rtsp and shared to homekit. Now I can't use the vacuum because it always pauses the cleaning

Same for me, i tried with ?video but it still makes a call and stops the vacuum. Frustrating, the roborock addon also becomes unavailable for a while.

Actually it does work perfectly, only if i send the robot to a pin location does it still pause. So using ?video at the end of the url and adding media: video seems to solve the issue.

Schlaudi commented 9 months ago

I think robot stops because audio. If you get rtsp://xxx:8554/robot1?video or http://xxx:1984/webrtc.html?src=robot1&media=video - go2rtc won't init audio stream. I can add option to use only video for stream. So user can create two streams in go2rtc yaml. One with audio and one without.

Can you add the only audio option? I'm using a generic camera pointing to rtsp and shared to homekit. Now I can't use the vacuum because it always pauses the cleaning

Same for me, i tried with ?video but it still makes a call and stops the vacuum. Frustrating, the roborock addon also becomes unavailable for a while.

Actually it does work perfectly, only if i send the robot to a pin location does it still pause. So using ?video at the end of the url and adding media: video seems to solve the issue.

HI ACS-LUX,

still struggling with the correct typo of the URL for my S7 MAXV .... my robot also stopps cleaning once i connect the remote viewing stream...

Could you please post how you URL looks like?

Mine is not working: rtsp://mqtt-eu-3.roborock.com:8883?HEREISMYTOKEN&pin=1234&video&media=video

BR

SeanVella commented 8 months ago

I think robot stops because audio. If you get rtsp://xxx:8554/robot1?video or http://xxx:1984/webrtc.html?src=robot1&media=video - go2rtc won't init audio stream.

I can add option to use only video for stream. So user can create two streams in go2rtc yaml. One with audio and one without.

I have the same problem as the others. When I load the stream, the vacuum stops the cleaning and states that it is paused for live stream.

dscarr10 commented 8 months ago

You can also add vacuum as media player and stream TTS and mp3 to it :) https://github.com/AlexxIT/WebRTC#stream-to-camera

Works only with S7 MaxV for now. You need to select opus codec.

Important - current webrtc integration has old go2rtc inside. You need to use it with external go2rtc.

This is a nice feature, but I wish there was a way to keep the robot from pausing cleaning while audio is connected. I would very much like my robot to be able to play music while cleaning.

niveknow commented 7 months ago

Same issue here as well. I can get the video stream, but it’ll stop the vacuums. Has anyone figured out the fix for this?

AlexxIT commented 7 months ago

This question has been answered a long time ago. Only video should be requested from go2rtc, without sound.

SeanVella commented 7 months ago

This question has been answered a long time ago. Only video should be requested from go2rtc, without sound.

Even when it is requested as video only on go2rtc, it still doesn't work and pauses the clean cycle.

AlexxIT commented 7 months ago

Maybe you doing something wrong. I don't have vacuum with camera, but other users said this is working method

niveknow commented 7 months ago

I did several reboots and reloads of the integration and I can confirm it is now working as expected. It might have been related to the full RSPT URL that I was referencing. I also noticed that the WebRTC card would continue to load while the robot is docked. I'm using the blueprint automation to disable when docked, but I went ahead and used a conditional card for the cam card to only display when NOT docked. The other puzzle is the occasional issue where the extracted vacuum card doesn't display, but I think this is related to the API as it magically reappears as it did this morning for me. Things appear to work as expected. Thank you.

jf4nathan commented 2 days ago

I did several reboots and reloads of the integration and I can confirm it is now working as expected. It might have been related to the full RSPT URL that I was referencing. I also noticed that the WebRTC card would continue to load while the robot is docked. I'm using the blueprint automation to disable when docked, but I went ahead and used a conditional card for the cam card to only display when NOT docked. The other puzzle is the occasional issue where the extracted vacuum card doesn't display, but I think this is related to the API as it magically reappears as it did this morning for me. Things appear to work as expected. Thank you.

@niveknow I'm at my wits end. How did you get this to work? I'm using the following url format: - roborock://mqtt-us.roborock.com:8883?u=xxxx&s=xxxx&k=xxxx&did=xxxx&key=xxxx&media=video&video

I tried rebooting the various integrations and restarting. It's still pausing the vacuum when I view the stream 😔