search

element-hq / element-desktop

A glossy Matrix collaboration client for desktop.
https://element.io
GNU Affero General Public License v3.0
1.17k stars 267 forks source link

Screensharing should share audio on selected window #791

Open ndren opened 3 years ago

ndren commented 3 years ago

Is your suggestion related to a problem? Please describe. When using screen sharing, the sound for the selected window is unavailable.

Describe the solution you'd like The audio from the selected window should be shared. If possible, this should also be available from sharing the entire screen, however I am not sure if there is an API available for taking all sound output.

t3chguy commented 3 years ago

I don't see any API in browsers for this from a quick glance, it feels like internal browser behaviour unfortunately, do you know of any web apps which have this feature?

SimonBrandner commented 3 years ago

@t3chguy, not sure about browsers, but Electron can definitely do this

SimonBrandner commented 3 years ago

I am wondering if this should be merged into #12652. When that gets done properly (multiple WebRTC streams), this shouldn't be a problem to implement

t3chguy commented 3 years ago

The electron docs suggest that audio is an all-or-nothing deal, so wouldn't be able to get it from a certain application

To capture video from a source provided by desktopCapturer the constraints passed to navigator.mediaDevices.getUserMedia must include chromeMediaSource: 'desktop', and audio: false.

To capture both audio and video from the entire desktop the constraints passed to navigator.mediaDevices.getUserMedia must include chromeMediaSource: 'desktop', for both audio and video, but should not include a chromeMediaSourceId constraint.

SimonBrandner commented 3 years ago

For this, we are going to need to give the user the ability to change the volume of individual feeds, something similar to what Discord has. @gaelledel, could you please take a look into this. I think we could just go with a feed right-click context menu which would have a Mute button and a volume slider. Does that sound good to you? Hopefully, designing this should be quite straightforward

viasux commented 3 years ago

Just saying, this is one of the major issues with matrix for me! I'd love to see this implemented so I can call friends on matrix more! I love that there's screen sharing now, but this would make it perfect. 💜

MarcelCoding commented 3 years ago

I would like to say that Discord in the Browser Version on Chromium based Browsers is able to transfer audio for screen casts, if you select a specific window. (Element Desktop is Electron based just like Discord Desktop)

simvux commented 2 years ago

I would like to say that Discord in the Browser Version on Chromium based Browsers is able to transfer audio for screen casts, if you select a specific window. (Element Desktop is Electron based just like Discord Desktop)

afaik Discord uses an undocumented hidden Windows API for this behaviour directly. And just recently got support for it on MacOS via I assume their platform-specific API's as well.

pavel-vorobyev commented 2 years ago

Anu updates bout this issue?

MarcelCoding commented 2 years ago

I would like to say that Discord in the Browser Version on Chromium based Browsers is able to transfer audio for screen casts, if you select a specific window. (Element Desktop is Electron based just like Discord Desktop)

afaik Discord uses an undocumented hidden Windows API for this behaviour directly. And just recently got support for it on MacOS via I assume their platform-specific API's as well.

I don't quite know or found the code for this feature but I found the following in the electron documentation. There is something mentioned about audio recording: https://www.electronjs.org/docs/latest/api/desktop-capturer This electron app is also using audio: https://github.com/wulkano/Kap/search?q=audio

Maybe you could get something out of it. 🙂

ndren commented 2 years ago

Capturing systemwide audio, selected window audio and/or microphone audio is possible, I have found the documentation here on MDN. It depends on what the user agent allows but from what I understand there are only a few issues with this, like the ones documented by Electron.

SimonBrandner commented 2 years ago

We are aware that this is technically possible though the issue is awaiting design work which is necessary for it to progress

MarcelCoding commented 2 years ago

In chromium-based browsers, you can share a tab. There the audio is automatically provided by the browser, but therefore the audio has to be enabled as described at the MDN Link https://github.com/vector-im/element-desktop/issues/791. It would be a nice starting point for audio transmitting in screencasts.

Shyyak commented 2 years ago

Hopefully this can be implemented soon!

SimonBrandner commented 2 years ago

This will be in the next version of Element Call

viasux commented 2 years ago

This will be in the next version of Element Call

awesome

fighuass commented 2 years ago

This will be in the next version of Element Call

Amazing, but what is Element Call exactly, is it the desktop/mobile client or something separate?

SimonBrandner commented 2 years ago

This will be in the next version of Element Call

Amazing, but what is Element Call exactly, is it the desktop/mobile client or something separate?

It's this: https://element.io/blog/introducing-native-matrix-voip-with-element-call/

kazin-kharizma commented 2 years ago

This will be in the next version of Element Call

Amazing, but what is Element Call exactly, is it the desktop/mobile client or something separate?

It's this: https://element.io/blog/introducing-native-matrix-voip-with-element-call/

I’ve been using the testing environment of Element call and as I understand it, it is what runs natively in the 1:1 chats of the current Desktop version but there are still a lot of bugs when trying to share audio during screen share. Is there any recent update to this?

t3chguy commented 2 years ago

@kazin-essen Element Call doesn't run natively in 1:1 calls. That stack predates Element Call.

kazin-kharizma commented 2 years ago

@kazin-essen Element Call doesn't run natively in 1:1 calls. That stack predates Element Call.

Ohh, I see, do 1:1 calls still use Jitsi then, or some version of Element Call? To be honest I’ve never been quite sure what the difference was between call.element.io being tested, Video Call Beta in Labs and the 1:1 E2EE chat. Group Spaces chat uses Jitsi still right? The goal is to have which version be the final one? Call? Current 1:1? Something new?

t3chguy commented 2 years ago

1:1 calls use the current specced version of VoIP in Matrix, Element Call uses a more experimental one. Element Call is working to replace Jitsi in Element products.

viasux commented 1 year ago

This will be in the next version of Element Call

Is it live yet?

SimonBrandner commented 1 year ago

This will be in the next version of Element Call

Is it live yet?

Yep :)

444945 commented 1 year ago

This will be in the next version of Element Call

Is it live yet?

Yep :)

Apologies if I failed to understand anything but will this be implemented in element client as well?

SimonBrandner commented 1 year ago

At some point, yes, we'd like to implement it

ozinka commented 8 months ago

I've just recently started to use Element and miss this feature. Do you have a plans to implement it? Have to use Skype so far :( Thnx

xxfogs commented 5 months ago

Anything new in this regard?

phantasia333 commented 4 months ago

When this is done there should be no real reason to switch to a different application. I suggest looking at the audio screen sharing of third party Discord applications such as Vencord or Webcord, who used their own solutions to upend Discord's sloth towards the issue on GNU/Linux-based systems. I believe the repo, for Vencord's implementation, is called Venmic. If you know about this already, I apologize for the redundancy. This is not a "hurry up" message. Any perceived urgency is not as such. I'm just saying there's framework of ideas to work with out there if you guys are stumped right now, especially since they did such a heavy duty modification in a closed source application seeing the same issues on Linux, also using Electron. As someone learning RE, I find that significant in of itself, a treasure trove even, for open source projects who want to devise a plan for this themselves. Idk. good luck 🌈

trivarialthea commented 4 months ago

When this is done there should be no real reason to switch to a different application. I suggest looking at the audio screen sharing of third party Discord applications such as Vencord or Webcord, who used their own solutions to upend Discord's sloth towards the issue on GNU/Linux-based systems. I believe the repo, for Vencord's implementation, is called Venmic. If you know about this already, I apologize for the redundancy. This is not a "hurry up" message. Any perceived urgency is not as such. I'm just saying there's framework of ideas to work with out there if you guys are stumped right now, especially since they did such a heavy duty modification in a closed source application seeing the same issues on Linux, also using Electron. As someone learning RE, I find that significant in of itself, a treasure trove even, for open source projects who want to devise a plan for this themselves. Idk. good luck 🌈

Yeah, pretty much all the work you need to implement screensharing is in these two repos: https://github.com/maltejur/discord-screenaudio https://github.com/edisionnano/Screenshare-with-audio-on-Discord-with-Linux

Which is what Vencord used to make their solution work https://github.com/Vencord/venmic

Twig6943 commented 1 month ago

bump

ahmet-cetinkaya commented 1 month ago

It's too bad that this simple feature is still missing, if you want to become a primary project and not an alternative, you have to take care of this kind of user usability.

AtaOzeren commented 1 month ago

It's disappointing that a basic feature like sharing audio from a specific window is still missing. Adding this feature would greatly enhance the usability and value of your platform. I hope this issue is addressed in future updates to meet user expectations.

KweezyCode commented 1 month ago

any update? 3 years passed already

Twig6943 commented 1 month ago

this needs to be added asap

KweezyCode commented 1 month ago

this needs to be added asap

enough time has already passed since discord was blocked in Russia, people can't sit forever with VPN, there is still no decent alternative, but matrix could well be it

Twig6943 commented 1 month ago

this needs to be added asap

enough time has already passed since discord was blocked in Russia, people can't sit forever with VPN, there is still no decent alternative, but matrix could well be it

I second this.

t3chguy commented 1 month ago

Contributions welcome

Twig6943 commented 1 month ago

:/ wish I knew how to program stuff (would contribute on god)