search

selkies-project / selkies-gstreamer

Open-Source Low-Latency Accelerated Linux WebRTC HTML5 Remote Desktop Streaming Platform for Self-Hosting, Containers, Kubernetes, or Cloud/HPC
Mozilla Public License 2.0
262 stars 43 forks source link

[META] Implement microphone, webcam, and other bidirectional capabilities #76

Open ehfd opened 1 year ago

ehfd commented 1 year ago

Basically, this is the opposite direction to the audio capabilities available now.

  1. Fetch microphone permissions from the web browser.
  2. Receive the microphone stream and encode to opus within the browser.
  3. Send the audio stream using RTP wrapped with WebRTC and the host/container Python component receives the stream.
  4. The host/container Python component first decodes the opus codec, and then creates a microphone device and sends audio to the host/container with pulsesink.

The gst-web components should be overhauled to accomodate new improvements such as webcams, printing, and file sharing in the future as well as to finish this job.

Would be convenient to investigate the following issues along the way for better usability if we are indeed overhauling gst-web for this task (Unfortunately, I don't know JavaScript too well compared to Python): #9 (AZERTY keyboard) #22 (Touch keyboard capabilities), #25 (URL parameters for authentication).

Additional issues (intertwined with gst-web and Python): #39 (Multiple users), #48 (WebTransport), #74 (WebRTC chunked DataChannel)

ehfd commented 1 year ago

This issue is from a request by @GeorgeDeac .

ehfd commented 1 year ago

Implemented support for Stereo on Opus. https://github.com/selkies-project/selkies-gstreamer/commit/1845bda05e0ed4baee9d5c6ac10ff884e2bd008f

ehfd commented 1 year ago

Good references:

https://github.com/evshiron/parsec-web-client https://parsec.app/blog/game-streaming-tech-in-the-browser-with-parsec-5b70d0f359bc https://github.com/parsec-cloud/parsec-sdk

satyaakam commented 2 months ago

Hi , i see this issue labelled under Funding , can anyone from core developers let me know what amount of funding are you expecting in order to see this issue resolved. Need few details like amount , timelines , and process of resolving it in order to get the ball rolling .

ehfd commented 2 months ago

Hi! @satyaakam

Unfortunately, the core developers are currently all busy, and no cost may justify the time we are to spend on this enhancement at this time.

However, @PMohanJ, another Indian developer and junior member of the Selkies team, is hired by his corporation to develop the webcam capabilities. I suggest waiting for them to open-source this capability.

satyaakam commented 2 months ago

Hi @ehfd thanks for the response , I am sorry if i sounded like someone who would like to throw money or people at a problem to get it resolved :-) , I understand the way FOSS communities work , my request was more from the point of view of collaboration, thanks for pointing to @PMohanJ will be more than happy to collaborate and work towards getting it resolved .

ehfd commented 2 months ago

I greatly appreciate your interest in this project. @satyaakam I would have taken the job if I had extra time. This project competes technically with well-funded proprietary remote desktop products and conversely relies on the free time of experienced developers in very specific, but wide-spectrum technology stacks. So donations never hurt. But currently, we really need people willing to learn the code and make their own contributions. Bounties might work out.

debaa98 commented 2 months ago

is there any bounty going to be paid to contributer?

ehfd commented 2 months ago

@debaa98 It is possible, if you are indeed interested.

debaa98 commented 2 months ago

Yaa I am intrested

ehfd commented 2 months ago

But I am not the entity which issues the bounty. Check with @satyaakam.

debaa98 commented 1 month ago

yas,i had a contact with @satyaakam