search

rarebreed / khadga

Web application to gather real time video and text data over WebRTC and websockets
Other
3 stars 0 forks source link

Create signaling server for WebRTC peers to find each other #11

Open rarebreed opened 4 years ago

rarebreed commented 4 years ago

Need to work on the signaling server so that two WebRTC agents can find each other and start sending data.

Depends on #17

rarebreed commented 4 years ago

I am considering having the client (ie, front end) side of this done in noesis, and use webassembly for the websockets and WebRTC parts. Then the front end can import this.

rarebreed commented 4 years ago

I've enabled a right click menu for users in the sidebar. It just displays a popup that asks if you wish to start a video call or not. It will popup an alert box if you click Start.

Eventually, when the user clicks Start, it will instead send an SDP video offer message. This will be the beginning of the SDP for the RTCPeerConnection so that we can establish a common messaging protocol for the incoming and outgoing video/audio media stream

rarebreed commented 4 years ago

Just ran into an issue trying to test my latest changes. I got it to the point where my latest code didn't break existing functionality, but trying to set up the remote video for testing is going to be tricky.

I don't think I can have my own webcam be both the local and remote video element. Also, I want to have a separate user to send the SDP offers to. If I had more money, I would create a dev GKE deployment with a couple of nodes, and deploy it. Then i could have 2 browsers on 2 separate machines sign in.

Right now, I can sort of fudge it by running a local version of khadga on my dev machine and make it start up on 0.0.0.0 (and then figure out what my PC's real IP address is). The problem is a second PC to use. Both of my other machines are a pixelbook and a pixel slate, which are both chromeos machines. They don't have editable /etc/host files.

I need the /etc/host file in order to point the domain to a non IP address hostname. This is required for the Google Authentication from the web client credential from https://console.developers.google.com. So i created a dummy host name called 'stoner-test' on my linux PC, and that works. But that's not going to work for my chromeos books. I have linux beta installed on both, but the problem there is that I only have firefox running on linux. And it appears that it doesn't have access to the webcam.

It actually logs me in ok, but the webcam is blank, and no video type MediaStreams can be found. I might have to pull my old laptop (circa 2016) out of mothballs and install linux on it.

rarebreed commented 4 years ago

I've been plodding through getting this to work. It's taken far longer than I thought. I've already rewritten the code twice.

Part of the challenge has been determining who should manage some of the state. There's also the problem of initializing the websocket vs. initializing the webcam vs. setting up the RTC peer connection. Another technical challenge was integrating a non-react component into the redux state management. Unlike a react component which is connected to the state store via connect, if you have a regular class, it can't make use of it. Redux is not reactive, and so how do you have a class be informed of new state? react-redux takes care of this for you, by having components connected via connect automatically get new state and be re-rendered. This is not the case for plain old classes.

And since Redux is not supposed to contain side-effects, this presented a challenge. My approach so far has been to have a new class called WebComm which is itself currently, an object that is stored in redux. Eventually, I'll replace this, since WebComm is mutable. Eventually, i'll place this inside of and then use react context to "distribute" availability of the instance to anyone that needs it.

WebComm itself has a couple of RxJS Observables and streams. Components will either obtain current state by subscribing, or push new state by calling next() on Subjects to pass in new state. Frankly, I probably should have used mobx and/or vanilla rxjs from the beginning, but I wanted to get better at Redux.

Right now, the code can setup SDP offers to a connected client. I've been struggling to get past some serdes errors on the Rust side. I finally figured out why the serialization was failing for one case, so I have added a few unit tests to message.rs.

What i need to work on next is getting all the messages working and the RTC session flow working correctly. The base code is there. It's a matter of fixing bugs and making sure everything is lined up now.

rarebreed commented 4 years ago

this has been quite a bit harder than I thought it would be.

Partially, this is due to heavy use of rxjs, which is always tricky to get right. Another part was that some of the code in the example github repo was incorrect. And lastly, there are differences between how chrome and firefox implement webrtc.

Currently, I have it to the point where the sdp offers and answers are being transmitted, and the ICE events are being negotiated. Unfortunately the negotiation eventually fails. I am not sure if this due to NAT on my system, which requires TURN instead of STUN, or if this a test setup failure.

The remote video cam shows up, but it does not display video. I noticed that the componentDidMount code only runs once even though the track handler is updating redux several times (which should force a rerender, and then componentDidMount). I will investigate this next.

I think this is ready for a depolyment at least, even if only to let others test it

rarebreed commented 4 years ago

I may wind up refactoring a lot of this code. I think that the WebComm class, while ok in and of itself, shouldn't be passed around the way it is.

The WebComm instance is acting in a sense as a secondary state store, which sort of violates what redux is doing. The more I use redux, the less I like it. It seems to me that the problem is:

I might poke my nose into mobx and see how they handle things. Another option would be to just use rxjs. However, I would need to solve one problem (which presumably mobx does for you). The problem is basically what react-redux does for you in the top level component.

The problem is something like this. The store of streams could be at the App level, or some higher level component (ala Provider). A producer of events, ie, an Observable or Subject could insert or register itself into this top level store. Any code (Component or non-Component) interested in these events will then need to somehow discover or hook itself into this store, so that it can subscribe to the stream.

The little bit I have looked at Mobx has discouraged me, since they use decorator syntax to automagically create observables. I don't like that because decorator syntax can't be applied to functions themselves. A part of me would like to use cyclejs, but I've already invested a lot of work on react, so I don't think that's a good way to go.

I think I'll hand roll an rxjs library on my own. And then have subscriptions or pipes map to the useEffect or useState to get Components to re-render when state changes.