Open rexroni opened 4 years ago
Ok, a bit more progress, on my webrtc branch. I found the source of the problem with set-remote-description
not running; the pipeline had to be started before the set-remote-description
event was emitted. I also removed all of the promise.interrupt()
calls, I added more debug printing around signals, and I disabled the on-negotiation-needed
in when role=="answer"
, since in all the examples, only the offerer uses it.
It seems that the webrtc
object's signaling-state
property is correctly being set to STABLE
after the local and remote descriptions are set. However, it seems that the webrtc
object is not able to actually make a connection. The latest thing I tried was manually hard-coding the turnserver url into the webrtc code, but I haven't really observed any effect.
Thanks for taking a shot at this.
We have two webrtc branches. One is using aiortc the other one as you noticed gstreamer.
The aiortc one successfully connects to a riot call and transmits the generated picture. Sadly as the media backend it uses doesn't use gstreamer so it's a bit harder to integrate it into an UI.
The GStreamer one can be easily integrated into a GTK based UI, but I never could make a connection to riot, note that the implementation as far as i remember did successfully work with itself (connecting two GStreamer webrtc clients together).
I assume that the GStreamer based webrtc implementation isn't fully done or correct. We would probably need to contact upstream and try to work out what's wrong with it.
Do you have any strong preferences on which solution is "better"? I think I'm reasonably impressed by the gstreamer library; I'd be happy to hack away at that solution.
No preference at all, whatever works™. Using gstreamer would be probably better since it has so much support in the differing GUI toolkits.
Well I got it Kinda Working™. I can auto-answer voice calls from riot. What were you thinking in terms of a user interface? I think it would be neat to have an interface where audio-only calls worked without any gui support, but I'm having trouble imagining how that might work.
That's great news.
For the user interface, nothing fancy is needed:
GUI-less audio support sounds neat, but probably hard to get right, and not everybody would be a fan of it. Lets focus first on the main use-case.
Any info on how you got it working?
Well I tried a lot of things, and I'm not 100% certain which ones were critical and which ones weren't. But the important ones seem to be:
negoitiation-needed
signal when answering, since that's intended to trigger the offerer to create-offer
(not sure why the signal fires on the answerer after all the normal signaling is complete, but that's what I was seeing).reply["answer"]
had to be replaced with reply.get_value("answer")
which is basically identical to the C api. Similarly, self.pipe.add(q, conv, sink)
had to become three separate self.pipe.add()
calls.start_pipeline()
was called right after set-remote-description
, which was resulting in a None
value for reply
from that promise... which was causing havoc right before we could set the local description.matrix
plugin side, especially around trying to parse messages. Looking back at the original code, there was a line that called print(self.role)
to stdout which may have been preventing the plugin-side message parsing from ever working, since unparsable messages were silently assumed to be incomplete messages. So if the buffer from stdout looked like answer\n{"type": ...}
no messages would ever get parsed. This was one of the last bugs I found, but by then the offending print()
call was one I added.Am I understanding correctly that this would allow voice chat with matrix through weechat? Just trying to understand the scope of what this plugin can currently do.
Well yes and no, you answer a call through Weechat and then it spawns a separate program to do the call.
Ok, Interesting. Would this be more "behind the scenes" or would this other program be the main way of interacting with the call?
The other program should spawn a window where you could see the video stream of the other participant and have some controls to mute, hide/show your camera, and end the call.
This is awesome, I hadn't even considered video calling, only voice. Well I certainly am looking forward to this feature!
I saw the
webrtc
branch, but it seems to have not gotten much attention in a long while.I did some investigating, because I would really like to have calling working. My branch with some debug print statements is here: https://github.com/rexroni/weechat-matrix/tree/webrtc
I was able to determine why the gst webrtc object is failing by building gstreamer from source and running weechat from inside a gstreamer devenv, as described here: https://github.com/GStreamer/gst-build/blob/master/README.md
When I did that, I started to get error messages from gstreamer on stderr. I was able to determine that:
set-remote-description
event is sent, and in the plugin the_set_description_task
is getting queued here_set_description_task
is not actually getting called immediatelycreate-answer
event is sent, which results in a call to_create_answer_task
, which prints the error message saying"Asked to create an answer without a remote description"
here_set_description_task
is called at a later time.However, I must admit I'm a bit baffled as to just how the gst architecture works. For example: what is the purpose of interrupting promises from emitted events here?