Not being able to see others video/audio

jakobkilian commented 3 years ago

Hi all and a big thx first: Galene works great for our team – awesome work.

There is one bug that bothers a little: Sometimes participants can't join the conversation. They appear in the sidebar, are able to click the ready button and even see their own video – but not the others' video/audio. From their perspective, all the others (despite being visible in the sidebar) didn't join the conversation yet. Meanwhile the others can see each other and even hear and see the person with the bug

The weird thing is the irregularity of this:

most of the users with this bug use Firefox or Safari (we, therefore, agreed on using Chromium browsers internally. But it can be difficult to explain that to external guests that only join once).
But unfortunately, it even happened to people using latest Google Chrome browsers.
Today it even happened to me using a Mac, latest Google Chrome with Internet from a Smartphone LTE Hotspot
Often it helps to restart the browser and or open the link in a private window, but sometimes not...

What makes the bug a problem is, that you cannot see that it is actually you having the bug. It just seems that the rest of the team did not join yet. Chat works btw.

We host Galene as part of a Yunohost installation on a VPS. Everything is up to date.

Tell me if I can provide any logs or more information. I am not so deep into programming etc. but I just thought it is good to report the issue.

erdnaxe commented 3 years ago

I observed this issue when I was on the same local network as my TURN server. Can you try disabling the TURN (./galene -turn "") and see if the issue persist?

jakobkilian commented 3 years ago

I just tried to understand what TURN does and how it is relevant to me. Unfortunately I don't really understand. I never configured it but turned it off for now as proposed. But: Actually I found a way to reproduce the issue (at least for firefox): With TURN turned off, picture 1 shows how it looks like when you enter the session with Firefox: empty boxes and no sound. Picture 2 Shows the same Session in Vivaldi (Chromium Browser) with all boxes (including FF there). Safari also worked flawlessly in this case. Now here is the interesting part: when a user is reentering the session after FF joined, FF can see him:her. Also when a user switches to blackboard Mode (and back), the user gets visible for FF...

Hope this helps?

jech commented 3 years ago

Could you please show the output of /stats in the problematic case? (/stats.html in the 0.4 branch.)

What happens when the user who doesn't see the other types /renegotiate?

What happens when the sending user types /renegotiate?

jakobkilian commented 3 years ago

I'd love to, but I don't now where to get the /stats output. Can't use it in the chat neither do I find it in the galene directory or on my server. Is it maybe because it just got added yesterday? As I use it as part of yunohost (galene v0.3.4) I can't add it easily. Sorry for opening up the issue here if it is the wrong place.

I tried '/renegotiate' in all of the following situations and with all users – with no change at all: I tried to reproduce the issue, but at first everything worked, even with FF. The only thing that I can always reproduce is when using LTE, where the other users never get visible in my case, no matter which browser or which device (phone, mac, ...)

signal-2021-05-01-214550

Participants have no audio/video, only black boxes

While trying to reproduce the issue on a laptop (with Wifi) and reloading the page several times the even worse variant of the bug happened (so far it was only reported to me): Suddenly all participants were only able to see themselves. Even the boxes of other users disappeared as you can see in the screenshot. As mentioned '/renegotiate' didn't help, neither did clicking 'panic' and reentering with 'ready'. The only way to solve this was to leave the room with all user and starting it all over again.

This is how all of the 4 user's views looked like: Participants aren't visible at all

EDIT: I am so sorry, the photo shows a session that has not yet joined (green "ready" button), but It looked the same when being joined...

jech commented 3 years ago

The stats entry is at the root of the Galène server, https://galene.example.org:8443/stats. It's called stats.html since yesterday, but the old stats file has existed since the beginning.

jakobkilian commented 3 years ago

Ok, got it, sry. Here are the stats. 1 user Chromium, 1 user Firefox, 1 user (2nd i guess) mobile on LTE:

05a74d82e05398c8a4903674fee78865
  | Up | 2ea078b8034ab35e03f83eea0beff4bb
  |   |   | 5992 | 0% | ±2.145799ms
  |   |   | 626520 | 0% | ±7.377704ms
  | Down | d38e7948ab15f8cc9807dc396206f78b | 777312
  |   |   | 18880/512000 | 0% | 23.43282ms±2.229166ms
  |   |   | 664312/840215 | 0% | 22.253621ms±26.555555ms

6e41ab7ff263483c116bed19e66f9c1c
  | Up | bd0b3f4dbc984fb91b522c0f190e2f68
  | Down | 2ea078b8034ab35e03f83eea0beff4bb | 655360
  |   |   | 0 | 0% |  
  |   |   | 0 | 0% |  
  | Down | d38e7948ab15f8cc9807dc396206f78b | 655360
  |   |   | 0 | 0% |  
  |   |   | 0 | 0% |  

df216dde9fb01c7f9c2c79907f568369
  | Up | d38e7948ab15f8cc9807dc396206f78b
  |   |   | 18888 | 0% | ±2.166632ms
  |   |   | 691968 | 0% | ±7.466592ms
  | Down | 2ea078b8034ab35e03f83eea0beff4bb | 1067888
  |   |   | 6024/512000 | 0% | 21.844278ms±3.125ms
  |   |   | 609728/799610 | 0% | 22.119913ms±22.488888ms

EDIT: I try to fetch the stats and dates about browser etc. when the issue appears next (without LTE)

jech commented 3 years ago

Thanks, that's interesting. Galène is getting the traffic from the sender, it has negotiated the streams towards the receivers, but it's not sending any actual data to one of the receivers.

If you get a chance, please show:

the output of /relay-test in Firefox;
anything that's in the Firefox console (Ctrl-k); and
the ICE table at about:webrtc

Thanks again for your help.

jakobkilian commented 3 years ago

I am happy to help if I can ;-) So currently I can reproduce the behaviour by just using the mobile internet (LTE) connection with my laptop. It fails with all browsers.

'/relay-test' always fails on all devices, browsers and internet connections
The console returns (myurl.org is a placeholder for the real url):

Vivaldi Console

protocol.js:238 [Report Only] Refused to connect to 'wss://myurl.org/ws' because it violates the following Content Security Policy directive: "default-src https: data: 'unsafe-inline' 'unsafe-eval'". Note that 'connect-src' was not explicitly set, so 'default-src' is used as a fallback.

ServerConnection.connect @ protocol.js:238
3x protocol.js:747 Uncaught Error: unknown down stream
    at ServerConnection.gotClose (protocol.js:747)
    at WebSocket.socket.onmessage (protocol.js:287)
galene.js:1562 Uncaught (in promise) DOMException: The play() request was interrupted by a new load request. https://goo.gl/LdLk22
888x Unchecked runtime.lastError: The message port closed before a response was received.
DevTools failed to load SourceMap: Could not load content for https://myurl.org/sm/1df7b098cd6209fd67b5cc8f6f6518b79e5214ec3802d91f56f825883253df69.map: HTTP error: status code 404, net::ERR_HTTP_RESPONSE_CODE_FAILURE
DevTools failed to load SourceMap: Could not load content for https://myurl.org/sm/9c0bbf2acc17f6468f9dd75307f4d772b55e466d0ddceef6dc95ee31ca309918.map: HTTP error: status code 404, net::ERR_HTTP_RESPONSE_CODE_FAILURE

Firefox Console

Content Security Policy: This site (https://myurl.org) has a Report-Only policy without a report URI. CSP will not block and cannot report violations of this policy. 
Content Security Policy: The page's settings observed the loading of a resource at wss://myurl.org/ws ("default-src"). A CSP report is being sent. protocol.js:238:16
Content Security Policy: This site (https://myurl.org) has a Report-Only policy without a report URI. CSP will not block and cannot report violations of this policy. 
Content Security Policy: The page's settings observed the loading of a resource at wss://myurl.org/ws ("default-src"). A CSP report is being sent. 
WebRTC: ICE failed, add a STUN server and see about:webrtc for more details

about:webrtc

I don't know what you need exactly, do you mean this?

about webrtc

I am starting to think, that I should learn more about TURN, ICE and these things. Never touched these things :-/

jech commented 3 years ago

protocol.js:238 [Report Only] Refused to connect to 'wss://myurl.org/ws' because it violates the following Content Security Policy directive: "default-src https: data: 'unsafe-inline' 'unsafe-eval'". Note that 'connect-src' was not explicitly set, so 'default-src' is used as a fallback.

Are you using a proxy that's rewriting the Content-Security-Policy header? If so, you should not do that — the CSP header in Galène is carefully tailored to work with all three major browsers, please do not modify it.

I don't know what you need exactly, do you mean this?

Yes, but I'd need the full list.

I suspect that your TURN server is not accessible from the LTE network. Can you please do

nc -v myurl.org 1194

and see if it reports "open"? If it doesn't, you'll need to find a port for your TURN server that the LTE network doesn't block.

jakobkilian commented 3 years ago

Are you using a proxy that's rewriting the Content-Security-Policy header?

No, at least not knowingly. I just use the bare installation from Yunohost, but maybe I will post it there and they have an idea what it could mean...

The following information refers to a client, using mobile network. This is the only way I can reproduce the issue right now. I asked a few friends, they can also reproduce it by using mobile network

Yes, but I'd need the full list.

I sent it to you by mail, not sure if it makes sense to post all the information...

... and see if it reports "open"? ...

This is the output.

found 0 associations
found 1 connections:
     1: flags=82<CONNECTED,PREFERRED>
    outif en1
    src 192.168.43.95 port 62133
    dst ip.add.of.server port 1194
    rank info not available
    TCP aux info available

Connection to myurl.org port 1194 [tcp/openvpn] succeeded!

jech commented 3 years ago

From the Firefox data you sent me by mail, it appears that all candidate pairs have failed. There are failed candidates of type relayed-tcp, which means that something goes wrong with TURN — this means that either (i) the TURN server is not accessible from the client, or (ii) the TURN server is unable to exchange UDP traffic with the Galène server.

The fact that /relay-test fails indicates that the client is unable to reach the TURN server. However, the test you made in https://github.com/jech/galene/issues/86#issuecomment-830781328 indicates that the TURN server is accessible. These two results seem to contradict each other — I'm at a loss.

jakobkilian commented 3 years ago

Thanks for your explanation. I just double-checked the reproduction with the same result. Netcat succeeds on mobile, but it cannot join with the very same connection. Please tell me, if I can help with anything else! I will try to understand what the TURN server does in my case and how it is maintained within yunohost and will keep you posted if there is anything new...

jakobkilian commented 3 years ago

Just wanted to add: Over at Yunohost there was a similar discussion (mobile network access). As mentioned I will try to understand my own configuration first.

jakobkilian commented 3 years ago

Ok, I solved at least part of the issue (at least the reproducible one – no access with mobile network) by sudo yunohost firewall allow Both 49152:65535 Somehow I overread that these ports need to be opened manually. If it is of any interest: the /relay-teststill fails... I will continue to monitor the behavior, but think that this was it...

jech commented 3 years ago

You had two distinct issues, and you only solved one.

The high ports were inaccessible, so the client was falling back to contacting the TURN server. Now the high ports are accessible, but the TURN server is still not functional — this means that your server will be unusable by anyone who's on a network that doesn't allow access to the high ports, which includes many enterprise and university networks.

I strongly recommend fixing the TURN issue (i.e. working out why /relay-test fails).

jakobkilian commented 3 years ago

solved by installing another branch of galene for Yunohost involving a own turn server: https://github.com/YunoHost-Apps/galene_ynh/issues/43#issuecomment-846627431

jech / galene