rtcstats-server & jitsi-meet deployment issue

[AdrianPerezUMA]

Hello Jitsi community,

I'm deploying a setup where I have the jitsi-meet cloned in a VM (let's call VM1 with IP 10.11.23.213) and the rtcstats-server cloned in another VM (let's call VM2 with IP 10.11.23.219).

My idea is to use the GetStats approach and send some information from users to the server and later postprocess it. In the VM1, I've also installed the Jitsi-meet using this procedure, where I created a certificate using Let's Encrypt! tool. The reason why I have both Jitsi version (gitclone and apt-install) is because I started installing one version but I noticed that a new version was available via cloning the repo.

So far, I've:

• VM1 using the gitclone version
• VM2 with srcstats-server using gitclone version

Both VMs have connectivity and since it's a local deployment, there's no issues with firewall rules, open ports, etc. As PC to start the meeting, I'm using a w10 PC with connectivity to both VMs. It uses Chrome as browser. In order to use a secure connectivity, I copied and installed the certificate (the one created by Let's Encrypt in the VM1) in this PC.

I also modified the /srv/jitsi-meet/config.js file to send the WebRTC statistics to the srcstats server, but apparently the server doesn't receive anything:

To run the test:

1. I check jitsi-videobridge2 and gninx are running using the systemctl status command.
2. I run the server in the VM2 using npm run start.

3. In the w10 PC, I open Chrome, go to the VM1 IP and start the Jitsi meeting.

The issue: After start the meeting, I can see nothing in the rtcstats-server side (not sure if I should be able to see something in real time). I also do a inspect in the browser and there are some issues with the WebSocket connection to the VM2.

Some checks: After noticed about this issue, I did some tests to check the Websocket connectivity using wscat tool.

• First test: wscat -c wss://10.11.23.219:5555 with output: error: self-signed certificate
• Second test: wscat --no-check -c wss://10.11.23.219:5555 with no errors, where the rtcstats-server prompted:

The questions:

• I'd like to know if there's a way to see in real time the user parameter in the rtcstats-server command line.
• Do you know if there would be any issue related to certificates? how could I fix them so?
• Is it the correct setup to build jitsi-meet and srcstats-server?

Thank you in advance and hope you can help me here! :)

Best Regards, Adrian Perez

[saghul]

@andrei-gavrilescu can you help?

[andrei-gavrilescu]

Hey, it does look like there's a problem with the self signed certificate, wscat probably doesn't check the CA and that's why it gets through. On the other hand, when establishing the wss connection, the browser first opens a https connection and if the CA isn't installed on the machine where it's open it won't work unless you explicitly allow it. You can confirm this by accessing https://10.11.23.219:5555/healthcheck (a default rtcstats-server healthcheck endpoint), if a certificate error shows up in the browser then that's the problem. If you then allow it in the browser as an exception it will probably work for that session, but you'd need to do this every time. My recommendation is to use something like mkcert and install the CA cert on every device you want to access your env.

[AdrianPerezUMA]

Hi @andrei-gavrilescu, thank you very much for the answer. It was literally what It needed to work. I used mkcert to create the CA cert and copy-paste it to all the endpoints.

[AdrianPerezUMA]

Hello @andrei-gavrilescu,

I followed your recommendation about the usage of CA cert on every device but I'm still getting the same error: "self-signed certificate".

Basically what I did was:

1. Create a CA certificate using the mkcert tool in the VM2 (where the rtcstats server is) via mkcert 10.11.23.219. 1.1 It gave me the certificate as "./10.11.23.219.pem" and the key at "./10.11.23.219-key.pem".
2. Update the rtcstats server config (default.yaml) to use the new certificate and key files.
3. Copy the generated certificate (10.11.23.219.pem) into the endpoint which is going to start the Jitsi meeting (Windows PC)
4. Install the certificate as "Trusted Root Certification Authorities", also in Chrome.

I'm still getting the error in the endpoint chrome browser: I also tested it via command line, using wscat -c wss://10.11.23.219:5555 which gives me the error:

Useful information:

• I tried the trick you suggested about accessing to the https://10.11.23.219:5555/healthcheck and starting again the meeting. As you said, it temporally fix the issue with the wss communication.
• I tested creating the CA certificate with another tools like easy-rsa (suggested here) but still getting same issues...

Do you know what would be happening?

Thanks in advance

[AdrianPerezUMA]

Would anyone in the community know what could be going on? Does anyone in the community know what might be going on? Should I create a certificate in some specific way?

[andrei-gavrilescu]

@AdrianPerezUMA Hey, if the root certificate is properly installed it should work. I notice that you specified the used command as mkcert 10.11.23.219, where as the wss in the print screen is 10.11.23.213 maybe the problem is the host mismatch?

[AdrianPerezUMA]

Hello @andrei-gavrilescu, thank you for your answer!

Maybe I misunderstood how certificates work on this architecture... Shouldn't I create the certificate on the machine where the rtc-stats server is and then copy the .crt file to the endpoint? Here's what I'm currently doing: I'm creating a CA on the machine 10.11.23.219 (where the RTC server is located) and trying to use the WSS connection on the endpoint. Should I also copy the 10.11.23.219.crt file to the machine where the Jitsi deployment is (10.11.23.213)?

Thank you again :)

[andrei-gavrilescu]

So there are two parts to this problem, one's the Root CA which needs to be installed on all machines (VM) from which you want to access the rtcstats-server, you do this by using mkcert install if I rmmbr correctly, you can find more details here https://github.com/FiloSottile/mkcert#installing-the-ca-on-other-systems.

And the other is the actual cert/key pair that you use in rtcstats-server, it needs to be issued to the host on which rtcstats-server is hosted, in your case 10.11.23.219. The generated cert/key pair only needs to be on 10.11.23.29.

From the diagram/screenshot it looks like you have your jitsi deployment configured to access rtcstats-server at 10.11.23.213 but it is hosted on 10.11.23.219, try changing the config to use 10.11.23.219 for rtcstats.

Just to clarify because it looks like there's some confusion here, there isn't any connection from VM1 (Jitsi Deployment) to VM2 (rtcstats-server), the connection to VM2 is established directly from the browser (VM10) to VM2, it doesn't go through VM1.

[AdrianPerezUMA]

Hello @andrei-gavrilescu,

I finally managed to fix the issue by following the guide you provided. I was able to establish the WSS connection once I started the Jitsi meeting. However, I still encounter the "unable to verify the first certificate" error when running the wscat command. This command was intended to check the connection between both machines without starting a Jitsi meeting, so it’s a bit surprising that the WSS connection works in one scenario but not the other.

Thank you very much for your help! :)

[TuanTRAN-CUBoulder]

I spin up the rtcstats server, and I can see the connect and disconnect messages being sent to the server when a client joins or leaves the conference. However, there is no getstats message sent.

I double check with websocat wss://webrtcsfu.ddns.net:9010 and I can certainly send and receive messages.

Do you have any idea what may cause the getstats message not being sent? and yes I did change the config.js

Thank you so much for the help!

[andrei-gavrilescu]

Hey @TuanTRAN-CUBoulder, what do the browser client logs say? Do you see any errors regarding rtcstats? Check out what's being sent over the rtcstats websocket or you can even break point inside the rtcstats client lib and see where it breaks.

[TuanTRAN-CUBoulder]

I inspect the conference, and I don't see any errors regarding rtcstats. How can I check what being sent over the rtcstats websocket? I only receive connect and disconnect message on the websocket server. In config.js of jitsi-meet, there are 2 flags. Where rtcstatsLogs being stored on SFU machine?

rtcstatsEnabled: true, rtcstatsStoreLogs: true,

[andrei-gavrilescu]

You can check what messages are being sent over the rtcstats websocket (if any) using the 'Network' pane and filtering by websockets.