Jitsi Security and Persistent Room

[noahgibbs]

Right now, RWOT has an unsecured Jitsi install working in its rwot_stackscript.sh. However, Jitsi is going to need significant improvements:

• a secure domain (as described here: https://jitsi.github.io/handbook/docs/devops-guide/secure-domain)
• an XMPP client that can hold a persistent chatroom open as moderator

I have the secure domain work partially done on this branch: https://github.com/noahgibbs/SkotOS/tree/jitsi_security

There's at least another day of work left on this, and probably 2-3 days.

[noahgibbs]

Reference URLs:

• https://crosstalksolutions.com/how-to-install-jitsi-server-on-ubuntu-19-10/
• https://www.digitalocean.com/community/tutorials/how-to-install-jitsi-meet-on-debian-10

[noahgibbs]

Branch where I'm working on the SkotOS-side Jitsi setup: https://github.com/noahgibbs/SkotOS/tree/jitsi_security Branch it's intended to work with: https://github.com/noahgibbs/prototype_vRWOT/tree/use_latest_skotos

[noahgibbs]

Okay. There's a Jitsi script to set it up to use local LetsEncrypt certificates (install-letsencrypt-cert.sh) which is failing silently partway through, before it successfully does much of anything. It's failing whether I request my own LetsEncrypt cert or let it try to get its own (which it fails to do.)

As a result, it's not using that cert in any of the places it needs. I can have the stackscript fix up its NGinX files pretty easily, but I'm having a more interesting time finding all the Jitsi files that need to be updated.

The best, and (I hope) long-term solution would be to get a working version of this script. But that's a reminder that Jitsi will sometimes not work. It's just a complicated and unpredictable dependency. And so long term it may make more sense to wrap it in Docker.

Jitsi has a Docker config. It just looks thoroughly horrible to configure (official guide: https://jitsi.github.io/handbook/docs/devops-guide/devops-guide-docker).

In the mean time I'll try to figure out how to fix this script...

[noahgibbs]

Okay, so the only complicated bit of the turn server configuration is that when Certbot rotates certs (automatically requests a new one after awhile) then we need to update the turn server to also use that new one, with a hook script.

Also, it looks like the letsencrypt setup will always fail if re-run, so it needs to all work perfectly first time, or it will never be fixable (because set -e will cause it to die if it can't use sed to replace something it wants to replace.)

[noahgibbs]

Also looks like if we want to use the Jitsi LetsEncrypt setup script, we may need to be sure to generate a self-signed cert first rather than use an existing cert. Basically, a lot of this Jitsi config is really fragile.

[noahgibbs]

The certificate errors I'm getting in /var/log/jitsi/jicofo.log and /var/log/jitsi/jvb.log do not seem to be specific to securing the domain. It looks like the most recent Jitsi debian packages are broken even if not secured, and even when the Jitsi LetsEncrypt setup completes.

It looks like the Docker version is kept more up to date. I'll try to figure out how bad it would be to set that up with Linode.

[noahgibbs]

With the latest jitsi_security branch, I'm successfully installing Jitsi using Docker, with authentication, and the skotosadmin Jitsi user can log in -- and is the only one who can create rooms. So that all looks good. Next, XMPP.

[noahgibbs]

Huh. Nope, it's letting guests users create rooms with the external API, just not from the meet web UI. Okay, that's probably the next thing to look into.

[noahgibbs]

Ah, okay. Nope, it's working right - it just also seems to keep some amount of Jitsi information in a cookie. So if I use an incognito browser window (no shared cookies) and log in as a random SkotOS user it does not grant moderator rights where it shouldn't. That seems to only happen if I've logged in as skotosadmin on the same browser. So it still won't grant moderator rights to non-admins, unless they can share cookies with admin users (and skotosadmin should never actually log in with a browser in prod.)

[noahgibbs]

Not surprisingly the XMPP client library scene isn't that active - Google killed off XMPP integration with Google Chat back in 2015. Neither Ruby clients (my preference) nor PHP clients (matches thin-auth at least?) look great. There's an active, healthy documented Ruby client (https://github.com/adhearsion/blather) that uses EventMachine, so it's on top of a pretty unusual library. But the other XMPP libraries all look young, fragile and/or poorly maintained. So right now, Blather (part of the Adhearsion OSS telephony framework) looks like the way to go. Blather does not seem to pull in the whole Adhearsion framework, though it looks like it pulls in ActiveSupport (Rails' low-level utility/compatibility lib.) That's fine - it should just do that into its own one process, not anything larger, and ActiveSupport isn't too bad.

[noahgibbs]

It looks like BOSH and Websocket are the two protocols that exposed ports can speak, and that's what the Jitsi browser implementation uses. And it looks like I won't get much choice about XMPP clients - Adhearsion has dubious MUC (Multi-User Chat) support, for instance, and doesn't seem to support websocket as a transport protocol. I'm not finding clear evidence in its source code that it supports BOSH, either.

So: if the client code is going to be weird and dodgy, we might as well use the same client that Jitsi does internally. It's a JS client called Strophe. I'm seeing evidence in the GitHub issues for it that these days it's Node.js-compatible (as of around 2017), which is how we'd want to use it.

I'm definitely not crying about dropping Blather -- I didn't really want to introduce EventMachine as one more thing to learn. I wish Strophe were documented better, though.

[noahgibbs]

Oh hey, at some scale Jitsi will do sharding (automatically, it looks like?) which might require multiple XMPP connections to hold the room open? Not sure. But it looks like it will cheerfully support 250 users with no sharding, and that's far more than our Linode will scale to. Apparently it will have two shards with 1000 simultaneous users, so the border must be somewhere between 250 and 1000.

[noahgibbs]

I feel like I'm close, but missing something obvious. I've written a little Node.js program that uses Strophe and tries to connect to my test Jitsi server, but so far no dice. I'm trying to dissect the in-browser version a bit to get the necessary XMPP information, but also so far I haven't gotten the part I need (presumably.)

So far Strophe is not easy to get error codes out of. So I'm not particularly sure what's going wrong.

[noahgibbs]

Never got a response to my first community post, so I routed around it. The current Node.js issues are more specific, so I'm trying posting again on the Jitsi forums. Worth a try.

[noahgibbs]

One of the Jitsi developers pointed me at a Jitsi command-line load tester that I should be able to modify and repurpose. It definitely looks like a useful start - a lot of what it's for is to hold open a bunch of chatrooms.

[noahgibbs]

Created a repo and working on getting it up and running... https://github.com/ChatTheatre/SkotOS-jitsi-admin

Looking through the jitsi-meet and lib-jitsi-meet code a fair bit, trying to figure out what the connect-to-a-room flow looks like.

[noahgibbs]

Observation: normal-vs-incognito window is not enough for Jitsi to not share info between multiple Chrome windows, including stuff like display name.

But multiple browsers (Chrome vs Firefox) are. Good.

[noahgibbs]

Okay. JXS isn't working even with an unsecured Jitsi install (meet.testing-8.madrubyscience.com). I'm getting a different error when it's secured, but if it doesn't work before my modifications, I'm not at all confident about it working after. I've posted to the forum with some fairly specific questions, but I don't get the impression jxs is basically supported by anybody.

So, if no Jitsi means no RWOT, what's my next fallback? One possibility is Puppeteer (e.g. https://gist.github.com/saghul/179feba3df9f12ddf316decd0181b03e). This will run a local headless Chrome. I think I can use only one copy to hold multiple chatrooms open, so at least it will be one copy of Chrome instead of one per chatroom (which is a complete non-starter.)

This would have the advantage of being exactly how people normally use Jitsi, and the disadvantage of using more resources since it's running headless Chrome and doing it all in browser Javascript. There doesn't not seem to be any good, tested path to running Jitsi headless, though :-( I think I can run it without having to hook up any media devices. We'll see. But I'll need to, because AWS VMs don't have any media devices. But the response comment there looks promising, since a guy is running it in CI (a test server, normally headless.)

[noahgibbs]

Also: for iFrame usage here is a forum post with some settings: https://github.com/jitsi/jitsi-meet/issues/7999

Specifically, it recommends that headless usage of the jitsi-meet browser API set: "startWithAudioMuted=true startWithVideoMuted=true channelLastN=0 prejoinPageEnabled=false" and maybe startSilent=true.

[noahgibbs]

Ref bug report: https://community.jitsi.org/t/cannot-join-the-conference-created-by-lib-jitsi-meet/78813

Ref packaged headless Jitsi: https://github.com/r-oung/jitsi-headless

[noahgibbs]

Okay, nope. It turns out that the Jitsi web interface will only join one conference room, which means N rooms means N copies of Chrome. Nope nope nope. So then, time to figure out how to capture network traffic so that I can make jxs work (or if not jxs, then that same basic approach.)

Also, the problem with jxs and testing-8.madrubyscience.com is that apparently testing-8 is not configured with websockets, only BOSH (XMPP over HTTP). Which is weird, but apparently true.

[noahgibbs]

Okay. So, I have to use the jxs approach (multiple connections, headless app) but for that, I need to figure out what XML to send. And it's an extensive amount, because XMPP is verbose and complicated. But there is no headless client for Jitsi, and Puppeteer doesn't help since it's too expensive.

So: next step is to log the websocket traffic from a connection doing what I want it to and then adapt that to work with jxs.

Also: right now jxs is really hard to debug because it's using a bizarre syntax added by webpacker to send its xml. Which is cute, but also means everything has to be debugged in packed/uglified JS, which is horrible.

So I'll log-and-decrypt the XMPP traffic while doing what I need to do in the Jitsi web UI as an admin. I'll write a tutorial about how to do that, because somebody (possibly me) will need to be able to add new actions to this server in the future (e.g. kicking somebody.) And then I'll adapt the XML to JXS, after removing this bizarre XML syntax that JXS currently uses so that I can get a normal stack trace. Debugging webpack'd code is a bad idea if it can be avoided, and it can.

Logging and decrypting the XMPP traffic means using a tool like SSLsplit or MITMProxy. So far MITMProxy has extremely friendly-looking but extremely useless documentation. So I think this is going to happen with SSLsplit and be Mac-only (for logging the traffic, not running the admin server.)

[noahgibbs]

If we wind up needing to do network capture on the server side: https://byteplumbing.net/2018/01/inspecting-docker-container-network-traffic/

That looks like it's using TCPDump, though - so that will be packet-based capture, not stream-based capture. So if I'm understanding correctly, that would add a step to figuring out what Jitsi is sending.

[noahgibbs]

Okay. Turns out Chrome will dump its full websocket traffic with a little coaxing. That's good, and I have some packet dumps.

So I need to figure out how to connect jxs-or-similar from the packet dumps.

What's odd is that when I search for "skotosadmin" (the account name), it is always received by the browser, not sent, for all the early exchanges. To put it another way: it does not appear that the login, where the client specifies that they are skotosadmin, occurs on the websocket connection at all. I'll try inspecting the POST where I log into the server. So I guess that's how it's being done?

But that's awkward if jxs is using a websocket connection, and the login isn't sent on the websocket connection.

I'll keep at it :-(

[noahgibbs]

Firefox does not seem to save websocket content in the HAR file like Chrome does, despite having nearly the same inspector. It saves the websocket requests, but not the content.

There does not seem to be a POST request when I enter the xmpp credentials, so that is not how it's matching it up. Maybe it does send something over the websocket but not what I think? I'll read through more of the Jitsi JS code again...

[noahgibbs]

Okay. So, XMPP supports several auth methods via SASL, a generalised modular pluggable authentication standard (everything in XMPP seems to be generalised and pluggable to the point that it's nearly unreadable.) It looks like SASL SCRAM-SHA-1 is the one used by Jitsi (based on Googling SCRAM-SHA-1 and Jitsi and finding bug reports.) That makes sense. So perhaps its passing a hash, but never passing the username plaintext anywhere?

Huh. Looking through the packet capture, I'm seeing SASL ANONYMOUS but no other SASL mechanism xml tags. So maybe it's not using SASL login at all? That's weird but possible.

[noahgibbs]

One possible worthwhile thing here... This is using the identity "skotosadmin@meet.jitsi", which is not what I've been using. And the room "newroom@muc.meet.jitsi", which is also not what I've been using. So maybe I have the Jitsi domain wrong and that's causing 403s? Worth a shot.

[noahgibbs]

Okay... Just updating the domain to meet.jitsi and logging in directly isn't working. Presumably that's a +1 on "this isn't actually using XMPP login in the normal way."

[noahgibbs]

Hm. Yeah, in the packet capture it's logging into "guest.meet.jitsi", so it's specifically an anon on a guest domain. We're getting a 403 because we're trying to log into the normal domain with my modified jxs.

Okay, so if I clear all my local storage data in Chrome and try again, mostly I'm seeing the same thing. Guest login, good. Service discovery, fine. Lots of "let me into the room" / "no room for you!" exchanges. Sure.

And then after one of the exchanges, the response from the server is different. Here's what the request and response look like:

          {
            "type": "send",
            "time": 1620740975.7583568,
            "opcode": 1,
            "data": "<iq id=\"7fc0efca-fccb-453f-b147-1f9b665e9d01:sendIQ\" to=\"focus.meet.jitsi\" type=\"set\" xmlns=\"jabber:client\"><conference machine-uid=\"9b579fcc6c716dff65a8032533245675\" room=\"bogoroom@muc.meet.jitsi\" session-id=\"53df2452-4508-412a-95c0-1333e3497bb7\" xmlns=\"http://jitsi.org/protocol/focus\"><property name=\"disableRtx\" value=\"false\"/><property name=\"startAudioMuted\" value=\"10\"/><property name=\"startVideoMuted\" value=\"10\"/><property name=\"stereo\" value=\"false\"/></conference></iq>"
          },

          {
            "type": "receive",
            "time": 1620740975.7921498,
            "opcode": 1,
            "data": "<iq xmlns='jabber:client' from='focus.meet.jitsi' to='9412d117-a564-4ae0-afe5-56d61a2e9369@guest.meet.jitsi/AZDRbt70' type='result' id='7fc0efca
-fccb-453f-b147-1f9b665e9d01:sendIQ'><conference xmlns='http://jitsi.org/protocol/focus' identity='skotosadmin@meet.jitsi' focusjid='focus@auth.meet.jitsi' session-i
d='53df2452-4508-412a-95c0-1333e3497bb7' ready='true' room='bogoroom@muc.meet.jitsi'><property name='authentication' value='true'/><property name='externalAuth' valu
e='false'/></conference></iq>"
          },

The request is unremarkable - there are a lot like it before that, and they all get a "go away" response. The session-id is echoed from the request to the response, so that's where that comes from. The response says auth true and externalAuth false. But I didn't see any new auth information in this request that wasn't in other requests.

But weirdly there is a second websocket connection open at the same time. It's to meet.jitsi instead of guest.meet.jitsi. It uses SCRAM-SHA-1 (hashed auth) instead of ANONYMOUS auth. It also receives info about skotosadmin long before it ever sends that string. But it does send some hashed info first, which might conceivably contain the userID somehow.

That seems weird - shouldn't you include a simple method for the server to find out what account you're logging in as? But it does not seem impossible, which is what I've been seeing up until now. So: that's a step forward. Now I just need to learn about SASL's SCRAM-SHA-1 auth method and how XMPP wraps it (sigh).

Also: while I haven't excerpted it here (these packet dumps are huge,) it looks like the non-anonymous exchange is establishing the session ID and then the anonymous connection is supplying that session ID, which is when it finally gets in successfully. So: the session ID is how the two connections are linked to each other.

Here's the SCRAM-SHA-1 exchange, which is still pretty opaque to me:

          {
            "type": "send",
            "time": 1620740975.34331,
            "opcode": 1,
            "data": "<open to=\"meet.jitsi\" version=\"1.0\" xmlns=\"urn:ietf:params:xml:ns:xmpp-framing\"/>"
          },
          {
            "type": "receive",
            "time": 1620740975.376384,
            "opcode": 1,
            "data": "<open xmlns='urn:ietf:params:xml:ns:xmpp-framing' from='meet.jitsi' version='1.0' xml:lang='en' id='997790be-32f5-4abb-b87d-653359003dc9'/>"
          },
          {
            "type": "receive",
            "time": 1620740975.376816,
            "opcode": 1,
            "data": "<stream:features xmlns='jabber:client' xmlns:stream='http://etherx.jabber.org/streams'><mechanisms xmlns='urn:ietf:params:xml:ns:xmpp-sasl'><mec
hanism>SCRAM-SHA-1</mechanism></mechanisms></stream:features>"
          },
          {
            "type": "send",
            "time": 1620740975.3779662,
            "opcode": 1,
            "data": "<auth mechanism=\"SCRAM-SHA-1\" xmlns=\"urn:ietf:params:xml:ns:xmpp-sasl\">biwsbj1za290b3NhZG1pbixyPWNiMjI4ODg4YWI0Mjk2ZGE5MTFiMDA5OWJjYWYyODRm<
/auth>"
          },
          {
            "type": "receive",
            "time": 1620740975.408884,
            "opcode": 1,
            "data": "<challenge xmlns='urn:ietf:params:xml:ns:xmpp-sasl'>cj1jYjIyODg4OGFiNDI5NmRhOTExYjAwOTliY2FmMjg0ZjM3MTRjZDUyLThhZWYtNDg5NS04ZjUyLTBmMDQyOTUzYTBm
NCxzPU1qUTBNMlF5TW1ZdFl6bGhaQzAwTldFMExUZ3hNR1V0WVdKaE5tUTFPVEF4WlRreSxpPTQwOTY=</challenge>"
          },
          {
            "type": "send",
            "time": 1620740975.457251,
            "opcode": 1,
            "data": "<response xmlns=\"urn:ietf:params:xml:ns:xmpp-sasl\">Yz1iaXdzLHI9Y2IyMjg4ODhhYjQyOTZkYTkxMWIwMDk5YmNhZjI4NGYzNzE0Y2Q1Mi04YWVmLTQ4OTUtOGY1Mi0wZjA
0Mjk1M2EwZjQscD1XaFBQbnp1dzBCZ0Y5NEhrVHZuK0ZUTzVuZGc9</response>"
          },
          {
            "type": "receive",
            "time": 1620740975.4920852,
            "opcode": 1,
            "data": "<success xmlns='urn:ietf:params:xml:ns:xmpp-sasl'>dj11TEdCRjdINk9pOGdYQnp5cVc2MUFEMTdWVWc9</success>"
          },
          {
            "type": "send",
            "time": 1620740975.4924982,
            "opcode": 1,
            "data": "<open to=\"meet.jitsi\" version=\"1.0\" xmlns=\"urn:ietf:params:xml:ns:xmpp-framing\"/>"
          },
          {
            "type": "receive",
            "time": 1620740975.5256102,
            "opcode": 1,
            "data": "<open xmlns='urn:ietf:params:xml:ns:xmpp-framing' from='meet.jitsi' version='1.0' xml:lang='en' id='00e32b48-ef0e-4bef-b02f-7373bc36f46a'/>"
          },

[noahgibbs]

Ah, okay, there's the trick. It's base64 encoded. If I base64 decode that first lump of line noise it's a lot clearer:

encoded: biwsbj1za290b3NhZG1pbixyPWNiMjI4ODg4YWI0Mjk2ZGE5MTFiMDA5OWJjYWYyODRm
decoded: n,,n=skotosadmin,r=cb228888ab4296da911b0099bcaf284f

So that's what's going on. It's sending the ID, but it's sending it base64 encoded.

[noahgibbs]

At this point, I don't think it's reasonable to get this approach working in time. I'm trying an end-run where I build in WebRTC more directly, because the XMPP protocol is simply too elaborate, and working around Jitsi's security model at every turn is simply not working. That's going to have some instability - WebRTC is a young protocol, browsers change, and even a lot of the official WebRTC demo code is broken, including for the "look, we'll smooth over the browser changes" adapter/polyfill libraries. But with more experience with Jitsi, I'm no longer confident they'll do a better job of that.

So: more directly into WebRTC I go. This should at least vastly reduce the complexity of the final product... if it works.

[noahgibbs]

I have a basic WebRTC chat server up, and the things I'm checking all look reasonable (yes, we can pick output devices; yes we can keep multiple peer connections; yes we can kick/disconnect users from the server.)

The big down-side of ditching Jitsi, and specifically Jitsi-videobridge, is that our scaling behaviour will be WebRTC baseline. In other words, if we wanted a chatroom where one staff member was speaking to forty attendees, they would be making forty simultaneous WebRTC connections.

That may be a non-starter. We should talk about this during the meeting today.

[noahgibbs]

Feedback from the meeting:

• I'll write up my experience trying to implement this model. This is already mostly done in a PR to SkotOS-Doc (https://github.com/ChatTheatre/SkotOS-Doc/pull/43)
• I'll write up the current/near-future security model, including its good and bad points
• I'll work on Jitsi/SkotOS feature-enablement with most of the remaining time
• We hope to have an NGinX plugin that will allow us to ban/kick people from their Jitsi and SkotOS accounts; this will allow a SkotOS ban to also kick from Jitsi, but not prevent abuse of the Jitsi API by people with a valid account

[noahgibbs]

I should record notes here, not just in Signal.

Docker doesn't include TURN config already. I'm copying and modifying the Docker config and copying it into a directory under SkotOS/deploy_scripts/stackscript. That will also let me expose the raw TCP XMPP port to the local host so we can connect to it.

I'm attempting to install a TURN server on the host directly via the jitsi-meet-turnserver Debian package. That also installs a modified-for-Jitsi prosody server automatically (listed as required, not recommended, so installation can't be turned off.) So I'm installing that and then deactivating it, but I don't want a fight over port 5222 (the XMPP TCP port), so the real (Docker) XMPP port has to be exposed as port 5223.

Turning off the prosody server is a matter of "/etc/init.d/prosody stop" and then "rm -f /etc/rc*.d/S01prosody" to prevent restart.

Once all this is done, in theory that should make direct connection via the XMPP port workable. I should be able to modify the Docker-Compose config on testing-14 and re-run to expose the XMPP port and make sure the scripts can successfully connect. But that seems like it should be reasonable. Certainly the various JS clients seem to have a lot of trouble with BOSH and websockets that would presumably be skipped by this method.

[noahgibbs]

Okay, got a Docker install that appears to work (have only checked the very basics), and that exposes the XMPP server on port 5223.

[noahgibbs]

I can use the old modified-from-jxs SkotOS-jitsi-admin code to begin connecting to port 5223. I have to set "NODE_TLS_REJECT_UNAUTHORIZED=0" as an environment variable to permit self-signed certs. It then gets a SASL (login/auth) failure saying it's using an invalid mechanism. So that's possibly a step up. I'll keep at it.

[noahgibbs]

Jitsi uses SCRAM-SHA-1 as its SASL mechanism, and xmpp.js (a.k.a @xmpp) supports that out-of-the-box via @xmpp/sasl-scram-sha-1. So I'm pretty sure it's not an actual invalid or mismatched SASL mechanism.

[noahgibbs]

cgbot thinks it can connect now -- or at least, it doesn't give an error message saying that it can't. It doesn't discernibly react differently to meetings that exist from meetings that don't exist. But it does react differently to a domain and user that exist from a domain and user that don't, so it's clearly connecting to XMPP even if I'm not sure it can see rooms/meetings.

[noahgibbs]

Yeah, cgbot appears to never get the 'online' event which would mean it was connected and would see notifications. So: no error on connection, but also no useful information from the server.

[noahgibbs]

Ah, never mind! It is getting the "online" event, but I'm connecting from a different machine and prosody is configured to not send anything, and that was hidden in a debug message that doesn't get displayed anywhere. Retrying from localhost.

[noahgibbs]

Hm. Nope, it turns out that the "Communication with remote domains is not enabled" error happens on localhost too. Doing some Googling, it looks like that's a Jitsi error, not an XMPP error. So it seems like the XMPP part is working fine, and now it's Jitsi that's not playing ball. So now I need to see how much Jitsi-specific XMPP interaction I can reverse-engineer.

[noahgibbs]

The message "Communication with remote domains is not enabled" doesn't seem to occur as a literal string in any obvious places in the server-side code. So: still no clue where that's coming from. Probably time to go back to looking at lib-jitsi-meet and packet dumps again to try to figure out how the client side is connecting. The method shown in the packet dumps is pretty ugly if done exactly the same way.

[jandrieu]

That looks like a CORS error. Who is reporting that message?

[noahgibbs]

I'm getting it wrapped in XML as an XMPP presence message:

<presence to="skotosadmin@meet.jitsi/ezVeQDGK" from="rwotlobby@conference.meet.jitsi/cgbot" id="5727d0c8-b5f1-40c1-a823-c1e27d2d9711" type="error">
  <error type="cancel">
    <not-allowed xmlns="urn:ietf:params:xml:ns:xmpp-stanzas"/>
    <text xmlns="urn:ietf:params:xml:ns:xmpp-stanzas">
      Communication with remote domains is not enabled
    </text>
  </error>
</presence>

However, it looks like I'm working around it - it seems to be Jitsi basically saying, "hey, Jicofo doesn't recognize you."

[noahgibbs]

I now have the SkotOS-jitsi-admin code connecting as far as cgbot did, and authenticating. And I'm just basically working through the errors message by message. So far that's being more productive than trying to locate particular responses in the Jitsi server code.

[noahgibbs]

Right now I'm chasing down how the sessionId gets set. It's clear that it's established first and then sent by the browser/skotos-jitsi-admin to jicofo, and without the sessionId the request fails. Looking in lib-jitsi-meet (the browser-side jitsi client library), the sessionId comes from localstorage, so it's getting set somewhere else. I'm not finding a packet that mentions session-id / sessionId in an obviously promising way -- I'll keep trying to track down where it comes from.

[noahgibbs]

Notes to myself for later:

In hostandmoderator-clean-meet (one of the packet captures), there are five packets that mention the session-id that gets established (53df2452-4508-412a-95c0-1333e3497bb7). The earliest-timestamped (last in file) of the five is a receive from Jitsi, not a send (good), so that's how it gets set up. That message chain (8cf654bb-8064-4540-bdf6-f3d291de8265) does include a Jicofo invitation with no session-id (usually they fail without that) and it receives the session-id in return.

Here's the magic message that is sent without a session-id and receives the session-id for the first time in return:

<iq id="8cf654bb-8064-4540-bdf6-f3d291de8265:sendIQ" to="focus.meet.jitsi" type="set" xmlns="jabber:client">
    <conference machine-uid="9b579fcc6c716dff65a8032533245675" room="bogoroom@muc.meet.jitsi" xmlns="http://jitsi.org/protocol/focus">
        <property name="disableRtx" value="false"/>
        <property name="startAudioMuted" value="10"/>
        <property name="startVideoMuted" value="10"/>
        <property name="stereo" value="false"/>
</conference></iq>

That's really close to the Jicofo invitation we're sending in SkotOS-jitsi-admin that gets no response (not even an error.) So: it doesn't appear that the magic of getting a session-id successfully is in that message. It also doesn't seem to be in that ID for the message chain -- tracing it back, there's basically only that exchange.

But the response to that message is how the in-browser client receives its session-id. So now it's a matter of figuring out why Jitsi likes this message that one time, but hates it a bunch of other times. The answer does not appear to be contained within the message itself, nor its ID-identified chain of messages.

[noahgibbs]

Interestingly, the machine-uid changes once in that file (d056eb88ba7ae4b6eaac110113833a86 vs 9b579fcc6c716dff65a8032533245675). So it may be set up per-connection. But there's nothing that obviously looks like the session-id is tied to the machine-uid. For instance, previous "open a room" exchanges with Jicofo from that same machine-uid are rejected. So I don't think it's the machine-uid.

[noahgibbs]

The Java-and-Kotlin code for Jicofo is only checking the machine-uid, the identity (email address / account name) and the room as far as I can tell. So maybe the machine-uid is more of the secret than I've realised and I just haven't figured out how yet. The session clearly starts with that machine-uid being bad (it gets told no.) And I don't clearly see anything that would act like a "machine UID login" happening.

But maybe the machine-uid is the key somehow anyway.

[noahgibbs]

We've had a lot of Jitsi progress of various kinds and I won't record all of it here. However, several things that helped:

The token_moderation plugin, when used with jwt login, allows users to be authenticated as themselves and enter rooms with no moderators, but not become moderators: https://github.com/nvonahsen/jitsi-token-moderation-plugin/blob/master/mod_token_moderation.lua (there's also a "token_affiliation" plugin that folks recommended as better, but it doesn't look like it has any new functionality we want or care about.)

That means we'll still want a server process, but that server process doesn't need to maintain a connection to every open room, or even connect to each room to create it. Handing out the tokens is enough, provided they're properly restricted.

We want to be able to restrict which rooms/channels users can connect to, especially because it's easy for users to use JWT tokens to connect via the Jitsi-meet interface. While there's clearly some kind of regex functionality for which rooms a JWT token is valid for (https://github.com/jitsi/jitsi-meet/blob/master/resources/prosody-plugins/token/util.lib.lua#L386), I don't think we want to play games where the name of the channel has to match a particular regex. I think it makes far more sense to just hand each user a separate JWT token for each channel they connect to. It is going to be hard for us to guarantee that the user has disconnected from a channel that we don't want them in. Once we have the server properly connected to Jitsi, I can look into auto-kicking them after a timeout if they're still there. Though I think we can assign fairly short timeouts to tokens so that they're not usable for very long. And I don't think Jitsi will kick somebody after their token expires, so I think we can treat tokens as effectively just for connecting/reconnecting, not for hanging out on a channel you're already connected to.

So: I am no longer attempting to reverse-engineer the XMPP protocol. That's good, because the Jitsi devs definitely don't consider that to be supported.

It doesn't look like using Node.js outside the browser is supported for lib-jitsi-meet, the lower-level build-your-own UI Jitsi lib) either, see dev reply: https://community.jitsi.org/t/connecting-to-lib-jitsi-meet-jitsi-meet-api-through-nodejs/88914. That makes sense, because it requires WebRTC. They suggest using a framebuffer-based headless Chrome, which is what jibri (Jitsi BRoadcasting) does for recording and streaming.

That can work for either lib-jitsi-meet or jitsi-meet, which also has some UI. So it probably makes more sense to use jitsi-meet, especially because we can probably grab its lib-jitsi-meet components in JS after it has them set up.

The reason we couldn't just do everything in headless Chrome before was that jitsi-meet really only allows connecting to one room at a time. But if we have a moderator account that only needs to connect to a room for an occasional half-second or so to kick people out of rooms, that's entirely doable. One headless chrome is fine, it's hundreds that we can't handle.