using `twilio-video.js` in a pure nodejs application

[Sollimann]

• [x] I have verified that the issue occurs with the latest twilio-video.js release and is not marked as a known issue in the CHANGELOG.md.
• [x] I reviewed the Common Issues and open GitHub issues and verified that this report represents a potentially new issue.
• [ ] I verified that the Quickstart application works in my environment.
• [x] I am not sharing any Personally Identifiable Information (PII) or sensitive account information (API keys, credentials, etc.) when reporting this issue.

Problem Formulation

Hey, Twilio team. I'm working as a software engineer in the robotics team in Cognite. We're building an application to allow for fleet managements of robots industrial settings, and one of our use case is streaming video from robots and other such devices to one or multiple end users such that they can monitor and control the robot in real time. The video streaming is one way only, users will never stream video from their device, only consume video streams from available robots. We've had a look at the client side sdks that you provide for browsers (twilio-video.js) to better understand how the video connection is established which from our understanding is via signalling over a websocket connection between twilio and the client device. Our current roadblock is that there is no sdk/api available that is practical to run on a robot or on our servers to establish a webrtc video connection. We hope it is possible to find a solution where we can integrate towards twilio video that is outside the scope of your provided sdks and publicly available documentation.

Here is a visual diagram of our use case:

Questions

• I might have overseen something, but is it in fact possible to use the twilio-video.js sdk in a non-browser environment? In other words, can we use the sdk in a pure nodejs application running on an edge computer with wifi/4g connectivity?
• Are you planning on supporting languages other than javascript, swift or Kotlin. E.g in our case our backend/server is mainly written in Rust, components on the robots/edge are written in Python, and the dashboards for the users are written in Typescript w/react
• If we wanted to create our own rust-video-sdk, can you publish some more documentation on how to use/connect to your WebSocket API and Twilio SFU?
• Lastly, if the if it is in fact not possible to use the twilio-video.js in a non-browser environment and you do not plan to support any new languages in the near future. Would it be possible to run your kotlin/android sdk in an kotlin application on a Linux Ubuntu OS? Or does it have to be run on an Android device?

Thanks!

[Sollimann]

@seancoleman2 hey, Sean! Any updates on this?

[seancoleman2]

Hi @Sollimann, thanks for the exhaustive info and seems like a really interesting use case.

Q: I might have overseen something, but is it in fact possible to use the twilio-video.js sdk in a non-browser environment? In other words, can we use the sdk in a pure nodejs application running on an edge computer with wifi/4g connectivity? With the current product, no, this is not possible. The best bet at the moment would be using twilio-video.js in a headless browser.

Q: Are you planning on supporting languages other than javascript, swift or Kotlin. E.g in our case our backend/server is mainly written in Rust, components on the robots/edge are written in Python, and the dashboards for the users are written in Typescript w/react No current plans. We do have an example app using Typescript w/react if you have not seen that yet.

Q: If we wanted to create our own rust-video-sdk, can you publish some more documentation on how to use/connect to your WebSocket API and Twilio SFU? We don't have plans to make these available externally. We do provide the STUN/TURN infrastructure that powers Programmable Video via the Network Traversal Service, although I know this is not what you are asking for.

Q: Lastly, if the if it is in fact not possible to use the twilio-video.js in a non-browser environment and you do not plan to support any new languages in the near future. Would it be possible to run your kotlin/android sdk in an kotlin application on a Linux Ubuntu OS? Or does it have to be run on an Android device? Has to be run on an Android device.

Are you working with anyone from Twilio sales? I'd like to get these product gaps filed on our end for future prioritization.

[Sollimann]

Okey, thank you for the feedback! :)

We ended up writing our own twilio-client in Rust for your REST and WebSocket api for managing video rooms and signaling. We are a bit worried though that you might push breaking changes to your API's that will affect our unofficial Rust client and potentially break in our production environment.

What's Twilios' practices wrt to breaking changes in your API? Are they frequent? Do you support backward compatibility? And do you alert users somehow about major changes? @seancoleman2

[seancoleman2]

@Sollimann for the publicly documented REST API, you can expect there will be no breaking changes. If for some reason, we needed to make a breaking change we would notify all customers (via email) well in advance, but this has never happened in the past and we don't plan for this (would version the REST API for breaking changes).

The WebSocket API is more use at your own risk. Breaking changes are likely not going to happen anytime soon and if they do, we would need to release new versions of our SDKs to handle the changes, but there is no official API contract as it wasn’t really intended for external consumption. FWIW I believe the last breaking change here was in 2018 (we moved to a v2 Room Signaling Protocol).

[Sollimann]

@seancoleman2 Okey, thank you for the feedback!

[Marincor]

@Sollimann, I'm currently working on a similar use case, trying to enter as a participant inside Twilio-video through nodejs pure application, I'm very close to accomplishing using some mocks (to the browser modules), but still with some errors. Can you share some of your thoughts about what you made with your rust client?