Using stunner to achieve "deploying 1 coturn server per service" in K8s

[cclo-astri]

Hi all,

I am now working on a K8s cluster service which will dynamically deploy k8s services (with related Pod, containers, ingress, etc) via a customized centralized python app with K8s python client API. Now user device could simply send a request to cluster's master IP and an unique K8s service will be created and dedicated to serve that user device. The service and related deployment will be teared down after use.

i.e. https://<cluster_master_ip>/<unique_service_name>/.... [Lots of RESTful APIs]

Backgrounds:

• Further more, the cluster_master_ip will be further routed from a public domain in order to allow user access the service outside the cluster subnet.
• 1 node of K8s may consist of several such service.
• No node machines will expose to public network. Only cluster master.

Under this circumstance, a new sub-feature will be added. We need to let user creating a webRTC peer connection from its own Chrome browser (PC) for continuous video / image sharing towards a COTURN server. The user device also connected to this COTURN as a webRTC peer. The image will now able to be sent from PC Chrome browser to user device. Then the application on user device could further process the video / image for some UX.

As The COTURN server will only serve this single p2p connection so that we want to embed the COTURN into the our runtime-created k8s service Pod. And we would like to tear down all related resources (ingress, Pod, services including COTURN) after use.

Together with above backgrounds, is it possible to access the COTURN inside that runtime-created K8s services Pod via cluster_master_ip/<unique_service_name> ?

Or we could also accept several copies of COTURN inside same node (# of COTURN = # of runtime-created K8s service) which they are listening to difference ports, but only via cluster_master_ip:PORT ?

I would like to ask does this could be done by using stunner ?

Sorry for my bad english.

Thanks so much.

[nmate]

Hey,

At first glance it does not seem impossible. Although I was wondering what could be the benefit for you to cascade two TURN servers in the same cluster? Since STUNner at the edge of the cluster already relays traffic into the cluster, what would be the practical benefit of coTURN in the second step? But anyway, I am not here to ask but to help you out, so getting back to your question STUNner does not care what is on the peer side, it can be another TURN server. As long as the TURN allocation points to your coTURN instance (either running behind a podIP, clusterIP, NodePort service) it simply opens a UDP or TCP route and relays traffic into Kubernetes and lets Kubernetes networking to do the rest. On the other hand, making a TURN allocation from behind another TURN server is something where tweaking on the client side could be needed, I guess. Does this help?

[cclo-astri]

Hi @nmate ,

Thanks for your reply.

The first idea is that I am not intend to cascade TURN servers in a cluster.

What I mean is if the cluster is serving some user devices (e.g. 3 user devices), it will have 3 services (accessible via a unique URI subpath, configured by K8s Ingress). Based on this, what I want is adding a TURN server per service, such that each service has its own TURN server. And I could simply tear down the services when user devices finished its life cycle without affecting others.

BTW, do you mean that I could use STUNner as an edge relay and connected by user devices and PC Chrome browser, to route corresponding p2p connection requests to the TURN servers behind STUNner ?

As STUNner only listen to single port, if user device no.1 wants to connect its own TURN server, how could I config STUNner to route p2p requests to its own TURN server no.1 ?

I wonder how could this actually be done in K8s. Could you mind give me some examples in YAML ?

[rg0now]

What I mean is if the cluster is serving some user devices (e.g. 3 user devices), it will have 3 services (accessible via a unique URI subpath, configured by K8s Ingress). Based on this, what I want is adding a TURN server per service, such that each service has its own TURN server. And I could simply tear down the services when user devices finished its life cycle without affecting others.

While this is clearly doable, it is definitely not the recommended way to do that. Instead, just deploy a single STUNner gateway service (you can scale it to arbitrary many pods) and use it to ingest and route the traffic of all users to the corresponding per-user backend service. When a user wants to join, the corresponding backend will generate an ICE candidate with its own pod IP and send it back to the user, that's how the user's browser will know which backend to connect to via STUNner, and the nice thing is that this is all automatic with WebRTC!

BTW, do you mean that I could use STUNner as an edge relay and connected by user devices and PC Chrome browser, to route corresponding p2p connection requests to the TURN servers behind STUNner ?

That's the purpose, but routing is not "to the TURN servers behind STUNner" (you don't need a TURN server behind STUNner, STUNner is a TURN server) but directly to your per-user backend services. Please make sure to read all the docs here, here and here.

Let's stay at your example: you have 3 users, served by 3 per-user backend deployments/services in Kubernetes, say, service-user1, service-user2, service-user3. Then, deploy a STUNner gateway hierarchy, add a single Gateway to expose STUNner on the protocol/port you want (let's call it my-gateway), and then add 3 separate UDPRoutes, one Route per user, e.g., something like this for user1:

apiVersion: gateway.networking.k8s.io/v1alpha2
kind: UDPRoute
metadata:
  name: route-user1
spec:
  parentRefs:
    - name: my-gateway
  rules:
    - backendRefs:
        - name: service-user1

You can also have a single UDPRoute for all users but it will be a bit more difficult to update it as users come and go:

apiVersion: gateway.networking.k8s.io/v1alpha2
kind: UDPRoute
metadata:
  name: my-route
spec:
  parentRefs:
    - name: my-gateway
  rules:
    - backendRefs:
        - name: service-user1
        - name: service-user2
        - name: service-user3

I wonder how could this actually be done in K8s. Could you mind give me some examples in YAML ?

We try to give you as much help as we can and you can count on our assistance in the future too, but please keep in mind that we have very limited time for unpaid support (we are in the consultancy business though).

[cclo-astri]

Hi @rg0now ,

Thanks for your reply.

Perhaps my english is bad or the situation is too complicated to explain by words. All I need is 1 COTURN server run as a simple container inside 1 Pod (with 1 associated Service + Ingress). And we do not need COTURN help us to route traffics to some arbitrary media services. We only need the p2p connection function + data channel for image sending (which mean we need a very simple video call feature, between user's PC browser and his device).

As our cluster only has 1 entrance, which is cluster_master_ip, in order to access different user-services, we use Ingress. In our current setting, we could access containers in Pod from external via K8s ingress only.

• User1: https://<cluster_master_ip>/user1/....
• User2: https://<cluster_master_ip>/user2/....

For the most early idea, I just think COTURN could be accessed behind Ingress such that users could connect to COTURN via a URI like turn:cluster_master_ip>/user1/coturn or turn:cluster_master_ip>/user2/coturn but I found this URI violates TURN's definition.

So at this moment, due to my lack of knowledge about WebRTC and COTURN stuff, I could only deploy 1 very simple COTURN server into my K8s cluster, and just expose it to public in terms of turn:cluster_master_ip>:<PORT>.

Thanks for your time.

[rg0now]

All I need is 1 COTURN server run as a simple container inside 1 Pod (with 1 associated Service + Ingress).

This is exactly what we propose not to do, because it will be a PITA to route your media to the required service. Deploy a single STUNner and use proper media routing via the Gateway API (i.e., UDPRoutes) to describe your intent.

Now I understand that in your situation maybe an unmanaged coturn sidecar container is the proper choice. We expect you back once you realize you need something that you can actually manage...:-)