search

l7mp / stunner

A Kubernetes media gateway for WebRTC. Contact: info@l7mp.io
https://l7mp.io
MIT License
737 stars 56 forks source link

Deployment in headless mode does not resolve public ip address of client #144

Closed viktor-sedov-gcx closed 5 months ago

viktor-sedov-gcx commented 5 months ago

Description

When deploying stunner in headless mode (following the example provided) stun resolves the client ip address to some internal kubernetes IP address instead of it's public ip address. Therefore STUN/TURN won't work.

Steps to Reproduce

Expected behavior: The clients IP address would be resolved correctly

Actual behavior: Internal kubernetes IP is being returned

Versions

Helm chart 0.18.0

Info

The issue seems to be the LoadBalancers externalTrafficPolicy. However when switching to "local", it works fine.

Is there any way of configuring the externalTrafficPolicy for the helm chart installation?

Gateway API status

apiVersion: v1
items:
- apiVersion: gateway.networking.k8s.io/v1
  kind: Gateway
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"gateway.networking.k8s.io/v1","kind":"Gateway","metadata":{"annotations":{"stunner.l7mp.io/enable-mixed-protocol-lb":"true"},"name":"stunner-netbird-mixed","namespace":"platform-stunner"},"spec":{"gatewayClassName":"stunner-netbird","listeners":[{"name":"udp-gateway","port":3478,"protocol":"TURN-UDP"},{"name":"tcp-gateway","port":3478,"protocol":"TURN-TCP"}]}}
      stunner.l7mp.io/enable-mixed-protocol-lb: "true"
    creationTimestamp: "2024-05-16T08:04:39Z"
    generation: 3
    name: stunner-netbird-mixed
    namespace: platform-stunner
    resourceVersion: "15060695"
    uid: 7b39273a-1185-4d02-8c97-6b28ab83a98c
  spec:
    gatewayClassName: stunner-netbird
    listeners:
    - allowedRoutes:
        namespaces:
          from: Same
      name: udp-gateway
      port: 3478
      protocol: TURN-UDP
    - allowedRoutes:
        namespaces:
          from: Same
      name: tcp-gateway
      port: 3478
      protocol: TURN-TCP
  status:
    addresses:
    - type: IPAddress
      value: 4.175.131.225
    conditions:
    - lastTransitionTime: "2024-05-16T08:04:42Z"
      message: gateway accepted by controller stunner.l7mp.io/gateway-operator
      observedGeneration: 3
      reason: Accepted
      status: "True"
      type: Accepted
    - lastTransitionTime: "2024-05-17T12:22:07Z"
      message: dataplane configuration successfully rendered
      observedGeneration: 3
      reason: Programmed
      status: "True"
      type: Programmed
    listeners:
    - attachedRoutes: 1
      conditions:
      - lastTransitionTime: "2024-05-17T12:22:07Z"
        message: listener accepted
        observedGeneration: 3
        reason: Accepted
        status: "True"
        type: Accepted
      - lastTransitionTime: "2024-05-17T12:22:07Z"
        message: listener protocol-port available
        observedGeneration: 3
        reason: NoConflicts
        status: "False"
        type: Conflicted
      - lastTransitionTime: "2024-05-17T12:22:07Z"
        message: listener object references sucessfully resolved
        observedGeneration: 3
        reason: ResolvedRefs
        status: "True"
        type: ResolvedRefs
      name: udp-gateway
      supportedKinds:
      - group: gateway.networking.k8s.io
        kind: UDPRoute
      - group: stunner.l7mp.io
        kind: UDPRoute
    - attachedRoutes: 1
      conditions:
      - lastTransitionTime: "2024-05-17T12:22:07Z"
        message: listener accepted
        observedGeneration: 3
        reason: Accepted
        status: "True"
        type: Accepted
      - lastTransitionTime: "2024-05-17T12:22:07Z"
        message: listener protocol-port available
        observedGeneration: 3
        reason: NoConflicts
        status: "False"
        type: Conflicted
      - lastTransitionTime: "2024-05-17T12:22:07Z"
        message: listener object references sucessfully resolved
        observedGeneration: 3
        reason: ResolvedRefs
        status: "True"
        type: ResolvedRefs
      name: tcp-gateway
      supportedKinds:
      - group: gateway.networking.k8s.io
        kind: UDPRoute
      - group: stunner.l7mp.io
        kind: UDPRoute
- apiVersion: stunner.l7mp.io/v1
  kind: GatewayConfig
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"stunner.l7mp.io/v1","kind":"GatewayConfig","metadata":{"annotations":{},"name":"stunner-netbird","namespace":"platform-stunner"},"spec":{"authRef":{"name":"stunner-basic-auth","namespace":"platform-stunner"},"dataplane":"default","logLevel":"all:INFO","realm":"stunner.l7mp.io"}}
    creationTimestamp: "2024-05-16T08:04:40Z"
    generation: 1
    name: stunner-netbird
    namespace: platform-stunner
    resourceVersion: "14484047"
    uid: f91bfbfc-7e7c-4082-a788-3815307a1749
  spec:
    authRef:
      group: ""
      kind: Secret
      name: stunner-basic-auth
      namespace: platform-stunner
    authType: plaintext
    dataplane: default
    logLevel: all:INFO
    realm: stunner.l7mp.io
- apiVersion: gateway.networking.k8s.io/v1
  kind: GatewayClass
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"gateway.networking.k8s.io/v1","kind":"GatewayClass","metadata":{"annotations":{},"name":"stunner-netbird"},"spec":{"controllerName":"stunner.l7mp.io/gateway-operator","description":"Default GatewayClass for Stunner used for netbird poc deployment","parametersRef":{"group":"stunner.l7mp.io","kind":"GatewayConfig","name":"stunner-netbird","namespace":"platform-stunner"}}}
    creationTimestamp: "2024-05-15T12:25:39Z"
    generation: 2
    name: stunner-netbird
    resourceVersion: "14484077"
    uid: aeafe850-ba2a-4adf-abef-a26eab1e5a71
  spec:
    controllerName: stunner.l7mp.io/gateway-operator
    description: Default GatewayClass for Stunner used for netbird poc deployment
    parametersRef:
      group: stunner.l7mp.io
      kind: GatewayConfig
      name: stunner-netbird
      namespace: platform-stunner
  status:
    conditions:
    - lastTransitionTime: "2024-05-16T08:04:42Z"
      message: GatewayClass is now managed by controller "stunner.l7mp.io/gateway-operator"
      observedGeneration: 2
      reason: Accepted
      status: "True"
      type: Accepted
- apiVersion: stunner.l7mp.io/v1
  kind: UDPRoute
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"stunner.l7mp.io/v1","kind":"UDPRoute","metadata":{"annotations":{},"name":"stunner-headless","namespace":"platform-stunner"},"spec":{"parentRefs":[{"name":"stunner-netbird-mixed"}],"rules":[{"backendRefs":[{"name":"stunner-netbird-mixed","namespace":"platform-stunner"}]}]}}
    creationTimestamp: "2024-05-16T08:04:40Z"
    generation: 1
    name: stunner-headless
    namespace: platform-stunner
    resourceVersion: "15060696"
    uid: dd4b9d3e-880f-4fc8-b818-5e4e6bf53124
  spec:
    parentRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: stunner-netbird-mixed
    rules:
    - backendRefs:
      - group: ""
        kind: Service
        name: stunner-netbird-mixed
        namespace: platform-stunner
  status:
    parents:
    - conditions:
      - lastTransitionTime: "2024-05-17T12:22:07Z"
        message: parent accepts the route
        observedGeneration: 1
        reason: Accepted
        status: "True"
        type: Accepted
      - lastTransitionTime: "2024-05-17T12:22:07Z"
        message: all backend references successfully resolved
        observedGeneration: 1
        reason: ResolvedRefs
        status: "True"
        type: ResolvedRefs
      controllerName: stunner.l7mp.io/gateway-operator
      parentRef:
        group: gateway.networking.k8s.io
        kind: Gateway
        name: stunner-netbird-mixed
kind: List
metadata:
  resourceVersion: ""

Operator logs

<details> 
  <summary>Logs</summary>
   2024-05-17T12:24:47.436819973Z  INFO    renderer        STUNner dataplane Deployment ready      {"generation": 2261, "deployment": "{\"metadata\":{\"name\":\"stunner-netbird-mixed\",\"namespace\":\"platform-stunner\",\"creationTimestamp\":null,\"labels\":{\"stunner.l7mp.io/owned-by\":\"stunner\",\"stunner.l7mp.io/related-gateway-name\":\"stunner-netbird-mixed\",\"stunner.l7mp.io/related-gateway-namespace\":\"platform-stunner\"},\"annotations\":{\"stunner.l7mp.io/enable-mixed-protocol-lb\":\"true\",\"stunner.l7mp.io/related-gateway-name\":\"platform-stunner/stunner-netbird-mixed\"},\"ownerReferences\":[{\"apiVersion\":\"gateway.networking.k8s.io/v1\",\"kind\":\"Gateway\",\"name\":\"stunner-netbird-mixed\",\"uid\":\"7b39273a-1185-4d02-8c97-6b28ab83a98c\"}]},\"spec\":{\"replicas\":1,\"selector\":{\"matchExpressions\":[{\"key\":\"app\",\"operator\":\"In\",\"values\":[\"stunner\"]},{\"key\":\"stunner.l7mp.io/related-gateway-name\",\"operator\":\"In\",\"values\":[\"stunner-netbird-mixed\"]},{\"key\":\"stunner.l7mp.io/related-gateway-namespace\",\"operator\":\"In\",\"values\":[\"platform-stunner\"]}]},\"template\":{\"metadata\":{\"creationTimestamp\":null,\"labels\":{\"app\":\"stunner\",\"stunner.l7mp.io/related-gateway-name\":\"stunner-netbird-mixed\",\"stunner.l7mp.io/related-gateway-namespace\":\"platform-stunner\"},\"annotations\":{\"stunner.l7mp.io/related-gateway-name\":\"platform-stunner/stunner-netbird-mixed\"}},\"spec\":{\"containers\":[{\"name\":\"stunner-daemon\",\"image\":\"docker.io/l7mp/stunnerd:0.18.0\",\"command\":[\"stunnerd\"],\"args\":[\"-w\",\"--udp-thread-num=16\"],\"env\":[{\"name\":\"STUNNER_ADDR\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"status.podIP\"}}},{\"name\":\"STUNNER_NAME\",\"value\":\"stunner-netbird-mixed\"},{\"name\":\"STUNNER_NAMESPACE\",\"value\":\"platform-stunner\"},{\"name\":\"STUNNER_CONFIG_ORIGIN\",\"value\":\"http://172.16.24.31:13478\"}],\"resources\":{\"limits\":{\"cpu\":\"2\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"500m\",\"memory\":\"128Mi\"}},\"livenessProbe\":{\"httpGet\":{\"path\":\"/live\",\"port\":8086,\"scheme\":\"HTTP\"},\"timeoutSeconds\":1,\"periodSeconds\":15,\"successThreshold\":1,\"failureThreshold\":3},\"readinessProbe\":{\"httpGet\":{\"path\":\"/ready\",\"port\":8086,\"scheme\":\"HTTP\"},\"timeoutSeconds\":1,\"periodSeconds\":15,\"successThreshold\":1,\"failureThreshold\":3},\"imagePullPolicy\":\"Always\"}],\"terminationGracePeriodSeconds\":3600}},\"strategy\":{}},\"status\":{}}"}
2024-05-17T12:24:47.436863801Z  INFO    renderer        STUNner dataplane configuration ready   {"generation": 2261, "config": "{version=\"v1\",admin:{name=\"platform-stunner/stunner-netbird-mixed\",logLevel=\"all:INFO\",health-check=\"http://:8086\"},static-auth:{realm=\"stunner.l7mp.io\",username=\"<SECRET>\",password=\"<SECRET>\"},listeners=[\"platform-stunner/stunner-netbird-mixed/udp-gateway\":{turn://0.0.0.0:3478,public=4.175.131.225:3478,cert/key=-/-,routes=[platform-stunner/stunner-headless]},\"platform-stunner/stunner-netbird-mixed/tcp-gateway\":{turn://0.0.0.0:3478,public=4.175.131.225:3478,cert/key=-/-,routes=[platform-stunner/stunner-headless]}],clusters=[\"platform-stunner/stunner-headless\":{type=\"STATIC\",protocol=\"UDP\",endpoints=[10.0.161.193,172.16.24.34]}]}"}
2024-05-17T12:24:47.436915163Z  INFO    cds-server      processing config update event  {"generation": 2261, "update": "update (gen: 2261): upsert-queue: gway-cls: 1, gway: 1, route: 1, routeV1A2: 0, svc: 1, confmap: 0, dp: 1 / delete-queue: gway-cls: 0, gway: 0, route: 0, routeV1A2: 0, svc: 0, confmap: 0, dp: 0 / config-queue: 1"}
2024-05-17T12:24:47.436994391Z  INFO    updater processing update event {"generation": 2261, "update": "update (gen: 2261): upsert-queue: gway-cls: 1, gway: 1, route: 1, routeV1A2: 0, svc: 1, confmap: 0, dp: 1 / delete-queue: gway-cls: 0, gway: 0, route: 0, routeV1A2: 0, svc: 0, confmap: 0, dp: 0 / config-queue: 1"}
2024-05-17T12:26:35.021495533Z  INFO    node-controller reconciling     {"node": "/aks-poolopxi-37140328-vmss000001"}
2024-05-17T12:26:35.021660096Z  INFO    node-controller failed to find node with valid external address {"reason": "end of node list reached after searching through 4 node(s)"}
2024-05-17T12:26:35.272075164Z  INFO    renderer        rendering configuration {"generation": 2262, "event": "render"}
2024-05-17T12:26:35.272103863Z  INFO    renderer        commencing dataplane render     {"mode": "managed"}
2024-05-17T12:26:35.272121792Z  INFO    renderer        rendering configuration {"gateway-class": "/stunner-netbird"}
2024-05-17T12:26:35.272434234Z  INFO    renderer        creating public service for gateway     {"service": "platform-stunner/stunner-netbird-mixed", "gateway": "platform-stunner/stunner-netbird-mixed", "service": "{\"kind\":\"Service\",\"apiVersion\":\"v1\",\"metadata\":{\"name\":\"stunner-netbird-mixed\",\"namespace\":\"platform-stunner\",\"uid\":\"3dfad60a-9815-4474-ac39-5107054cb7fa\",\"resourceVersion\":\"14991483\",\"creationTimestamp\":\"2024-05-16T08:04:42Z\",\"labels\":{\"stunner.l7mp.io/owned-by\":\"stunner\",\"stunner.l7mp.io/related-gateway-name\":\"stunner-netbird-mixed\",\"stunner.l7mp.io/related-gateway-namespace\":\"platform-stunner\"},\"annotations\":{\"stunner.l7mp.io/enable-mixed-protocol-lb\":\"true\",\"stunner.l7mp.io/related-gateway-name\":\"platform-stunner/stunner-netbird-mixed\"},\"ownerReferences\":[{\"apiVersion\":\"gateway.networking.k8s.io/v1\",\"kind\":\"Gateway\",\"name\":\"stunner-netbird-mixed\",\"uid\":\"7b39273a-1185-4d02-8c97-6b28ab83a98c\"}],\"finalizers\":[\"service.kubernetes.io/load-balancer-cleanup\"]},\"spec\":{\"ports\":[{\"name\":\"udp-gateway\",\"protocol\":\"UDP\",\"port\":3478,\"targetPort\":3478,\"nodePort\":31024},{\"name\":\"tcp-gateway\",\"protocol\":\"TCP\",\"port\":3478,\"targetPort\":3478,\"nodePort\":31024}],\"selector\":{\"app\":\"stunner\",\"stunner.l7mp.io/related-gateway-name\":\"stunner-netbird-mixed\",\"stunner.l7mp.io/related-gateway-namespace\":\"platform-stunner\"},\"clusterIP\":\"10.0.161.193\",\"clusterIPs\":[\"10.0.161.193\"],\"type\":\"LoadBalancer\",\"sessionAffinity\":\"None\",\"loadBalancerIP\":\"4.175.131.225\",\"externalTrafficPolicy\":\"Local\",\"healthCheckNodePort\":31809,\"ipFamilies\":[\"IPv4\"],\"ipFamilyPolicy\":\"SingleStack\",\"allocateLoadBalancerNodePorts\":true,\"internalTrafficPolicy\":\"Cluster\"},\"status\":{\"loadBalancer\":{\"ingress\":[{\"ip\":\"4.175.131.225\"}]}}}"}
2024-05-17T12:26:35.272591935Z  INFO    renderer        update (gen: 2262): upsert-queue: gway-cls: 0, gway: 1, route: 1, routeV1A2: 0, svc: 1, confmap: 0, dp: 0 / delete-queue: gway-cls: 0, gway: 0, route: 0, routeV1A2: 0, svc: 0, confmap: 0, dp: 0 / config-queue: 0
2024-05-17T12:26:35.27293623Z   INFO    renderer        STUNner dataplane Deployment ready      {"generation": 2262, "deployment": "{\"metadata\":{\"name\":\"stunner-netbird-mixed\",\"namespace\":\"platform-stunner\",\"creationTimestamp\":null,\"labels\":{\"stunner.l7mp.io/owned-by\":\"stunner\",\"stunner.l7mp.io/related-gateway-name\":\"stunner-netbird-mixed\",\"stunner.l7mp.io/related-gateway-namespace\":\"platform-stunner\"},\"annotations\":{\"stunner.l7mp.io/enable-mixed-protocol-lb\":\"true\",\"stunner.l7mp.io/related-gateway-name\":\"platform-stunner/stunner-netbird-mixed\"},\"ownerReferences\":[{\"apiVersion\":\"gateway.networking.k8s.io/v1\",\"kind\":\"Gateway\",\"name\":\"stunner-netbird-mixed\",\"uid\":\"7b39273a-1185-4d02-8c97-6b28ab83a98c\"}]},\"spec\":{\"replicas\":1,\"selector\":{\"matchExpressions\":[{\"key\":\"app\",\"operator\":\"In\",\"values\":[\"stunner\"]},{\"key\":\"stunner.l7mp.io/related-gateway-name\",\"operator\":\"In\",\"values\":[\"stunner-netbird-mixed\"]},{\"key\":\"stunner.l7mp.io/related-gateway-namespace\",\"operator\":\"In\",\"values\":[\"platform-stunner\"]}]},\"template\":{\"metadata\":{\"creationTimestamp\":null,\"labels\":{\"app\":\"stunner\",\"stunner.l7mp.io/related-gateway-name\":\"stunner-netbird-mixed\",\"stunner.l7mp.io/related-gateway-namespace\":\"platform-stunner\"},\"annotations\":{\"stunner.l7mp.io/related-gateway-name\":\"platform-stunner/stunner-netbird-mixed\"}},\"spec\":{\"containers\":[{\"name\":\"stunner-daemon\",\"image\":\"docker.io/l7mp/stunnerd:0.18.0\",\"command\":[\"stunnerd\"],\"args\":[\"-w\",\"--udp-thread-num=16\"],\"env\":[{\"name\":\"STUNNER_ADDR\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"status.podIP\"}}},{\"name\":\"STUNNER_NAME\",\"value\":\"stunner-netbird-mixed\"},{\"name\":\"STUNNER_NAMESPACE\",\"value\":\"platform-stunner\"},{\"name\":\"STUNNER_CONFIG_ORIGIN\",\"value\":\"http://172.16.24.31:13478\"}],\"resources\":{\"limits\":{\"cpu\":\"2\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"500m\",\"memory\":\"128Mi\"}},\"livenessProbe\":{\"httpGet\":{\"path\":\"/live\",\"port\":8086,\"scheme\":\"HTTP\"},\"timeoutSeconds\":1,\"periodSeconds\":15,\"successThreshold\":1,\"failureThreshold\":3},\"readinessProbe\":{\"httpGet\":{\"path\":\"/ready\",\"port\":8086,\"scheme\":\"HTTP\"},\"timeoutSeconds\":1,\"periodSeconds\":15,\"successThreshold\":1,\"failureThreshold\":3},\"imagePullPolicy\":\"Always\"}],\"terminationGracePeriodSeconds\":3600}},\"strategy\":{}},\"status\":{}}"}
2024-05-17T12:26:35.273039314Z  INFO    renderer        STUNner dataplane configuration ready   {"generation": 2262, "config": "{version=\"v1\",admin:{name=\"platform-stunner/stunner-netbird-mixed\",logLevel=\"all:INFO\",health-check=\"http://:8086\"},static-auth:{realm=\"stunner.l7mp.io\",username=\"<SECRET>\",password=\"<SECRET>\"},listeners=[\"platform-stunner/stunner-netbird-mixed/udp-gateway\":{turn://0.0.0.0:3478,public=4.175.131.225:3478,cert/key=-/-,routes=[platform-stunner/stunner-headless]},\"platform-stunner/stunner-netbird-mixed/tcp-gateway\":{turn://0.0.0.0:3478,public=4.175.131.225:3478,cert/key=-/-,routes=[platform-stunner/stunner-headless]}],clusters=[\"platform-stunner/stunner-headless\":{type=\"STATIC\",protocol=\"UDP\",endpoints=[10.0.161.193,172.16.24.34]}]}"}
2024-05-17T12:26:35.273121369Z  INFO    cds-server      processing config update event  {"generation": 2262, "update": "update (gen: 2262): upsert-queue: gway-cls: 1, gway: 1, route: 1, routeV1A2: 0, svc: 1, confmap: 0, dp: 1 / delete-queue: gway-cls: 0, gway: 0, route: 0, routeV1A2: 0, svc: 0, confmap: 0, dp: 0 / config-queue: 1"}
2024-05-17T12:26:35.273162868Z  INFO    updater processing update event {"generation": 2262, "update": "update (gen: 2262): upsert-queue: gway-cls: 1, gway: 1, route: 1, routeV1A2: 0, svc: 1, confmap: 0, dp: 1 / delete-queue: gway-cls: 0, gway: 0, route: 0, routeV1A2: 0, svc: 0, confmap: 0, dp: 0 / config-queue: 1"}
</details>
rg0now commented 5 months ago

This is intended. As long as the STUNner dataplane is deployed into ordinary Kubernetes pods and runs on private IPs (which is The Way For Deploying STUNner), it will return a fairly random private IP in STUN responses (unless you use some ugly Kubernetes [hacks]()).

This is exactly why we do not encourage people to use STUNner as a STUN service. If you really insist on using STUN though, deploy the STUNner dataplane into the host-network namespace. But you'd better not do that, as it fails in many cloud providers' service.

The good news is that it does not matter: STUNner is intended to be used as a TURN Gateway and luckily TURN does not care about the client IP at all.

rg0now commented 5 months ago

Hopefully implementing this issue would solve this.

viktor-sedov-gcx commented 5 months ago

thank you very much for your quick response and already implementing this feature!

we're using stunner as the backbone for a peer-to-peer networking solution and the docs implied that the headless deployment did exactly what I stated above ( at least for me)

do you have a schedule for when this feature will be available in a release?

rg0now commented 5 months ago

We plan to release an RC-3 this or early next month, but there are still some outstanding issues we need to close first. Until then, you can test the latest goodies from the dev release channel.