search

l7mp / stunner

A Kubernetes media gateway for WebRTC. Contact: info@l7mp.io
https://l7mp.io
MIT License
751 stars 58 forks source link

Question about debugging message on UDP gateway pod #118

Closed tyohan closed 8 months ago

tyohan commented 10 months ago

Hi, I'm using Stunner latest release, v0.17.0, and I can't get it to work on one of my services. I have multiple webrtc services, and one of my services is working fine with my current setup, but another can't connect.

My setup uses the asynchronous mode, and on the failed service, I can see that both candidates from the client and server are private IPs, as expected. However, the client always failed to connect to the server. In the udp-gateway pod I can see there is a permission denied like this:

07:02:34.743527 turn.go:239: turn INFO: permission denied for client 10.1.0.114:17224 to peer 10.1.0.96

Any info about the message? What possible cause of the issue? I can confirm that the turn authentication is the same between the one that is working and the one that is not working. So I don't think it's an authentication issue. Do you have any clue where I should dig?

Thanks

rg0now commented 10 months ago

This usually means that the backend Service in the UDPRoute does not cover the server's pod IP the client wants to connect to. Make sure that all your WebRTC media server pods are actually part of one of your backend Services, otherwise STUNner will deny access (as per the above message) for security reasons.

tyohan commented 10 months ago

Based on the service endpoint in the attached screenshot below, the service targets the right pod. That inlive-stream-xxx pod is one of the WebRTC media servers I'm trying to connect with; after attempting to ensure that the service points to the right pod, the client still can't connect to the WebRTC media server.

image

Another message from udp-gateway also printed this one:

16:26:54.477950 allocation.go:283: turn INFO: No Permission or Channel exists for 10.1.0.143:50005 on allocation 10.1.0.78:36504 (suppressed 2 log events)

Can you also help with that message? Where can I read the code that is responsible for printing both messages?

Thank you

rg0now commented 10 months ago

Unfortunately, this input is not enough to pinpoint the problem. Can you please post the full stunnerd logs (preferably at a high loglevel like all:TRACE) and the operator logs?

Can you also help with that message? Where can I read the code that is responsible for printing both messages?

These are coming from pion/turn, the TURN implementation we're using. The issue is that your client is trying to reach a peer via STUNner that it has not obtained a permission for. This error can be a normal conclusion of the previous problems. Once you post the full logs, we'll know more.

tyohan commented 9 months ago

Below are the logs from deploying the media plane pod to my cluster. This pod is dynamically deployed when an API endpoint is called. When the API endpoint is called, it will deploy the media plane pod, create a service, and stunner UDP route in the same time. Based on the operator logs below, the UDP route is trying to connect to the service, but the service was just recently created and is probably not ready yet. What I know is that all K8s create resource APIs that are async, and that's why the service is probably not found when the UDP route is created.

My next question is, when the UDP route is created, must the service exist, or will it retry until it gets the service?

This is the operator logs:

2024-02-05T10:15:27.496920941Z  INFO    renderer    STUNner dataplane Deployment ready  {"generation": 2058, "deployment": "{\"metadata\":{\"name\":\"udp-gateway\",\"namespace\":\"stunner\",\"creationTimestamp\":null,\"labels\":{\"stunner.l7mp.io/owned-by\":\"stunner\",\"stunner.l7mp.io/related-gateway-name\":\"udp-gateway\",\"stunner.l7mp.io/related-gateway-namespace\":\"stunner\"},\"annotations\":{\"stunner.l7mp.io/related-gateway-name\":\"stunner/udp-gateway\"},\"ownerReferences\":[{\"apiVersion\":\"gateway.networking.k8s.io/v1\",\"kind\":\"Gateway\",\"name\":\"udp-gateway\",\"uid\":\"d1cf77dc-ac96-492e-b9df-31d9d47ad24b\"}]},\"spec\":{\"replicas\":1,\"selector\":{\"matchExpressions\":[{\"key\":\"app\",\"operator\":\"In\",\"values\":[\"stunner\"]},{\"key\":\"stunner.l7mp.io/related-gateway-name\",\"operator\":\"In\",\"values\":[\"udp-gateway\"]},{\"key\":\"stunner.l7mp.io/related-gateway-namespace\",\"operator\":\"In\",\"values\":[\"stunner\"]}]},\"template\":{\"metadata\":{\"creationTimestamp\":null,\"labels\":{\"app\":\"stunner\",\"stunner.l7mp.io/related-gateway-name\":\"udp-gateway\",\"stunner.l7mp.io/related-gateway-namespace\":\"stunner\"},\"annotations\":{\"stunner.l7mp.io/related-gateway-name\":\"stunner/udp-gateway\"}},\"spec\":{\"containers\":[{\"name\":\"stunner-daemon\",\"image\":\"l7mp/stunnerd:0.17.0\",\"command\":[\"stunnerd\"],\"args\":[\"-w\",\"--udp-thread-num=16\"],\"env\":[{\"name\":\"STUNNER_ADDR\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"status.podIP\"}}},{\"name\":\"STUNNER_NAME\",\"value\":\"udp-gateway\"},{\"name\":\"STUNNER_NAMESPACE\",\"value\":\"stunner\"},{\"name\":\"STUNNER_CONFIG_ORIGIN\",\"value\":\"http://10.1.0.45:13478\"}],\"resources\":{\"limits\":{\"cpu\":\"2\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"500m\",\"memory\":\"128Mi\"}},\"livenessProbe\":{\"httpGet\":{\"path\":\"/live\",\"port\":8086,\"scheme\":\"HTTP\"},\"timeoutSeconds\":1,\"periodSeconds\":15,\"successThreshold\":1,\"failureThreshold\":3},\"readinessProbe\":{\"httpGet\":{\"path\":\"/ready\",\"port\":8086,\"scheme\":\"HTTP\"},\"timeoutSeconds\":1,\"periodSeconds\":15,\"successThreshold\":1,\"failureThreshold\":3},\"imagePullPolicy\":\"Always\"}],\"terminationGracePeriodSeconds\":3600}},\"strategy\":{}},\"status\":{}}"}
2024-02-05T10:15:27.496946258Z  INFO    renderer    STUNner data plane configuration ready  {"generation": 2058, "config": "{version=\"v1\",admin:{name=\"stunner/udp-gateway\",logLevel=\"all:INFO\",health-check=\"http://:8086\"},auth:{realm=\"turn.example.com\",type=\"static\",username=\"<SECRET>\",password=\"<SECRET>\"},listeners=[\"stunner/udp-gateway/udp-listener\":{turn://0.0.0.0:3478,public=x.x.x.x:3478,cert/key=-/-,routes=[stunner/dev-hub,stunner/hub]}],clusters=[\"stunner/dev-hub\":{type=\"STATIC\",protocol=\"UDP\",endpoints=[10.1.0.182,10.152.183.173]},\"stunner/hub\":{type=\"STATIC\",protocol=\"UDP\",endpoints=[10.1.0.180,10.152.183.71]}]}"}
2024-02-05T10:15:27.496972678Z  INFO    cds-server  processing config update event  {"generation": 2058, "update": "update (gen: 2058): upsert-queue: gway-cls: 1, gway: 1, route: 2, routeV1A2: 0, svc: 1, confmap: 1, dp: 1 / delete-queue: gway-cls: 0, gway: 0, route: 0, routeV1A2: 0, svc: 0, confmap: 0, dp: 0 / config-queue: 1"}
2024-02-05T10:15:27.496986564Z  INFO    updater processing update event {"generation": 2058, "update": "update (gen: 2058): upsert-queue: gway-cls: 1, gway: 1, route: 2, routeV1A2: 0, svc: 1, confmap: 1, dp: 1 / delete-queue: gway-cls: 0, gway: 0, route: 0, routeV1A2: 0, svc: 0, confmap: 0, dp: 0 / config-queue: 1"}
2024-02-05T10:20:32.820877807Z  INFO    node-controller reconciling {"node": "/node01"}
2024-02-05T10:20:32.820952899Z  INFO    node-controller failed to find node with valid external address {"reason": "end of node list reached after searching through 1 node(s)"}
2024-02-05T10:20:33.071347387Z  INFO    renderer    rendering configuration {"generation": 2059, "event": "render"}
2024-02-05T10:20:33.071373076Z  INFO    renderer    commencing dataplane render {"mode": "managed"}
2024-02-05T10:20:33.071382353Z  INFO    renderer    rendering configuration {"gateway-class": "/stunner-gatewayclass"}
2024-02-05T10:20:33.0715874Z    INFO    renderer    creating public service for gateway {"service": "stunner/udp-gateway", "gateway": "stunner/udp-gateway", "service": "{\"kind\":\"Service\",\"apiVersion\":\"v1\",\"metadata\":{\"name\":\"udp-gateway\",\"namespace\":\"stunner\",\"uid\":\"a8ce9f53-960b-4588-a3f3-ca625bbeb53a\",\"resourceVersion\":\"12285358\",\"creationTimestamp\":\"2024-01-29T17:40:57Z\",\"labels\":{\"stunner.l7mp.io/owned-by\":\"stunner\",\"stunner.l7mp.io/related-gateway-name\":\"udp-gateway\",\"stunner.l7mp.io/related-gateway-namespace\":\"stunner\"},\"annotations\":{\"metallb.universe.tf/allow-shared-ip\":\"x.x.x.x\",\"stunner.l7mp.io/related-gateway-name\":\"stunner/udp-gateway\"},\"ownerReferences\":[{\"apiVersion\":\"gateway.networking.k8s.io/v1\",\"kind\":\"Gateway\",\"name\":\"udp-gateway\",\"uid\":\"d1cf77dc-ac96-492e-b9df-31d9d47ad24b\"}]},\"spec\":{\"ports\":[{\"name\":\"udp-listener\",\"protocol\":\"UDP\",\"port\":3478,\"targetPort\":3478,\"nodePort\":32298}],\"selector\":{\"app\":\"stunner\",\"stunner.l7mp.io/related-gateway-name\":\"udp-gateway\",\"stunner.l7mp.io/related-gateway-namespace\":\"stunner\"},\"clusterIP\":\"10.152.183.130\",\"clusterIPs\":[\"10.152.183.130\"],\"type\":\"LoadBalancer\",\"sessionAffinity\":\"None\",\"loadBalancerIP\":\"x.x.x.x\",\"externalTrafficPolicy\":\"Cluster\",\"ipFamilies\":[\"IPv4\"],\"ipFamilyPolicy\":\"SingleStack\",\"allocateLoadBalancerNodePorts\":true,\"internalTrafficPolicy\":\"Cluster\"},\"status\":{\"loadBalancer\":{\"ingress\":[{\"ip\":\"x.x.x.x\"}]}}}"}
2024-02-05T10:20:33.071655007Z  INFO    renderer    update (gen: 2059): upsert-queue: gway-cls: 0, gway: 1, route: 2, routeV1A2: 0, svc: 1, confmap: 0, dp: 0 / delete-queue: gway-cls: 0, gway: 0, route: 0, routeV1A2: 0, svc: 0, confmap: 0, dp: 0 / config-queue: 0
2024-02-05T10:20:33.071876124Z  INFO    renderer    STUNner dataplane Deployment ready  {"generation": 2059, "deployment": "{\"metadata\":{\"name\":\"udp-gateway\",\"namespace\":\"stunner\",\"creationTimestamp\":null,\"labels\":{\"stunner.l7mp.io/owned-by\":\"stunner\",\"stunner.l7mp.io/related-gateway-name\":\"udp-gateway\",\"stunner.l7mp.io/related-gateway-namespace\":\"stunner\"},\"annotations\":{\"stunner.l7mp.io/related-gateway-name\":\"stunner/udp-gateway\"},\"ownerReferences\":[{\"apiVersion\":\"gateway.networking.k8s.io/v1\",\"kind\":\"Gateway\",\"name\":\"udp-gateway\",\"uid\":\"d1cf77dc-ac96-492e-b9df-31d9d47ad24b\"}]},\"spec\":{\"replicas\":1,\"selector\":{\"matchExpressions\":[{\"key\":\"app\",\"operator\":\"In\",\"values\":[\"stunner\"]},{\"key\":\"stunner.l7mp.io/related-gateway-name\",\"operator\":\"In\",\"values\":[\"udp-gateway\"]},{\"key\":\"stunner.l7mp.io/related-gateway-namespace\",\"operator\":\"In\",\"values\":[\"stunner\"]}]},\"template\":{\"metadata\":{\"creationTimestamp\":null,\"labels\":{\"app\":\"stunner\",\"stunner.l7mp.io/related-gateway-name\":\"udp-gateway\",\"stunner.l7mp.io/related-gateway-namespace\":\"stunner\"},\"annotations\":{\"stunner.l7mp.io/related-gateway-name\":\"stunner/udp-gateway\"}},\"spec\":{\"containers\":[{\"name\":\"stunner-daemon\",\"image\":\"l7mp/stunnerd:0.17.0\",\"command\":[\"stunnerd\"],\"args\":[\"-w\",\"--udp-thread-num=16\"],\"env\":[{\"name\":\"STUNNER_ADDR\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"status.podIP\"}}},{\"name\":\"STUNNER_NAME\",\"value\":\"udp-gateway\"},{\"name\":\"STUNNER_NAMESPACE\",\"value\":\"stunner\"},{\"name\":\"STUNNER_CONFIG_ORIGIN\",\"value\":\"http://10.1.0.45:13478\"}],\"resources\":{\"limits\":{\"cpu\":\"2\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"500m\",\"memory\":\"128Mi\"}},\"livenessProbe\":{\"httpGet\":{\"path\":\"/live\",\"port\":8086,\"scheme\":\"HTTP\"},\"timeoutSeconds\":1,\"periodSeconds\":15,\"successThreshold\":1,\"failureThreshold\":3},\"readinessProbe\":{\"httpGet\":{\"path\":\"/ready\",\"port\":8086,\"scheme\":\"HTTP\"},\"timeoutSeconds\":1,\"periodSeconds\":15,\"successThreshold\":1,\"failureThreshold\":3},\"imagePullPolicy\":\"Always\"}],\"terminationGracePeriodSeconds\":3600}},\"strategy\":{}},\"status\":{}}"}
2024-02-05T10:20:33.071900901Z  INFO    renderer    STUNner dataplane configuration ready   {"generation": 2059, "config": "{version=\"v1\",admin:{name=\"stunner/udp-gateway\",logLevel=\"all:INFO\",health-check=\"http://:8086\"},auth:{realm=\"turn.example.com\",type=\"static\",username=\"<SECRET>\",password=\"<SECRET>\"},listeners=[\"stunner/udp-gateway/udp-listener\":{turn://0.0.0.0:3478,public=x.x.x.x:3478,cert/key=-/-,routes=[stunner/dev-hub,stunner/hub]}],clusters=[\"stunner/dev-hub\":{type=\"STATIC\",protocol=\"UDP\",endpoints=[10.1.0.182,10.152.183.173]},\"stunner/hub\":{type=\"STATIC\",protocol=\"UDP\",endpoints=[10.1.0.180,10.152.183.71]}]}"}
2024-02-05T10:20:33.071927581Z  INFO    cds-server  processing config update event  {"generation": 2059, "update": "update (gen: 2059): upsert-queue: gway-cls: 1, gway: 1, route: 2, routeV1A2: 0, svc: 1, confmap: 1, dp: 1 / delete-queue: gway-cls: 0, gway: 0, route: 0, routeV1A2: 0, svc: 0, confmap: 0, dp: 0 / config-queue: 1"}
2024-02-05T10:20:33.071952328Z  INFO    updater processing update event {"generation": 2059, "update": "update (gen: 2059): upsert-queue: gway-cls: 1, gway: 1, route: 2, routeV1A2: 0, svc: 1, confmap: 1, dp: 1 / delete-queue: gway-cls: 0, gway: 0, route: 0, routeV1A2: 0, svc: 0, confmap: 0, dp: 0 / config-queue: 1"}
2024-02-05T10:21:31.132396986Z  INFO    udproute-controller reconciling {"resource": "stunner/inlive-stream-55"}
2024-02-05T10:21:31.132494069Z  INFO    udproute-controller no Service found for UDPRoute backend   {"udproute": "stunner/inlive-stream-55", "namespace": "inlive", "name": "inlive-stream-55"}
2024-02-05T10:21:31.13249997Z   INFO    udproute-controller no Endpoints found for UDPRoute backend {"udproute": "stunner/inlive-stream-55", "namespace": "inlive", "name": "inlive-stream-55"}
2024-02-05T10:21:31.135617038Z  INFO    udproute-controller reconciling {"resource": "inlive/inlive-stream-55"}
2024-02-05T10:21:31.135691659Z  INFO    udproute-controller no Endpoints found for UDPRoute backend {"udproute": "stunner/inlive-stream-55", "namespace": "inlive", "name": "inlive-stream-55"}
2024-02-05T10:21:31.142910722Z  INFO    udproute-controller reconciling {"resource": "inlive/inlive-stream-55"}
2024-02-05T10:21:31.382685661Z  INFO    renderer    rendering configuration {"generation": 2060, "event": "render"}
2024-02-05T10:21:31.382721138Z  INFO    renderer    commencing dataplane render {"mode": "managed"}
2024-02-05T10:21:31.382733491Z  INFO    renderer    rendering configuration {"gateway-class": "/stunner-gatewayclass"}
2024-02-05T10:21:31.382936173Z  INFO    renderer    creating public service for gateway {"service": "stunner/udp-gateway", "gateway": "stunner/udp-gateway", "service": "{\"kind\":\"Service\",\"apiVersion\":\"v1\",\"metadata\":{\"name\":\"udp-gateway\",\"namespace\":\"stunner\",\"uid\":\"a8ce9f53-960b-4588-a3f3-ca625bbeb53a\",\"resourceVersion\":\"12285358\",\"creationTimestamp\":\"2024-01-29T17:40:57Z\",\"labels\":{\"stunner.l7mp.io/owned-by\":\"stunner\",\"stunner.l7mp.io/related-gateway-name\":\"udp-gateway\",\"stunner.l7mp.io/related-gateway-namespace\":\"stunner\"},\"annotations\":{\"metallb.universe.tf/allow-shared-ip\":\"x.x.x.x\",\"stunner.l7mp.io/related-gateway-name\":\"stunner/udp-gateway\"},\"ownerReferences\":[{\"apiVersion\":\"gateway.networking.k8s.io/v1\",\"kind\":\"Gateway\",\"name\":\"udp-gateway\",\"uid\":\"d1cf77dc-ac96-492e-b9df-31d9d47ad24b\"}]},\"spec\":{\"ports\":[{\"name\":\"udp-listener\",\"protocol\":\"UDP\",\"port\":3478,\"targetPort\":3478,\"nodePort\":32298}],\"selector\":{\"app\":\"stunner\",\"stunner.l7mp.io/related-gateway-name\":\"udp-gateway\",\"stunner.l7mp.io/related-gateway-namespace\":\"stunner\"},\"clusterIP\":\"10.152.183.130\",\"clusterIPs\":[\"10.152.183.130\"],\"type\":\"LoadBalancer\",\"sessionAffinity\":\"None\",\"loadBalancerIP\":\"x.x.x.x\",\"externalTrafficPolicy\":\"Cluster\",\"ipFamilies\":[\"IPv4\"],\"ipFamilyPolicy\":\"SingleStack\",\"allocateLoadBalancerNodePorts\":true,\"internalTrafficPolicy\":\"Cluster\"},\"status\":{\"loadBalancer\":{\"ingress\":[{\"ip\":\"x.x.x.x\"}]}}}"}
2024-02-05T10:21:31.38302467Z   INFO    renderer    update (gen: 2060): upsert-queue: gway-cls: 0, gway: 1, route: 3, routeV1A2: 0, svc: 1, confmap: 0, dp: 0 / delete-queue: gway-cls: 0, gway: 0, route: 0, routeV1A2: 0, svc: 0, confmap: 0, dp: 0 / config-queue: 0
2024-02-05T10:21:31.383274722Z  INFO    renderer    STUNner dataplane Deployment ready  {"generation": 2060, "deployment": "{\"metadata\":{\"name\":\"udp-gateway\",\"namespace\":\"stunner\",\"creationTimestamp\":null,\"labels\":{\"stunner.l7mp.io/owned-by\":\"stunner\",\"stunner.l7mp.io/related-gateway-name\":\"udp-gateway\",\"stunner.l7mp.io/related-gateway-namespace\":\"stunner\"},\"annotations\":{\"stunner.l7mp.io/related-gateway-name\":\"stunner/udp-gateway\"},\"ownerReferences\":[{\"apiVersion\":\"gateway.networking.k8s.io/v1\",\"kind\":\"Gateway\",\"name\":\"udp-gateway\",\"uid\":\"d1cf77dc-ac96-492e-b9df-31d9d47ad24b\"}]},\"spec\":{\"replicas\":1,\"selector\":{\"matchExpressions\":[{\"key\":\"app\",\"operator\":\"In\",\"values\":[\"stunner\"]},{\"key\":\"stunner.l7mp.io/related-gateway-name\",\"operator\":\"In\",\"values\":[\"udp-gateway\"]},{\"key\":\"stunner.l7mp.io/related-gateway-namespace\",\"operator\":\"In\",\"values\":[\"stunner\"]}]},\"template\":{\"metadata\":{\"creationTimestamp\":null,\"labels\":{\"app\":\"stunner\",\"stunner.l7mp.io/related-gateway-name\":\"udp-gateway\",\"stunner.l7mp.io/related-gateway-namespace\":\"stunner\"},\"annotations\":{\"stunner.l7mp.io/related-gateway-name\":\"stunner/udp-gateway\"}},\"spec\":{\"containers\":[{\"name\":\"stunner-daemon\",\"image\":\"l7mp/stunnerd:0.17.0\",\"command\":[\"stunnerd\"],\"args\":[\"-w\",\"--udp-thread-num=16\"],\"env\":[{\"name\":\"STUNNER_ADDR\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"status.podIP\"}}},{\"name\":\"STUNNER_NAME\",\"value\":\"udp-gateway\"},{\"name\":\"STUNNER_NAMESPACE\",\"value\":\"stunner\"},{\"name\":\"STUNNER_CONFIG_ORIGIN\",\"value\":\"http://10.1.0.45:13478\"}],\"resources\":{\"limits\":{\"cpu\":\"2\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"500m\",\"memory\":\"128Mi\"}},\"livenessProbe\":{\"httpGet\":{\"path\":\"/live\",\"port\":8086,\"scheme\":\"HTTP\"},\"timeoutSeconds\":1,\"periodSeconds\":15,\"successThreshold\":1,\"failureThreshold\":3},\"readinessProbe\":{\"httpGet\":{\"path\":\"/ready\",\"port\":8086,\"scheme\":\"HTTP\"},\"timeoutSeconds\":1,\"periodSeconds\":15,\"successThreshold\":1,\"failureThreshold\":3},\"imagePullPolicy\":\"Always\"}],\"terminationGracePeriodSeconds\":3600}},\"strategy\":{}},\"status\":{}}"}
2024-02-05T10:21:31.383307794Z  INFO    renderer    STUNner dataplane configuration ready   {"generation": 2060, "config": "{version=\"v1\",admin:{name=\"stunner/udp-gateway\",logLevel=\"all:INFO\",health-check=\"http://:8086\"},auth:{realm=\"turn.example.com\",type=\"static\",username=\"<SECRET>\",password=\"<SECRET>\"},listeners=[\"stunner/udp-gateway/udp-listener\":{turn://0.0.0.0:3478,public=x.x.x.x:3478,cert/key=-/-,routes=[stunner/inlive-stream-55,stunner/dev-hub,stunner/hub]}],clusters=[\"stunner/inlive-stream-55\":{type=\"STATIC\",protocol=\"UDP\",endpoints=[10.152.183.251]},\"stunner/dev-hub\":{type=\"STATIC\",protocol=\"UDP\",endpoints=[10.1.0.182,10.152.183.173]},\"stunner/hub\":{type=\"STATIC\",protocol=\"UDP\",endpoints=[10.1.0.180,10.152.183.71]}]}"}
2024-02-05T10:21:31.383339504Z  INFO    cds-server  processing config update event  {"generation": 2060, "update": "update (gen: 2060): upsert-queue: gway-cls: 1, gway: 1, route: 3, routeV1A2: 0, svc: 1, confmap: 1, dp: 1 / delete-queue: gway-cls: 0, gway: 0, route: 0, routeV1A2: 0, svc: 0, confmap: 0, dp: 0 / config-queue: 1"}
2024-02-05T10:21:31.383349082Z  INFO    updater processing update event {"generation": 2060, "update": "update (gen: 2060): upsert-queue: gway-cls: 1, gway: 1, route: 3, routeV1A2: 0, svc: 1, confmap: 1, dp: 1 / delete-queue: gway-cls: 0, gway: 0, route: 0, routeV1A2: 0, svc: 0, confmap: 0, dp: 0 / config-queue: 1"}
2024-02-05T10:21:32.42207051Z   INFO    udproute-controller reconciling {"resource": "inlive/inlive-stream-55"}
2024-02-05T10:21:32.672389744Z  INFO    renderer    rendering configuration {"generation": 2061, "event": "render"}
2024-02-05T10:21:32.672418358Z  INFO    renderer    commencing dataplane render {"mode": "managed"}
2024-02-05T10:21:32.672429038Z  INFO    renderer    rendering configuration {"gateway-class": "/stunner-gatewayclass"}
2024-02-05T10:21:32.67260537Z   INFO    renderer    creating public service for gateway {"service": "stunner/udp-gateway", "gateway": "stunner/udp-gateway", "service": "{\"kind\":\"Service\",\"apiVersion\":\"v1\",\"metadata\":{\"name\":\"udp-gateway\",\"namespace\":\"stunner\",\"uid\":\"a8ce9f53-960b-4588-a3f3-ca625bbeb53a\",\"resourceVersion\":\"12285358\",\"creationTimestamp\":\"2024-01-29T17:40:57Z\",\"labels\":{\"stunner.l7mp.io/owned-by\":\"stunner\",\"stunner.l7mp.io/related-gateway-name\":\"udp-gateway\",\"stunner.l7mp.io/related-gateway-namespace\":\"stunner\"},\"annotations\":{\"metallb.universe.tf/allow-shared-ip\":\"x.x.x.x\",\"stunner.l7mp.io/related-gateway-name\":\"stunner/udp-gateway\"},\"ownerReferences\":[{\"apiVersion\":\"gateway.networking.k8s.io/v1\",\"kind\":\"Gateway\",\"name\":\"udp-gateway\",\"uid\":\"d1cf77dc-ac96-492e-b9df-31d9d47ad24b\"}]},\"spec\":{\"ports\":[{\"name\":\"udp-listener\",\"protocol\":\"UDP\",\"port\":3478,\"targetPort\":3478,\"nodePort\":32298}],\"selector\":{\"app\":\"stunner\",\"stunner.l7mp.io/related-gateway-name\":\"udp-gateway\",\"stunner.l7mp.io/related-gateway-namespace\":\"stunner\"},\"clusterIP\":\"10.152.183.130\",\"clusterIPs\":[\"10.152.183.130\"],\"type\":\"LoadBalancer\",\"sessionAffinity\":\"None\",\"loadBalancerIP\":\"x.x.x.x\",\"externalTrafficPolicy\":\"Cluster\",\"ipFamilies\":[\"IPv4\"],\"ipFamilyPolicy\":\"SingleStack\",\"allocateLoadBalancerNodePorts\":true,\"internalTrafficPolicy\":\"Cluster\"},\"status\":{\"loadBalancer\":{\"ingress\":[{\"ip\":\"x.x.x.x\"}]}}}"}
2024-02-05T10:21:32.672682626Z  INFO    renderer    update (gen: 2061): upsert-queue: gway-cls: 0, gway: 1, route: 3, routeV1A2: 0, svc: 1, confmap: 0, dp: 0 / delete-queue: gway-cls: 0, gway: 0, route: 0, routeV1A2: 0, svc: 0, confmap: 0, dp: 0 / config-queue: 0
2024-02-05T10:21:32.672885078Z  INFO    renderer    STUNner dataplane Deployment ready  {"generation": 2061, "deployment": "{\"metadata\":{\"name\":\"udp-gateway\",\"namespace\":\"stunner\",\"creationTimestamp\":null,\"labels\":{\"stunner.l7mp.io/owned-by\":\"stunner\",\"stunner.l7mp.io/related-gateway-name\":\"udp-gateway\",\"stunner.l7mp.io/related-gateway-namespace\":\"stunner\"},\"annotations\":{\"stunner.l7mp.io/related-gateway-name\":\"stunner/udp-gateway\"},\"ownerReferences\":[{\"apiVersion\":\"gateway.networking.k8s.io/v1\",\"kind\":\"Gateway\",\"name\":\"udp-gateway\",\"uid\":\"d1cf77dc-ac96-492e-b9df-31d9d47ad24b\"}]},\"spec\":{\"replicas\":1,\"selector\":{\"matchExpressions\":[{\"key\":\"app\",\"operator\":\"In\",\"values\":[\"stunner\"]},{\"key\":\"stunner.l7mp.io/related-gateway-name\",\"operator\":\"In\",\"values\":[\"udp-gateway\"]},{\"key\":\"stunner.l7mp.io/related-gateway-namespace\",\"operator\":\"In\",\"values\":[\"stunner\"]}]},\"template\":{\"metadata\":{\"creationTimestamp\":null,\"labels\":{\"app\":\"stunner\",\"stunner.l7mp.io/related-gateway-name\":\"udp-gateway\",\"stunner.l7mp.io/related-gateway-namespace\":\"stunner\"},\"annotations\":{\"stunner.l7mp.io/related-gateway-name\":\"stunner/udp-gateway\"}},\"spec\":{\"containers\":[{\"name\":\"stunner-daemon\",\"image\":\"l7mp/stunnerd:0.17.0\",\"command\":[\"stunnerd\"],\"args\":[\"-w\",\"--udp-thread-num=16\"],\"env\":[{\"name\":\"STUNNER_ADDR\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"status.podIP\"}}},{\"name\":\"STUNNER_NAME\",\"value\":\"udp-gateway\"},{\"name\":\"STUNNER_NAMESPACE\",\"value\":\"stunner\"},{\"name\":\"STUNNER_CONFIG_ORIGIN\",\"value\":\"http://10.1.0.45:13478\"}],\"resources\":{\"limits\":{\"cpu\":\"2\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"500m\",\"memory\":\"128Mi\"}},\"livenessProbe\":{\"httpGet\":{\"path\":\"/live\",\"port\":8086,\"scheme\":\"HTTP\"},\"timeoutSeconds\":1,\"periodSeconds\":15,\"successThreshold\":1,\"failureThreshold\":3},\"readinessProbe\":{\"httpGet\":{\"path\":\"/ready\",\"port\":8086,\"scheme\":\"HTTP\"},\"timeoutSeconds\":1,\"periodSeconds\":15,\"successThreshold\":1,\"failureThreshold\":3},\"imagePullPolicy\":\"Always\"}],\"terminationGracePeriodSeconds\":3600}},\"strategy\":{}},\"status\":{}}"}
2024-02-05T10:21:32.672911848Z  INFO    renderer    STUNner dataplane configuration ready   {"generation": 2061, "config": "{version=\"v1\",admin:{name=\"stunner/udp-gateway\",logLevel=\"all:INFO\",health-check=\"http://:8086\"},auth:{realm=\"turn.example.com\",type=\"static\",username=\"<SECRET>\",password=\"<SECRET>\"},listeners=[\"stunner/udp-gateway/udp-listener\":{turn://0.0.0.0:3478,public=x.x.x.x:3478,cert/key=-/-,routes=[stunner/inlive-stream-55,stunner/dev-hub,stunner/hub]}],clusters=[\"stunner/inlive-stream-55\":{type=\"STATIC\",protocol=\"UDP\",endpoints=[10.1.0.249,10.152.183.251]},\"stunner/dev-hub\":{type=\"STATIC\",protocol=\"UDP\",endpoints=[10.1.0.182,10.152.183.173]},\"stunner/hub\":{type=\"STATIC\",protocol=\"UDP\",endpoints=[10.1.0.180,10.152.183.71]}]}"}
2024-02-05T10:21:32.672941254Z  INFO    cds-server  processing config update event  {"generation": 2061, "update": "update (gen: 2061): upsert-queue: gway-cls: 1, gway: 1, route: 3, routeV1A2: 0, svc: 1, confmap: 1, dp: 1 / delete-queue: gway-cls: 0, gway: 0, route: 0, routeV1A2: 0, svc: 0, confmap: 0, dp: 0 / config-queue: 1"}
2024-02-05T10:21:32.672971591Z  INFO    updater processing update event {"generation": 2061, "update": "update (gen: 2061): upsert-queue: gway-cls: 1, gway: 1, route: 3, routeV1A2: 0, svc: 1, confmap: 1, dp: 1 / delete-queue: gway-cls: 0, gway: 0, route: 0, routeV1A2: 0, svc: 0, confmap: 0, dp: 0 / config-queue: 1"}

And the gateway logs:

09:49:59.136396 reconcile.go:112: stunner INFO: setting loglevel to "all:INFO"
09:49:59.136435 reconcile.go:176: stunner INFO: reconciliation ready: new objects: 0, changed objects: 1, deleted objects: 0, started objects: 0, restarted objects: 0
09:49:59.136445 reconcile.go:180: stunner INFO: status: READY, realm: turn.replaced.com, authentication: static, listeners: stunner/udp-gateway/udp-listener: [turn-udp://10.1.0.78:3478<0:0>], active allocations: 0
10:21:31.383928 reconcile.go:112: stunner INFO: setting loglevel to "all:INFO"
10:21:31.383976 reconcile.go:176: stunner INFO: reconciliation ready: new objects: 1, changed objects: 1, deleted objects: 0, started objects: 0, restarted objects: 0
10:21:31.383986 reconcile.go:180: stunner INFO: status: READY, realm: turn.replaced.com, authentication: static, listeners: stunner/udp-gateway/udp-listener: [turn-udp://10.1.0.78:3478<0:0>], active allocations: 0
10:21:32.116444 handlers.go:25: stunner-auth INFO: static auth request: username="inlive" realm="turn.replaced.com" srcAddr=10.1.0.114:55365
10:21:32.673512 reconcile.go:112: stunner INFO: setting loglevel to "all:INFO"
10:21:32.673553 reconcile.go:176: stunner INFO: reconciliation ready: new objects: 0, changed objects: 2, deleted objects: 0, started objects: 0, restarted objects: 0
10:21:32.673563 reconcile.go:180: stunner INFO: status: READY, realm: turn.replaced.com, authentication: static, listeners: stunner/udp-gateway/udp-listener: [turn-udp://10.1.0.78:3478<0:0>], active allocations: 1
10:21:32.726985 allocation.go:283: turn INFO: No Permission or Channel exists for 10.1.0.249:50005 on allocation 10.1.0.78:36118
10:21:32.727017 allocation.go:283: turn INFO: No Permission or Channel exists for 10.1.0.249:50005 on allocation 10.1.0.78:36118
10:21:32.927715 allocation.go:283: turn INFO: No Permission or Channel exists for 10.1.0.249:50005 on allocation 10.1.0.78:36118
10:26:32.003346 reconcile.go:112: stunner INFO: setting loglevel to "all:INFO"
10:26:32.003371 reconcile.go:176: stunner INFO: reconciliation ready: new objects: 0, changed objects: 1, deleted objects: 1, started objects: 0, restarted objects: 0
10:26:32.003377 reconcile.go:180: stunner INFO: status: READY, realm: turn.replaced.com, authentication: static, listeners: stunner/udp-gateway/udp-listener: [turn-udp://10.1.0.78:3478<0:0>], active allocations: 1
10:30:46.478421 reconcile.go:112: stunner INFO: setting loglevel to "all:INFO"
10:30:46.478447 reconcile.go:176: stunner INFO: reconciliation ready: new objects: 0, changed objects: 1, deleted objects: 0, started objects: 0, restarted objects: 0
10:30:46.478453 reconcile.go:180: stunner INFO: status: READY, realm: turn.replaced.com, authentication: static, listeners: stunner/udp-gateway/udp-listener: [turn-udp://10.1.0.78:3478<0:0>], active allocations: 1
10:35:53.286683 reconcile.go:112: stunner INFO: setting loglevel to "all:INFO"
10:35:53.286719 reconcile.go:176: stunner INFO: reconciliation ready: new objects: 0, changed objects: 1, deleted objects: 0, started objects: 0, restarted objects: 0
10:35:53.286729 reconcile.go:180: stunner INFO: status: READY, realm: turn.replaced.com, authentication: static, listeners: stunner/udp-gateway/udp-listener: [turn-udp://10.1.0.78:3478<0:0>], active allocations: 0
tyohan commented 9 months ago

An update. I was created UDP route before create the service. After I switch them and create service first before create the UDP route, the log of no service found in operator logs has gone. Below are the operator log with the new resource create order:

operator logs:

2024-02-05T11:47:17.40082965Z   INFO    renderer    commencing dataplane render {"mode": "managed"}
2024-02-05T11:47:17.400842684Z  INFO    renderer    rendering configuration {"gateway-class": "/stunner-gatewayclass"}
2024-02-05T11:47:17.40111653Z   INFO    renderer    creating public service for gateway {"service": "stunner/udp-gateway", "gateway": "stunner/udp-gateway", "service": "{\"kind\":\"Service\",\"apiVersion\":\"v1\",\"metadata\":{\"name\":\"udp-gateway\",\"namespace\":\"stunner\",\"uid\":\"a8ce9f53-960b-4588-a3f3-ca625bbeb53a\",\"resourceVersion\":\"12285358\",\"creationTimestamp\":\"2024-01-29T17:40:57Z\",\"labels\":{\"stunner.l7mp.io/owned-by\":\"stunner\",\"stunner.l7mp.io/related-gateway-name\":\"udp-gateway\",\"stunner.l7mp.io/related-gateway-namespace\":\"stunner\"},\"annotations\":{\"metallb.universe.tf/allow-shared-ip\":\"x.x.x.x\",\"stunner.l7mp.io/related-gateway-name\":\"stunner/udp-gateway\"},\"ownerReferences\":[{\"apiVersion\":\"gateway.networking.k8s.io/v1\",\"kind\":\"Gateway\",\"name\":\"udp-gateway\",\"uid\":\"d1cf77dc-ac96-492e-b9df-31d9d47ad24b\"}]},\"spec\":{\"ports\":[{\"name\":\"udp-listener\",\"protocol\":\"UDP\",\"port\":3478,\"targetPort\":3478,\"nodePort\":32298}],\"selector\":{\"app\":\"stunner\",\"stunner.l7mp.io/related-gateway-name\":\"udp-gateway\",\"stunner.l7mp.io/related-gateway-namespace\":\"stunner\"},\"clusterIP\":\"10.152.183.130\",\"clusterIPs\":[\"10.152.183.130\"],\"type\":\"LoadBalancer\",\"sessionAffinity\":\"None\",\"loadBalancerIP\":\"x.x.x.x\",\"externalTrafficPolicy\":\"Cluster\",\"ipFamilies\":[\"IPv4\"],\"ipFamilyPolicy\":\"SingleStack\",\"allocateLoadBalancerNodePorts\":true,\"internalTrafficPolicy\":\"Cluster\"},\"status\":{\"loadBalancer\":{\"ingress\":[{\"ip\":\"x.x.x.x\"}]}}}"}
2024-02-05T11:47:17.401210304Z  INFO    renderer    update (gen: 2079): upsert-queue: gway-cls: 0, gway: 1, route: 2, routeV1A2: 0, svc: 1, confmap: 0, dp: 0 / delete-queue: gway-cls: 0, gway: 0, route: 0, routeV1A2: 0, svc: 0, confmap: 0, dp: 0 / config-queue: 0
2024-02-05T11:47:17.401512302Z  INFO    renderer    STUNner dataplane Deployment ready  {"generation": 2079, "deployment": "{\"metadata\":{\"name\":\"udp-gateway\",\"namespace\":\"stunner\",\"creationTimestamp\":null,\"labels\":{\"stunner.l7mp.io/owned-by\":\"stunner\",\"stunner.l7mp.io/related-gateway-name\":\"udp-gateway\",\"stunner.l7mp.io/related-gateway-namespace\":\"stunner\"},\"annotations\":{\"stunner.l7mp.io/related-gateway-name\":\"stunner/udp-gateway\"},\"ownerReferences\":[{\"apiVersion\":\"gateway.networking.k8s.io/v1\",\"kind\":\"Gateway\",\"name\":\"udp-gateway\",\"uid\":\"d1cf77dc-ac96-492e-b9df-31d9d47ad24b\"}]},\"spec\":{\"replicas\":1,\"selector\":{\"matchExpressions\":[{\"key\":\"app\",\"operator\":\"In\",\"values\":[\"stunner\"]},{\"key\":\"stunner.l7mp.io/related-gateway-name\",\"operator\":\"In\",\"values\":[\"udp-gateway\"]},{\"key\":\"stunner.l7mp.io/related-gateway-namespace\",\"operator\":\"In\",\"values\":[\"stunner\"]}]},\"template\":{\"metadata\":{\"creationTimestamp\":null,\"labels\":{\"app\":\"stunner\",\"stunner.l7mp.io/related-gateway-name\":\"udp-gateway\",\"stunner.l7mp.io/related-gateway-namespace\":\"stunner\"},\"annotations\":{\"stunner.l7mp.io/related-gateway-name\":\"stunner/udp-gateway\"}},\"spec\":{\"containers\":[{\"name\":\"stunner-daemon\",\"image\":\"l7mp/stunnerd:0.17.0\",\"command\":[\"stunnerd\"],\"args\":[\"-w\",\"--udp-thread-num=16\"],\"env\":[{\"name\":\"STUNNER_ADDR\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"status.podIP\"}}},{\"name\":\"STUNNER_NAME\",\"value\":\"udp-gateway\"},{\"name\":\"STUNNER_NAMESPACE\",\"value\":\"stunner\"},{\"name\":\"STUNNER_CONFIG_ORIGIN\",\"value\":\"http://10.1.0.45:13478\"}],\"resources\":{\"limits\":{\"cpu\":\"2\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"500m\",\"memory\":\"128Mi\"}},\"livenessProbe\":{\"httpGet\":{\"path\":\"/live\",\"port\":8086,\"scheme\":\"HTTP\"},\"timeoutSeconds\":1,\"periodSeconds\":15,\"successThreshold\":1,\"failureThreshold\":3},\"readinessProbe\":{\"httpGet\":{\"path\":\"/ready\",\"port\":8086,\"scheme\":\"HTTP\"},\"timeoutSeconds\":1,\"periodSeconds\":15,\"successThreshold\":1,\"failureThreshold\":3},\"imagePullPolicy\":\"Always\"}],\"terminationGracePeriodSeconds\":3600}},\"strategy\":{}},\"status\":{}}"}
2024-02-05T11:47:17.4015494Z    INFO    renderer    STUNner dataplane configuration ready   {"generation": 2079, "config": "{version=\"v1\",admin:{name=\"stunner/udp-gateway\",logLevel=\"all:INFO\",health-check=\"http://:8086\"},auth:{realm=\"turn.example.com\",type=\"static\",username=\"<SECRET>\",password=\"<SECRET>\"},listeners=[\"stunner/udp-gateway/udp-listener\":{turn://0.0.0.0:3478,public=x.x.x.x:3478,cert/key=-/-,routes=[stunner/dev-hub,stunner/hub]}],clusters=[\"stunner/dev-hub\":{type=\"STATIC\",protocol=\"UDP\",endpoints=[10.1.0.182,10.152.183.173]},\"stunner/hub\":{type=\"STATIC\",protocol=\"UDP\",endpoints=[10.1.0.180,10.152.183.71]}]}"}
2024-02-05T11:47:17.401592009Z  INFO    cds-server  processing config update event  {"generation": 2079, "update": "update (gen: 2079): upsert-queue: gway-cls: 1, gway: 1, route: 2, routeV1A2: 0, svc: 1, confmap: 1, dp: 1 / delete-queue: gway-cls: 0, gway: 0, route: 0, routeV1A2: 0, svc: 0, confmap: 0, dp: 0 / config-queue: 1"}
2024-02-05T11:47:17.401609682Z  INFO    updater processing update event {"generation": 2079, "update": "update (gen: 2079): upsert-queue: gway-cls: 1, gway: 1, route: 2, routeV1A2: 0, svc: 1, confmap: 1, dp: 1 / delete-queue: gway-cls: 0, gway: 0, route: 0, routeV1A2: 0, svc: 0, confmap: 0, dp: 0 / config-queue: 1"}
2024-02-05T11:52:22.874162731Z  INFO    node-controller reconciling {"node": "/node01"}
2024-02-05T11:52:22.874238201Z  INFO    node-controller failed to find node with valid external address {"reason": "end of node list reached after searching through 1 node(s)"}
2024-02-05T11:52:23.124608365Z  INFO    renderer    rendering configuration {"generation": 2080, "event": "render"}
2024-02-05T11:52:23.12462727Z   INFO    renderer    commencing dataplane render {"mode": "managed"}
2024-02-05T11:52:23.124636507Z  INFO    renderer    rendering configuration {"gateway-class": "/stunner-gatewayclass"}
2024-02-05T11:52:23.124812783Z  INFO    renderer    creating public service for gateway {"service": "stunner/udp-gateway", "gateway": "stunner/udp-gateway", "service": "{\"kind\":\"Service\",\"apiVersion\":\"v1\",\"metadata\":{\"name\":\"udp-gateway\",\"namespace\":\"stunner\",\"uid\":\"a8ce9f53-960b-4588-a3f3-ca625bbeb53a\",\"resourceVersion\":\"12285358\",\"creationTimestamp\":\"2024-01-29T17:40:57Z\",\"labels\":{\"stunner.l7mp.io/owned-by\":\"stunner\",\"stunner.l7mp.io/related-gateway-name\":\"udp-gateway\",\"stunner.l7mp.io/related-gateway-namespace\":\"stunner\"},\"annotations\":{\"metallb.universe.tf/allow-shared-ip\":\"x.x.x.x\",\"stunner.l7mp.io/related-gateway-name\":\"stunner/udp-gateway\"},\"ownerReferences\":[{\"apiVersion\":\"gateway.networking.k8s.io/v1\",\"kind\":\"Gateway\",\"name\":\"udp-gateway\",\"uid\":\"d1cf77dc-ac96-492e-b9df-31d9d47ad24b\"}]},\"spec\":{\"ports\":[{\"name\":\"udp-listener\",\"protocol\":\"UDP\",\"port\":3478,\"targetPort\":3478,\"nodePort\":32298}],\"selector\":{\"app\":\"stunner\",\"stunner.l7mp.io/related-gateway-name\":\"udp-gateway\",\"stunner.l7mp.io/related-gateway-namespace\":\"stunner\"},\"clusterIP\":\"10.152.183.130\",\"clusterIPs\":[\"10.152.183.130\"],\"type\":\"LoadBalancer\",\"sessionAffinity\":\"None\",\"loadBalancerIP\":\"x.x.x.x\",\"externalTrafficPolicy\":\"Cluster\",\"ipFamilies\":[\"IPv4\"],\"ipFamilyPolicy\":\"SingleStack\",\"allocateLoadBalancerNodePorts\":true,\"internalTrafficPolicy\":\"Cluster\"},\"status\":{\"loadBalancer\":{\"ingress\":[{\"ip\":\"x.x.x.x\"}]}}}"}
2024-02-05T11:52:23.124894324Z  INFO    renderer    update (gen: 2080): upsert-queue: gway-cls: 0, gway: 1, route: 2, routeV1A2: 0, svc: 1, confmap: 0, dp: 0 / delete-queue: gway-cls: 0, gway: 0, route: 0, routeV1A2: 0, svc: 0, confmap: 0, dp: 0 / config-queue: 0
2024-02-05T11:52:23.125133416Z  INFO    renderer    STUNner dataplane Deployment ready  {"generation": 2080, "deployment": "{\"metadata\":{\"name\":\"udp-gateway\",\"namespace\":\"stunner\",\"creationTimestamp\":null,\"labels\":{\"stunner.l7mp.io/owned-by\":\"stunner\",\"stunner.l7mp.io/related-gateway-name\":\"udp-gateway\",\"stunner.l7mp.io/related-gateway-namespace\":\"stunner\"},\"annotations\":{\"stunner.l7mp.io/related-gateway-name\":\"stunner/udp-gateway\"},\"ownerReferences\":[{\"apiVersion\":\"gateway.networking.k8s.io/v1\",\"kind\":\"Gateway\",\"name\":\"udp-gateway\",\"uid\":\"d1cf77dc-ac96-492e-b9df-31d9d47ad24b\"}]},\"spec\":{\"replicas\":1,\"selector\":{\"matchExpressions\":[{\"key\":\"app\",\"operator\":\"In\",\"values\":[\"stunner\"]},{\"key\":\"stunner.l7mp.io/related-gateway-name\",\"operator\":\"In\",\"values\":[\"udp-gateway\"]},{\"key\":\"stunner.l7mp.io/related-gateway-namespace\",\"operator\":\"In\",\"values\":[\"stunner\"]}]},\"template\":{\"metadata\":{\"creationTimestamp\":null,\"labels\":{\"app\":\"stunner\",\"stunner.l7mp.io/related-gateway-name\":\"udp-gateway\",\"stunner.l7mp.io/related-gateway-namespace\":\"stunner\"},\"annotations\":{\"stunner.l7mp.io/related-gateway-name\":\"stunner/udp-gateway\"}},\"spec\":{\"containers\":[{\"name\":\"stunner-daemon\",\"image\":\"l7mp/stunnerd:0.17.0\",\"command\":[\"stunnerd\"],\"args\":[\"-w\",\"--udp-thread-num=16\"],\"env\":[{\"name\":\"STUNNER_ADDR\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"status.podIP\"}}},{\"name\":\"STUNNER_NAME\",\"value\":\"udp-gateway\"},{\"name\":\"STUNNER_NAMESPACE\",\"value\":\"stunner\"},{\"name\":\"STUNNER_CONFIG_ORIGIN\",\"value\":\"http://10.1.0.45:13478\"}],\"resources\":{\"limits\":{\"cpu\":\"2\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"500m\",\"memory\":\"128Mi\"}},\"livenessProbe\":{\"httpGet\":{\"path\":\"/live\",\"port\":8086,\"scheme\":\"HTTP\"},\"timeoutSeconds\":1,\"periodSeconds\":15,\"successThreshold\":1,\"failureThreshold\":3},\"readinessProbe\":{\"httpGet\":{\"path\":\"/ready\",\"port\":8086,\"scheme\":\"HTTP\"},\"timeoutSeconds\":1,\"periodSeconds\":15,\"successThreshold\":1,\"failureThreshold\":3},\"imagePullPolicy\":\"Always\"}],\"terminationGracePeriodSeconds\":3600}},\"strategy\":{}},\"status\":{}}"}
2024-02-05T11:52:23.125163923Z  INFO    renderer    STUNner dataplane configuration ready   {"generation": 2080, "config": "{version=\"v1\",admin:{name=\"stunner/udp-gateway\",logLevel=\"all:INFO\",health-check=\"http://:8086\"},auth:{realm=\"turn.example.com\",type=\"static\",username=\"<SECRET>\",password=\"<SECRET>\"},listeners=[\"stunner/udp-gateway/udp-listener\":{turn://0.0.0.0:3478,public=x.x.x.x:3478,cert/key=-/-,routes=[stunner/dev-hub,stunner/hub]}],clusters=[\"stunner/dev-hub\":{type=\"STATIC\",protocol=\"UDP\",endpoints=[10.1.0.182,10.152.183.173]},\"stunner/hub\":{type=\"STATIC\",protocol=\"UDP\",endpoints=[10.1.0.180,10.152.183.71]}]}"}
2024-02-05T11:52:23.12519466Z   INFO    cds-server  processing config update event  {"generation": 2080, "update": "update (gen: 2080): upsert-queue: gway-cls: 1, gway: 1, route: 2, routeV1A2: 0, svc: 1, confmap: 1, dp: 1 / delete-queue: gway-cls: 0, gway: 0, route: 0, routeV1A2: 0, svc: 0, confmap: 0, dp: 0 / config-queue: 1"}
2024-02-05T11:52:23.125216089Z  INFO    updater processing update event {"generation": 2080, "update": "update (gen: 2080): upsert-queue: gway-cls: 1, gway: 1, route: 2, routeV1A2: 0, svc: 1, confmap: 1, dp: 1 / delete-queue: gway-cls: 0, gway: 0, route: 0, routeV1A2: 0, svc: 0, confmap: 0, dp: 0 / config-queue: 1"}
2024-02-05T11:55:57.219539741Z  INFO    udproute-controller reconciling {"resource": "stunner/inlive-stream-56"}
2024-02-05T11:55:57.46974478Z   INFO    renderer    rendering configuration {"generation": 2081, "event": "render"}
2024-02-05T11:55:57.469776689Z  INFO    renderer    commencing dataplane render {"mode": "managed"}
2024-02-05T11:55:57.469787319Z  INFO    renderer    rendering configuration {"gateway-class": "/stunner-gatewayclass"}
2024-02-05T11:55:57.47001471Z   INFO    renderer    creating public service for gateway {"service": "stunner/udp-gateway", "gateway": "stunner/udp-gateway", "service": "{\"kind\":\"Service\",\"apiVersion\":\"v1\",\"metadata\":{\"name\":\"udp-gateway\",\"namespace\":\"stunner\",\"uid\":\"a8ce9f53-960b-4588-a3f3-ca625bbeb53a\",\"resourceVersion\":\"12285358\",\"creationTimestamp\":\"2024-01-29T17:40:57Z\",\"labels\":{\"stunner.l7mp.io/owned-by\":\"stunner\",\"stunner.l7mp.io/related-gateway-name\":\"udp-gateway\",\"stunner.l7mp.io/related-gateway-namespace\":\"stunner\"},\"annotations\":{\"metallb.universe.tf/allow-shared-ip\":\"x.x.x.x\",\"stunner.l7mp.io/related-gateway-name\":\"stunner/udp-gateway\"},\"ownerReferences\":[{\"apiVersion\":\"gateway.networking.k8s.io/v1\",\"kind\":\"Gateway\",\"name\":\"udp-gateway\",\"uid\":\"d1cf77dc-ac96-492e-b9df-31d9d47ad24b\"}]},\"spec\":{\"ports\":[{\"name\":\"udp-listener\",\"protocol\":\"UDP\",\"port\":3478,\"targetPort\":3478,\"nodePort\":32298}],\"selector\":{\"app\":\"stunner\",\"stunner.l7mp.io/related-gateway-name\":\"udp-gateway\",\"stunner.l7mp.io/related-gateway-namespace\":\"stunner\"},\"clusterIP\":\"10.152.183.130\",\"clusterIPs\":[\"10.152.183.130\"],\"type\":\"LoadBalancer\",\"sessionAffinity\":\"None\",\"loadBalancerIP\":\"x.x.x.x\",\"externalTrafficPolicy\":\"Cluster\",\"ipFamilies\":[\"IPv4\"],\"ipFamilyPolicy\":\"SingleStack\",\"allocateLoadBalancerNodePorts\":true,\"internalTrafficPolicy\":\"Cluster\"},\"status\":{\"loadBalancer\":{\"ingress\":[{\"ip\":\"x.x.x.x\"}]}}}"}
2024-02-05T11:55:57.470102863Z  INFO    renderer    update (gen: 2081): upsert-queue: gway-cls: 0, gway: 1, route: 3, routeV1A2: 0, svc: 1, confmap: 0, dp: 0 / delete-queue: gway-cls: 0, gway: 0, route: 0, routeV1A2: 0, svc: 0, confmap: 0, dp: 0 / config-queue: 0
2024-02-05T11:55:57.470344209Z  INFO    renderer    STUNner dataplane Deployment ready  {"generation": 2081, "deployment": "{\"metadata\":{\"name\":\"udp-gateway\",\"namespace\":\"stunner\",\"creationTimestamp\":null,\"labels\":{\"stunner.l7mp.io/owned-by\":\"stunner\",\"stunner.l7mp.io/related-gateway-name\":\"udp-gateway\",\"stunner.l7mp.io/related-gateway-namespace\":\"stunner\"},\"annotations\":{\"stunner.l7mp.io/related-gateway-name\":\"stunner/udp-gateway\"},\"ownerReferences\":[{\"apiVersion\":\"gateway.networking.k8s.io/v1\",\"kind\":\"Gateway\",\"name\":\"udp-gateway\",\"uid\":\"d1cf77dc-ac96-492e-b9df-31d9d47ad24b\"}]},\"spec\":{\"replicas\":1,\"selector\":{\"matchExpressions\":[{\"key\":\"app\",\"operator\":\"In\",\"values\":[\"stunner\"]},{\"key\":\"stunner.l7mp.io/related-gateway-name\",\"operator\":\"In\",\"values\":[\"udp-gateway\"]},{\"key\":\"stunner.l7mp.io/related-gateway-namespace\",\"operator\":\"In\",\"values\":[\"stunner\"]}]},\"template\":{\"metadata\":{\"creationTimestamp\":null,\"labels\":{\"app\":\"stunner\",\"stunner.l7mp.io/related-gateway-name\":\"udp-gateway\",\"stunner.l7mp.io/related-gateway-namespace\":\"stunner\"},\"annotations\":{\"stunner.l7mp.io/related-gateway-name\":\"stunner/udp-gateway\"}},\"spec\":{\"containers\":[{\"name\":\"stunner-daemon\",\"image\":\"l7mp/stunnerd:0.17.0\",\"command\":[\"stunnerd\"],\"args\":[\"-w\",\"--udp-thread-num=16\"],\"env\":[{\"name\":\"STUNNER_ADDR\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"status.podIP\"}}},{\"name\":\"STUNNER_NAME\",\"value\":\"udp-gateway\"},{\"name\":\"STUNNER_NAMESPACE\",\"value\":\"stunner\"},{\"name\":\"STUNNER_CONFIG_ORIGIN\",\"value\":\"http://10.1.0.45:13478\"}],\"resources\":{\"limits\":{\"cpu\":\"2\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"500m\",\"memory\":\"128Mi\"}},\"livenessProbe\":{\"httpGet\":{\"path\":\"/live\",\"port\":8086,\"scheme\":\"HTTP\"},\"timeoutSeconds\":1,\"periodSeconds\":15,\"successThreshold\":1,\"failureThreshold\":3},\"readinessProbe\":{\"httpGet\":{\"path\":\"/ready\",\"port\":8086,\"scheme\":\"HTTP\"},\"timeoutSeconds\":1,\"periodSeconds\":15,\"successThreshold\":1,\"failureThreshold\":3},\"imagePullPolicy\":\"Always\"}],\"terminationGracePeriodSeconds\":3600}},\"strategy\":{}},\"status\":{}}"}
2024-02-05T11:55:57.470378623Z  INFO    renderer    STUNner dataplane configuration ready   {"generation": 2081, "config": "{version=\"v1\",admin:{name=\"stunner/udp-gateway\",logLevel=\"all:INFO\",health-check=\"http://:8086\"},auth:{realm=\"turn.example.com\",type=\"static\",username=\"<SECRET>\",password=\"<SECRET>\"},listeners=[\"stunner/udp-gateway/udp-listener\":{turn://0.0.0.0:3478,public=x.x.x.x:3478,cert/key=-/-,routes=[stunner/inlive-stream-56,stunner/dev-hub,stunner/hub]}],clusters=[\"stunner/dev-hub\":{type=\"STATIC\",protocol=\"UDP\",endpoints=[10.1.0.182,10.152.183.173]},\"stunner/hub\":{type=\"STATIC\",protocol=\"UDP\",endpoints=[10.1.0.180,10.152.183.71]},\"stunner/inlive-stream-56\":{type=\"STATIC\",protocol=\"UDP\",endpoints=[10.152.183.87]}]}"}
2024-02-05T11:55:57.470417716Z  INFO    cds-server  processing config update event  {"generation": 2081, "update": "update (gen: 2081): upsert-queue: gway-cls: 1, gway: 1, route: 3, routeV1A2: 0, svc: 1, confmap: 1, dp: 1 / delete-queue: gway-cls: 0, gway: 0, route: 0, routeV1A2: 0, svc: 0, confmap: 0, dp: 0 / config-queue: 1"}
2024-02-05T11:55:57.470427874Z  INFO    updater processing update event {"generation": 2081, "update": "update (gen: 2081): upsert-queue: gway-cls: 1, gway: 1, route: 3, routeV1A2: 0, svc: 1, confmap: 1, dp: 1 / delete-queue: gway-cls: 0, gway: 0, route: 0, routeV1A2: 0, svc: 0, confmap: 0, dp: 0 / config-queue: 1"}
2024-02-05T11:55:57.91787645Z   INFO    udproute-controller reconciling {"resource": "inlive/inlive-stream-56"}
2024-02-05T11:55:58.168959173Z  INFO    renderer    rendering configuration {"generation": 2082, "event": "render"}
2024-02-05T11:55:58.168975153Z  INFO    renderer    commencing dataplane render {"mode": "managed"}
2024-02-05T11:55:58.168982997Z  INFO    renderer    rendering configuration {"gateway-class": "/stunner-gatewayclass"}
2024-02-05T11:55:58.169144306Z  INFO    renderer    creating public service for gateway {"service": "stunner/udp-gateway", "gateway": "stunner/udp-gateway", "service": "{\"kind\":\"Service\",\"apiVersion\":\"v1\",\"metadata\":{\"name\":\"udp-gateway\",\"namespace\":\"stunner\",\"uid\":\"a8ce9f53-960b-4588-a3f3-ca625bbeb53a\",\"resourceVersion\":\"12285358\",\"creationTimestamp\":\"2024-01-29T17:40:57Z\",\"labels\":{\"stunner.l7mp.io/owned-by\":\"stunner\",\"stunner.l7mp.io/related-gateway-name\":\"udp-gateway\",\"stunner.l7mp.io/related-gateway-namespace\":\"stunner\"},\"annotations\":{\"metallb.universe.tf/allow-shared-ip\":\"x.x.x.x\",\"stunner.l7mp.io/related-gateway-name\":\"stunner/udp-gateway\"},\"ownerReferences\":[{\"apiVersion\":\"gateway.networking.k8s.io/v1\",\"kind\":\"Gateway\",\"name\":\"udp-gateway\",\"uid\":\"d1cf77dc-ac96-492e-b9df-31d9d47ad24b\"}]},\"spec\":{\"ports\":[{\"name\":\"udp-listener\",\"protocol\":\"UDP\",\"port\":3478,\"targetPort\":3478,\"nodePort\":32298}],\"selector\":{\"app\":\"stunner\",\"stunner.l7mp.io/related-gateway-name\":\"udp-gateway\",\"stunner.l7mp.io/related-gateway-namespace\":\"stunner\"},\"clusterIP\":\"10.152.183.130\",\"clusterIPs\":[\"10.152.183.130\"],\"type\":\"LoadBalancer\",\"sessionAffinity\":\"None\",\"loadBalancerIP\":\"x.x.x.x\",\"externalTrafficPolicy\":\"Cluster\",\"ipFamilies\":[\"IPv4\"],\"ipFamilyPolicy\":\"SingleStack\",\"allocateLoadBalancerNodePorts\":true,\"internalTrafficPolicy\":\"Cluster\"},\"status\":{\"loadBalancer\":{\"ingress\":[{\"ip\":\"x.x.x.x\"}]}}}"}
2024-02-05T11:55:58.16921701Z   INFO    renderer    update (gen: 2082): upsert-queue: gway-cls: 0, gway: 1, route: 3, routeV1A2: 0, svc: 1, confmap: 0, dp: 0 / delete-queue: gway-cls: 0, gway: 0, route: 0, routeV1A2: 0, svc: 0, confmap: 0, dp: 0 / config-queue: 0
2024-02-05T11:55:58.16943798Z   INFO    renderer    STUNner dataplane Deployment ready  {"generation": 2082, "deployment": "{\"metadata\":{\"name\":\"udp-gateway\",\"namespace\":\"stunner\",\"creationTimestamp\":null,\"labels\":{\"stunner.l7mp.io/owned-by\":\"stunner\",\"stunner.l7mp.io/related-gateway-name\":\"udp-gateway\",\"stunner.l7mp.io/related-gateway-namespace\":\"stunner\"},\"annotations\":{\"stunner.l7mp.io/related-gateway-name\":\"stunner/udp-gateway\"},\"ownerReferences\":[{\"apiVersion\":\"gateway.networking.k8s.io/v1\",\"kind\":\"Gateway\",\"name\":\"udp-gateway\",\"uid\":\"d1cf77dc-ac96-492e-b9df-31d9d47ad24b\"}]},\"spec\":{\"replicas\":1,\"selector\":{\"matchExpressions\":[{\"key\":\"app\",\"operator\":\"In\",\"values\":[\"stunner\"]},{\"key\":\"stunner.l7mp.io/related-gateway-name\",\"operator\":\"In\",\"values\":[\"udp-gateway\"]},{\"key\":\"stunner.l7mp.io/related-gateway-namespace\",\"operator\":\"In\",\"values\":[\"stunner\"]}]},\"template\":{\"metadata\":{\"creationTimestamp\":null,\"labels\":{\"app\":\"stunner\",\"stunner.l7mp.io/related-gateway-name\":\"udp-gateway\",\"stunner.l7mp.io/related-gateway-namespace\":\"stunner\"},\"annotations\":{\"stunner.l7mp.io/related-gateway-name\":\"stunner/udp-gateway\"}},\"spec\":{\"containers\":[{\"name\":\"stunner-daemon\",\"image\":\"l7mp/stunnerd:0.17.0\",\"command\":[\"stunnerd\"],\"args\":[\"-w\",\"--udp-thread-num=16\"],\"env\":[{\"name\":\"STUNNER_ADDR\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"status.podIP\"}}},{\"name\":\"STUNNER_NAME\",\"value\":\"udp-gateway\"},{\"name\":\"STUNNER_NAMESPACE\",\"value\":\"stunner\"},{\"name\":\"STUNNER_CONFIG_ORIGIN\",\"value\":\"http://10.1.0.45:13478\"}],\"resources\":{\"limits\":{\"cpu\":\"2\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"500m\",\"memory\":\"128Mi\"}},\"livenessProbe\":{\"httpGet\":{\"path\":\"/live\",\"port\":8086,\"scheme\":\"HTTP\"},\"timeoutSeconds\":1,\"periodSeconds\":15,\"successThreshold\":1,\"failureThreshold\":3},\"readinessProbe\":{\"httpGet\":{\"path\":\"/ready\",\"port\":8086,\"scheme\":\"HTTP\"},\"timeoutSeconds\":1,\"periodSeconds\":15,\"successThreshold\":1,\"failureThreshold\":3},\"imagePullPolicy\":\"Always\"}],\"terminationGracePeriodSeconds\":3600}},\"strategy\":{}},\"status\":{}}"}
2024-02-05T11:55:58.169462475Z  INFO    renderer    STUNner dataplane configuration ready   {"generation": 2082, "config": "{version=\"v1\",admin:{name=\"stunner/udp-gateway\",logLevel=\"all:INFO\",health-check=\"http://:8086\"},auth:{realm=\"turn.example.com\",type=\"static\",username=\"<SECRET>\",password=\"<SECRET>\"},listeners=[\"stunner/udp-gateway/udp-listener\":{turn://0.0.0.0:3478,public=x.x.x.x:3478,cert/key=-/-,routes=[stunner/dev-hub,stunner/hub,stunner/inlive-stream-56]}],clusters=[\"stunner/dev-hub\":{type=\"STATIC\",protocol=\"UDP\",endpoints=[10.1.0.182,10.152.183.173]},\"stunner/hub\":{type=\"STATIC\",protocol=\"UDP\",endpoints=[10.1.0.180,10.152.183.71]},\"stunner/inlive-stream-56\":{type=\"STATIC\",protocol=\"UDP\",endpoints=[10.1.0.199,10.152.183.87]}]}"}
2024-02-05T11:55:58.169488723Z  INFO    cds-server  processing config update event  {"generation": 2082, "update": "update (gen: 2082): upsert-queue: gway-cls: 1, gway: 1, route: 3, routeV1A2: 0, svc: 1, confmap: 1, dp: 1 / delete-queue: gway-cls: 0, gway: 0, route: 0, routeV1A2: 0, svc: 0, confmap: 0, dp: 0 / config-queue: 1"}
2024-02-05T11:55:58.169522967Z  INFO    updater processing update event {"generation": 2082, "update": "update (gen: 2082): upsert-queue: gway-cls: 1, gway: 1, route: 3, routeV1A2: 0, svc: 1, confmap: 1, dp: 1 / delete-queue: gway-cls: 0, gway: 0, route: 0, routeV1A2: 0, svc: 0, confmap: 0, dp: 0 / config-queue: 1"}

And the gateway log

11:37:06.117350 reconcile.go:112: stunner INFO: setting loglevel to "all:INFO"
11:37:06.117383 reconcile.go:176: stunner INFO: reconciliation ready: new objects: 0, changed objects: 1, deleted objects: 0, started objects: 0, restarted objects: 0
11:37:06.117393 reconcile.go:180: stunner INFO: status: READY, realm: turn.example.com, authentication: static, listeners: stunner/udp-gateway/udp-listener: [turn-udp://10.1.0.78:3478<0:0>], active allocations: 0
11:47:17.402316 reconcile.go:112: stunner INFO: setting loglevel to "all:INFO"
11:47:17.402365 reconcile.go:176: stunner INFO: reconciliation ready: new objects: 0, changed objects: 1, deleted objects: 0, started objects: 0, restarted objects: 0
11:47:17.402377 reconcile.go:180: stunner INFO: status: READY, realm: turn.example.com, authentication: static, listeners: stunner/udp-gateway/udp-listener: [turn-udp://10.1.0.78:3478<0:0>], active allocations: 0
11:55:57.470979 reconcile.go:112: stunner INFO: setting loglevel to "all:INFO"
11:55:57.471041 reconcile.go:176: stunner INFO: reconciliation ready: new objects: 1, changed objects: 1, deleted objects: 0, started objects: 0, restarted objects: 0
11:55:57.471051 reconcile.go:180: stunner INFO: status: READY, realm: turn.example.com, authentication: static, listeners: stunner/udp-gateway/udp-listener: [turn-udp://10.1.0.78:3478<0:0>], active allocations: 0
11:55:58.169989 reconcile.go:112: stunner INFO: setting loglevel to "all:INFO"
11:55:58.170027 reconcile.go:176: stunner INFO: reconciliation ready: new objects: 0, changed objects: 2, deleted objects: 0, started objects: 0, restarted objects: 0
11:55:58.170036 reconcile.go:180: stunner INFO: status: READY, realm: turn.example.com, authentication: static, listeners: stunner/udp-gateway/udp-listener: [turn-udp://10.1.0.78:3478<0:0>], active allocations: 0
11:55:58.289758 handlers.go:25: stunner-auth INFO: static auth request: username="inlive" realm="turn.example.com" srcAddr=10.1.0.114:51112
11:55:58.296422 allocation.go:283: turn INFO: No Permission or Channel exists for 10.1.0.199:50005 on allocation 10.1.0.78:34464 (suppressed 7 log events)
11:55:58.296444 allocation.go:283: turn INFO: No Permission or Channel exists for 10.1.0.199:50005 on allocation 10.1.0.78:34464
11:55:58.497181 allocation.go:283: turn INFO: No Permission or Channel exists for 10.1.0.199:50005 on allocation 10.1.0.78:34464
11:55:59.300481 allocation.go:283: turn INFO: No Permission or Channel exists for 10.1.0.199:50005 on allocation 10.1.0.78:34464 (suppressed 7 log events)
tyohan commented 9 months ago

Another question to clarify: I saw that when I created the UDP route successfully, it triggered the reconciliation onbthe UDP gateway. I wonder if I should wait for a moment, until everything is ready to relay the connection to the new pod. Because my app will trying to connect instantly once the creation of the pod, service, and UDP route through the API endpoint has completed.

rg0now commented 9 months ago

As per UDPRoute and Service availability: you don't have to manually synchronize the creation of the resources (in fact, in Kubernetes you never have to). If the UDPRoute is created before the Service STUNner will wait until the Service becomes ready and, once this happens, it updates the dataplane config accordingly. You can create anything in any order and things will just work.

rg0now commented 9 months ago

This is terribly weird:

11:55:58.289758 handlers.go:25: stunner-auth INFO: static auth request: username="inlive" realm="turn.example.com" srcAddr=10.1.0.114:51112
11:55:58.296422 allocation.go:283: turn INFO: No Permission or Channel exists for 10.1.0.199:50005 on allocation 10.1.0.78:34464 (suppressed 7 log events)

Your client creates an allocation with the (NATted) source address 10.1.0.114:51112 but then it tries to send traffic from another source address (10.1.0.199:50005). This should never happen unless something weird is going on with your client or cloud load-balancer. In order to confirm this, please repeat the experiment with setting the dataplane loglevel to all:TRACE and re-post the logs. Hopefully we'll see where things go south.

Also, in order to guarantee that your WebRTC code is OK you can redeploy with a well-tested Kubernetes distro, like Minikube or a k3s, and check if everything works there. We have a good understanding of how things should happen there and then we could extrapolate from that to your specific cloud provider (which is what, BTW?).

rg0now commented 8 months ago

Closing this for now because this seems like a LoadBalancer issue (still weird). If you have new input then please reopen, we'd like to know the solution.