tsnet: unable to create multiple funnel listeners for tsnet.Server

ifross89 commented 10 months ago

What is the issue?

I am trying to create multiple funnel only listeners on a *tsnet.Server. When I serve the listeners and try and connect, I can only connect to the listener that is created last.

For example, if I try and create funnel only listeners on ports 443 and 8443, I am able to connect to the server listening on port 8443, but I am unable to connect to the server listening on port 443.

When I try and connect to the :443 server, in firefox I get: PR_END_OF_FILE_ERROR. In the logs I get:

localbackend: got ingress conn for unconfigured "test.my-tailnet.ts.net:443"; rejecting

I would expect to be able to connect to both servers.

I had a quick look, and the issue seems to be when calling LocalClient.GetServeConfig(ctx) in ListenFunnel, an empty serve config is returned, even when a previous serve config has been configured.

Steps to reproduce

func TestMultipleFunnelListen(t *testing.T) {
    const dir = "test"
    err := os.MkdirAll(dir, 0700)
    if err != nil {
        t.Fatal(err)
    }
    srv := tsnet.Server{
        Dir:       dir,
        Hostname:  dir,
    }
    ln1, err := srv.ListenFunnel("tcp", ":443", tsnet.FunnelOnly())
    if err != nil {
        t.Fatal(err)
    }
    ln2, err := srv.ListenFunnel("tcp", ":8443", tsnet.FunnelOnly())
    if err != nil {
        t.Fatal(err)
    }

    handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        w.Write([]byte("hello"))
    })

    go http.Serve(ln1, handler)
    go http.Serve(ln2, handler)

    ctx, cancel := signal.NotifyContext(context.Background(), os.Interrupt)
    defer cancel()
    <-ctx.Done()
}

if you then curl https://test.MY-TAILNET.ts.net:443 you will not be able to connect, you will see the following in the verbose output:

curl: (35) error:0A000126:SSL routines::unexpected eof while reading

and you will see in the tsnet logs:

2023/08/06 13:55:33 localbackend: got ingress conn for unconfigured "test.MY-TAILNET.ts.net:443"; rejecting

Are there any recent changes that introduced the issue?

No response

OS

Linux

OS version

Ubuntu 22.04.2 LTS

Tailscale version

1.46.1

Other software

No response

Bug report

BUG-74d541883ee147c620b4124f9063acf130b7b15de80544de1d28db81e5860e2d-20230806125844Z-7425c0e1bb8207c2

bradfitz commented 10 months ago

Maybe http.Serve is returning an error with a clue?

ifross89 commented 10 months ago

Hi, thanks for the prompt reply.

I updated the test code to read:

    go func() {
        panic(http.Serve(ln1, handler))
    }()
    go func() {
        panic(http.Serve(ln2, handler))
    }()

and there is no panic, and the issue remains.

Thanks

Hobbit44 commented 4 weeks ago

I'm getting the exact same symptoms, but with a slightly different use. I'm running the k8s operator and trying to create a funnel with according to the docs.

resource "kubernetes_manifest" "webhook_ingress" {
  manifest = {
    apiVersion = "networking.k8s.io/v1"
    kind       = "Ingress"
    metadata = {
      name = "webhook"
      namespace = local.namespaces.home
      annotations = {
        "tailscale.com/funnel" = "true"
      }
    }

    spec = {
      ingressClassName = "tailscale"
      defaultBackend = {
        service = {
          name = "homeassistant"
          port = {
            number = 8123
          }
        }
      }
      tls = [{
        hosts = [
          "webhook",
        ]
      }]
    }
  }
}

but when i try and curl the created dns endpoint i get:

❯ curl https://webhook.[domain]
curl: (35) error:0A000126:SSL routines::unexpected eof while reading

Hobbit44 commented 4 weeks ago

Well, i tried a different service, and made it worse:

resource "kubernetes_manifest" "webhook_ingress" {
  manifest = {
    apiVersion = "networking.k8s.io/v1"
    kind       = "Ingress"
    metadata = {
      name = "paperless-external"
      namespace = local.namespaces.home
      annotations = {
        "tailscale.com/funnel" = "true"
        "cert-manager.io/cluster-issuer" = local.cert_manager.issuer
      }
    }

    spec = {
      ingressClassName = "tailscale"
      rules = [{
        host = "paperless.${local.domain}"
      }]
      tls = [{
        hosts = [
          "paperless.${local.domain}",
        ]
        secretName = "paperless-tls"
      }]
    }
  }
}

and ended up crashing the operator pod with:

{"level":"info","ts":"2024-05-17T15:15:26Z","msg":"Starting workers","controller":"ingress","controllerGroup":"networking.k8s.io","controllerKind":"Ingress","worker count":1}                                                                                                                                      {"level":"info","ts":"2024-05-17T15:15:26Z","msg":"Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference","controller":"ingress","controllerGroup":"networking.k8s.io","controllerKind":"Ingress","Ingress":{"name":"paperless-external","namespace":"home"},"namespace":"home","name":"paperless-external","reconcileID":"f53f0c00-d65a-467e-8bd9-fc22d5a518bd"}
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1f6d193]

goroutine 535 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()                                                                        sigs.k8s.io/controller-runtime@v0.16.2/pkg/internal/controller/controller.go:116 +0x1e5
panic({0x21ef260?, 0x3bda900?})                                                                                                                               runtime/panic.go:770 +0x132                                                                                                                           main.(*IngressReconciler).maybeProvision(0xc002c578c0, {0x296bd98, 0xc000459200}, 0xc000392100, 0xc002a458c0)                                                 tailscale.com/cmd/k8s-operator/ingress.go:230 +0x973
main.(*IngressReconciler).Reconcile(0xc002c578c0, {0x296bd98, 0xc000459200}, {{{0xc002fb0198?, 0x5?}, {0xc002f4cc18?, 0xc002dbfd10?}}})                       tailscale.com/cmd/k8s-operator/ingress.go:76 +0x3e7                                                                                                   sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x2971840?, {0x296bd98?, 0xc000459200?}, {{{0xc002fb0198?, 0xb?}, {0xc002f4cc18?, 0x0?}}})
    sigs.k8s.io/controller-runtime@v0.16.2/pkg/internal/controller/controller.go:119 +0xb7
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc002cf5540, {0x296bdd0, 0xc002cf36d0}, {0x22e2620, 0xc00304ad20})
    sigs.k8s.io/controller-runtime@v0.16.2/pkg/internal/controller/controller.go:316 +0x3bc
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc002cf5540, {0x296bdd0, 0xc002cf36d0})
    sigs.k8s.io/controller-runtime@v0.16.2/pkg/internal/controller/controller.go:266 +0x1be
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
    sigs.k8s.io/controller-runtime@v0.16.2/pkg/internal/controller/controller.go:227 +0x79
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 401
    sigs.k8s.io/controller-runtime@v0.16.2/pkg/internal/controller/controller.go:223 +0x50c

tailscale / tailscale