pprof profiling does not work with tekton-pipelines-controller deployment

[gabemontero]

Expected Behavior

Should be able to make http requests against the tekton-pipelines-controller pprof port (8008) when profiling is enabled

Actual Behavior

gmontero ~/go/src/github.com/tektoncd/pipeline  (main)$ curl http://127.0.0.1:8008/debug/pprof/goroutine?debug=2
curl: (52) Empty reply from server
gmontero ~/go/src/github.com/tektoncd/pipeline  (main)$ 

Steps to Reproduce the Problem

1. set profiling.enable = "true" in the config-observability config map
2. run ko apply -R -f ./config from your branch against your k8s cluster
3. confirm profiling is enable in the tekton-pipelines-controller logs ... something like {"severity":"info","timestamp":"2024-08-05T14:58:27.082Z","logger":"tekton-pipelines-controller","caller":"profiling/server.go:106","message":"Profiling enabled: true","commit":"0db5ca2-dirty"}
4. in a separate window, run kubectl port-forward -n tekton-pipelines service/tekton-pipelines-controller 8080
5. in your first window, run curl http://127.0.0.1:8008/debug/pprof/goroutine?debug=2

Additional Info

• Kubernetes version:

  kind v0.23, but also seen on openshift 4.14, 4.15

• Tekton Pipeline version:

gmontero ~/go/src/github.com/tektoncd/pipeline  (main)$ git log -1
commit 0db5ca21181b5a966642fd336650cf69a6b0d433 (HEAD -> main, upstream/main, origin/main)
Author: Samarth Maganahalli <samarth.maganahalli@gmail.com>
Date:   Sat Jul 6 16:57:30 2024 +0530

    fix step action datatype
gmontero ~/go/src/github.com/tektoncd/pipeline  (main)$ 

We've been able to do the above with other knative based tekton controllers, like the watcher from tekton-results

[gabemontero]

I've also tried with the pprof command trying to connect to the port. In addition to failing on the pprof side, it caused the kubectl port-forward to exit out:

~/go/src/github.com/tektoncd/pipeline  (test-pprof-on-kind)$ pprof --text 127.0.0.1:8008
Using remote profile at 127.0.0.1:8008.
Use of uninitialized value $line in substitution (s///) at /usr/bin/pprof line 3326.
http://127.0.0.1:8008/pprof/symbol doesn't exist
~/go/src/github.com/tektoncd/pipeline  (test-pprof-on-kind)$ 

and

~/go/src/github.com/tektoncd/pipeline  (test-pprof-on-kind)$ kubectl port-forward -n tekton-pipelines service/tekton-pipelines-controller 8008
Forwarding from 127.0.0.1:8008 -> 8008
Forwarding from [::1]:8008 -> 8008
Handling connection for 8008
E0820 15:21:54.560477 1862495 portforward.go:409] an error occurred forwarding 8008 -> 8008: error forwarding port 8008 to pod 28833df381b8f1e57257df3eace0abd4d8714fc7db431ab999442616ac9ce188, uid : failed to execute portforward in network namespace "/var/run/netns/cni-4cde766c-4394-b5b0-9fbf-5c10ea2e0de7": failed to connect to localhost:8008 inside namespace "28833df381b8f1e57257df3eace0abd4d8714fc7db431ab999442616ac9ce188", IPv4: dial tcp4 127.0.0.1:8008: connect: connection refused IPv6 dial tcp6 [::1]:8008: connect: connection refused 
error: lost connection to pod
~/go/src/github.com/tektoncd/pipeline  (test-pprof-on-kind)$

[gabemontero]

@vdemeester FYI ^^