Enable pprof support in ctrl runtime

[skonto]

Proposed Changes

• We recently had an issue that could have needed pprof support see discussion.
• Right now pprof is not configurable in knative-openshift deployment
• ctrl runtime is not configurable via a cm, there was a design in the past but it seems not done. Ideally we would like to set up things the Knative way at some point: install a handler that depending on a cm value uses the pprof handler or some dummy that returns 403. The latter allows to dynamically switch on/off pprof support and get the data at the time of the issue without pod restart.
• Here we re-use the pprof support ctrl manager which is static thus we need a restart. That is ok for now given the low number of issues we have faced in practice and that pod restart would work for most cases (but would require some time for the issue to re-appear once a pod is restarted).
• Probably it would be ok even to have it on by default but let's not be that brave for now.
• User can enable this temporarily by editing the csv in case of debugging a case.
• Pprof is exposed only locally for security reasons, available via port-forward (but could be open on ":8008" as Knative if we want to be less restrictive).
• See sample heap profile here. It shows that ctrl runtime dynamicclient consumes most part of memory during cache initialization.
• Tested it locally by editing the csv (pod is restarted automatically with the right env var value set).

[skonto]

@matzew gentle ping

[openshift-ci[bot]]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: matzew, ReToCode, skonto

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files: - ~~[OWNERS](https://github.com/openshift-knative/serverless-operator/blob/main/OWNERS)~~ [ReToCode,matzew,skonto] Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment

[skonto]

/cherry-pick release-1.33

[openshift-cherrypick-robot]

@skonto: once the present PR merges, I will cherry-pick it on top of release-1.33 in a new PR and assign it to you.

In response to [this](https://github.com/openshift-knative/serverless-operator/pull/2766#issuecomment-2222847447): >/cherry-pick release-1.33 Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.

[skonto]

/unhold

[openshift-ci-robot]

/retest-required

Remaining retests: 0 against base HEAD edce15ed36b5a2cf38ecb2a22148e05a48057ba6 and 2 for PR HEAD 986ebadf0c5b537938ba274f5414175cc23fc939 in total

[openshift-cherrypick-robot]

@skonto: new pull request created: #2769

In response to [this](https://github.com/openshift-knative/serverless-operator/pull/2766#issuecomment-2222847447): >/cherry-pick release-1.33 Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.