support using proxy subresources when connecting to Ray head node

[andrewsykim]

Why are these changes needed?

There are some cases where Kuberay may not be able to directly connect to a Ray head node. For example, there might be a NetworkPolicy disallowing ingress from all Pods or KubeRay is running on a network with no connectivity to Pods. This PR allows Kuberay to use the services/proxy subresource to proxy HTTP requests to the Ray head node. This allows Kuberay to make requests to the head node without every connecting to it directly.

Here are some sample HTTP requests in apiserver from my testing using the proxy subresource:

I0310 14:30:19.596708       1 httplog.go:131] "HTTP" verb="GET" URI="/api/v1/namespaces/default/services/rayjob-sample-raycluster-phsj9-head-svc:dashboard/proxy/api/jobs/rayjob-sample-th44t" latency="7.239221ms" userAgent="manager/v0.0.0 (linux/amd64) kubernetes/$Format" audit-ID="c40152de-7a81-46b9-ac4a-f2ea296e44f0" srcIP="10.244.0.6:49524" resp=200

I0310 15:20:43.105571       1 httplog.go:131] "HTTP" verb="GET" URI="/api/v1/namespaces/default/pods/rayservice-sample-raycluster-qm2m2-head-xj44d:8000/proxy/-/healthz" latency="2.915789ms" userAgent="manager/v0.0.0 (linux/amd64) kubernetes/$Format" audit-ID="bad00c23-de01-45ed-9fb5-05bc8b2f6c2d" srcIP="10.244.0.6:47274" resp=200

I0310 15:21:07.446881       1 httplog.go:131] "HTTP" verb="GET" URI="/api/v1/namespaces/default/services/rayservice-sample-raycluster-qm2m2-head-svc:dashboard/proxy/api/serve/applications/" latency="15.347398ms" userAgent="manager/v0.0.0 (linux/amd64) kubernetes/$Format" audit-ID="54544346-b827-4f34-b593-bcbc78199611" srcIP="10.244.0.6:47274" resp=200

Checks

• [X] I've made sure the tests are passing.
• Testing Strategy
  • [X] Unit tests
  • [X] Manual tests
  • [ ] This PR is not tested :(

[kevin85421]

I tested this PR manually.

• Step 0: Create a Kind cluster with the following configurations:

  kind: Cluster
  apiVersion: kind.x-k8s.io/v1alpha4
  nodes:
  - role: control-plane
  kubeadmConfigPatches:
  - |
    kind: ClusterConfiguration
    apiServer:
        extraArgs:
          v: "4"

• Step 1: Update deployment.yaml (helm-chart/kuberay-operator/templates/deployment.yaml) to enable this feature.

  {{- $argList = append $argList "--use-kubernetes-proxy" -}}

• Step 2: Install the KubeRay operator.
• Step 3: Create a RayJob CR.
• Step 4: Check the API server log

  I0419 01:57:09.080550       1 httplog.go:132] "HTTP" verb="GET" URI="/api/v1/namespaces/default/services/rayjob-sample-raycluster-rxhpc-head-svc:dashboard/proxy/api/jobs/rayjob-sample-zcjbj" latency="4.394603ms" userAgent="kuberay-operator/nightly" audit-ID="db0c2d98-a5d0-479f-81f8-7e9d7e3dc38b" srcIP="10.244.0.5:47796" resp=200

[andrewsykim]

thanks @kevin85421! I'll look into adding an e2e test as well