Kwok creating cluster with prometheus somehow make api server give HTTP response to HTTPS client

How to use it?

[X] kwok
[ ] kwokctl --runtime=docker (default runtime)
[ ] kwokctl --runtime=binary
[ ] kwokctl --runtime=nerdctl
[ ] kwokctl --runtime=kind

What happened?

I'm using Kwok all-in-one image in a k8s cluster to do some experiments about cross cluster scheduler. I deploy the image and use Kwok inside the container to simulating a cluster. Everything goes well until I create a cluster enabling prometheus.

here is exmple of a well-working kwok. normal_kwok

Now if I create a cluster enabling prometheus, the api-server goes wrong. problem_screen

I tried to use --kube-apiserver-insecure-port 8080, hoping this would work, but it doesn't.

Then I delete the prometheus-enabled cluster and create a new one using kwokctl create cluster, and problem still exists. problem_screen_2

What did you expect to happen?

Kwok all-in-one image in K8s cluster could work well in a container simulating cross cluster scheduling. Prometheus within a container works with Kwok.

How can we reproduce it (as minimally and precisely as possible)?

As mentioned in the screen shots above

Anything else we need to know?

No response

Kwok version

```console $ kwok --version kwok version v0.5.0-167-g60d7b64-dirty go1.22.3 (linux/amd64) $ kwokctl --version kwokctl version v0.5.0-167-g60d7b64-dirty go1.22.3 (linux/amd64) ```

OS version

```console # On Linux: $ cat /etc/os-release NAME="Alpine Linux" ID=alpine VERSION_ID=3.20.0 PRETTY_NAME="Alpine Linux v3.20" HOME_URL="https://alpinelinux.org/" BUG_REPORT_URL="https://gitlab.alpinelinux.org/alpine/aports/-/issues" $ uname -a Linux kwok-node-1-ff469c74c-xfrkb 5.15.0-112-generic #122-Ubuntu SMP Thu May 23 07:48:21 UTC 2024 x86_64 Linux # On Darwin: $ uname -a # paste output here # On Windows: C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture # paste output here ```

I've tried it and it looks fine. Is there something I'm missing?

$ docker run -e KWOK_PROMETHEUS_PORT=9090 --rm -it -p 8080:8080 registry.k8s.io/kwok/cluster:v0.5.2-k8s.v1.29.2
Unable to find image 'registry.k8s.io/kwok/cluster:v0.5.2-k8s.v1.29.2' locally
v0.5.2-k8s.v1.29.2: Pulling from kwok/cluster
4abcf2066143: Pull complete 
c3cab689956f: Pull complete 
e1341fad90bf: Pull complete 
bd6d5a467c0c: Pull complete 
de935f35f339: Pull complete 
Digest: sha256:c9876ee5431620b755915f78a64455961e743b1a30b84a4d21a2e698f0f9917c
Status: Downloaded newer image for registry.k8s.io/kwok/cluster:v0.5.2-k8s.v1.29.2
Cluster is creating                                                                                                                                        cluster=kwok
Download                                       uri=https://github.com/prometheus/prometheus/releases/download/v2.49.1/prometheus-2.49.1.linux-amd64.tar.gz cluster=kwok
############################################################| 100%  0s   
Cluster is created                                                                                                                              elapsed=4s cluster=kwok
Cluster is starting                                                                                                                                        cluster=kwok
Cluster is started                                                                                                                            elapsed=2.5s cluster=kwok
You can now use your cluster with:

        kubectl cluster-info --context kwok-kwok

Thanks for using kwok!
Starting to serve on [::]:8080

kubernetes-sigs / kwok