search

kubernetes-sigs / kwok

Kubernetes WithOut Kubelet - Simulates thousands of Nodes and Clusters.
https://kwok.sigs.k8s.io
Apache License 2.0
2.39k stars 190 forks source link

Creating kwok cluster during docker build fails to be accessible after running the container #1077

Open expruc opened 2 months ago

expruc commented 2 months ago

How to use it?

What happened?

I'm trying to create a container containing a cluster created on build time (running kwokctl create cluster on Dockerfile). When running the container, the cluster is shown as created (after running the container and entering it, running kwokctl get clusters returns the proper result), but the cluster is inaccessible (running kwokctl kubectl get pods for example, returns connection refused on port 32764 - the default port kwok opens on cluster creation). Moreover, when creating a new cluster, it creates the cluster on the same port successfully (and new cluster is accessible).

I saw on the examples in this repository that the cluster is served using kubectl proxy. Is it a must to access the cluster? I couldn't find documentation about building an image with the cluster, especially when the kwok is meant to run in the background and the entrypoint is another application.

From what I saw, the cluster creation during the build was fine and all the files under /root/.kwok/clusters/<my-cluster> are fine. I tried also to move the creation of the cluster command to the entrypoint, without success.

What did you expect to happen?

Cluster is running on that port after creating and running the container. Entering the container and running kubectl get pods shouldn't return connection refused

How can we reproduce it (as minimally and precisely as possible)?

The Dockerfile:

FROM golang:1.22.3 as build

RUN apt-get update && \
    apt-get install -y jq

RUN KWOK_REPO=kubernetes-sigs/kwok && \
    KWOK_LATEST_RELEASE=$(curl "https://api.github.com/repos/${KWOK_REPO}/releases/latest" | jq -r '.tag_name') && \
    # kwokctl
    wget -O kwokctl -c "https://github.com/${KWOK_REPO}/releases/download/${KWOK_LATEST_RELEASE}/kwokctl-$(go env GOOS)-$(go env GOARCH)"&& \
    chmod +x kwokctl && \
    # kwok
    wget -O kwok -c "https://github.com/${KWOK_REPO}/releases/download/${KWOK_LATEST_RELEASE}/kwok-$(go env GOOS)-$(go env GOARCH)" && \
    chmod +x kwok && \
    # kubectl
    curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" && \
    chmod +x kubectl && \
    mv kwokctl kwok kubectl /usr/local/bin && \
    kwokctl create cluster --name=background-cluster
docker build -t <image-name> .
docker run --rm -dit -p 8080:8080 <image-name>

docker exec -it <container-name> bash

And inside the container the results of the commands:

root@199b24037cdb:/go# kwokctl get clusters
background-cluster
root@199b24037cdb:/go# kubectl get pods
E0514 18:17:25.890363      99 memcache.go:265] couldn't get current server API group list: Get "https://127.0.0.1:32764/api?timeout=32s": dial tcp 127.0.0.1:32764: connect: connection refused
E0514 18:17:25.890617      99 memcache.go:265] couldn't get current server API group list: Get "https://127.0.0.1:32764/api?timeout=32s": dial tcp 127.0.0.1:32764: connect: connection refused
E0514 18:17:25.891945      99 memcache.go:265] couldn't get current server API group list: Get "https://127.0.0.1:32764/api?timeout=32s": dial tcp 127.0.0.1:32764: connect: connection refused
E0514 18:17:25.892069      99 memcache.go:265] couldn't get current server API group list: Get "https://127.0.0.1:32764/api?timeout=32s": dial tcp 127.0.0.1:32764: connect: connection refused
E0514 18:17:25.905428      99 memcache.go:265] couldn't get current server API group list: Get "https://127.0.0.1:32764/api?timeout=32s": dial tcp 127.0.0.1:32764: connect: connection refused
The connection to the server 127.0.0.1:32764 was refused - did you specify the right host or port?
root@199b24037cdb:/go# kwokctl create cluster --name=new-cluster
Cluster is creating                                                                                                                                      cluster=new-cluster
Cluster is created                                                                                                                          elapsed=0.3s cluster=new-cluster
Cluster is starting                                                                                                                                      cluster=new-cluster
Cluster is started                                                                                                                          elapsed=0.7s cluster=new-cluster
You can now use your cluster with:

        kubectl cluster-info --context kwok-new-cluster

Thanks for using kwok!
root@199b24037cdb:/go# kubectl get pods
No resources found in default namespace.

Anything else we need to know?

I'm not sure if this behavior should be expected or not, though I don't see any reason for it to fail.

Kwok version

```console $ kwok --version kwok version v0.5.2 go1.21.9 (linux/amd64) $ kwokctl --version kwokctl version v0.5.2 go1.21.9 (linux/amd64) ```

OS version

```console # On Linux: $ cat /etc/os-release PRETTY_NAME="Debian GNU/Linux 12 (bookworm)" NAME="Debian GNU/Linux" VERSION_ID="12" VERSION="12 (bookworm)" VERSION_CODENAME=bookworm ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/" $ uname -a Linux 193b14697e0f 6.5.0-28-generic #29-Ubuntu SMP PREEMPT_DYNAMIC Thu Mar 28 23:46:48 UTC 2024 x86_64 GNU/Linux # On Darwin: $ uname -a # paste output here # On Windows: C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture # paste output here ```
network-charles commented 2 months ago

It looks like you’re missing this in your RUN section “go install sigs.k8s.io/kwok/cmd/{kwok,kwokctl}@${KWOK_LATEST_RELEASE}”

wzshiming commented 2 months ago

I saw on the examples in this repository that the cluster is served using kubectl proxy. Is it a must to access the cluster? I couldn't find documentation about building an image with the cluster, especially when the kwok is meant to run in the background and the entrypoint is another application.

Not required, if you don't use kubectl proxy, you need to copy the TLS certificates out of the container when using the cluster outside the container.

When running the container, the cluster is shown as created (after running the container and entering it, running kwokctl get clusters returns the proper result), but the cluster is inaccessible (running kwokctl kubectl get pods for example, returns connection refused on port 32764 - the default port kwok opens on cluster creation).

When the container starts, the only process running is your shell, the cluster doesn't actually start.

expruc commented 2 months ago

It looks like you’re missing this in your RUN section “go install sigs.k8s.io/kwok/cmd/{kwok,kwokctl}@${KWOK_LATEST_RELEASE}”

This didn't help and I'm still experiencing the same error.

I saw on the examples in this repository that the cluster is served using kubectl proxy. Is it a must to access the cluster? I couldn't find documentation about building an image with the cluster, especially when the kwok is meant to run in the background and the entrypoint is another application.

Not required, if you don't use kubectl proxy, you need to copy the TLS certificates out of the container when using the cluster outside the container.

I'm trying to run kubectl commands from inside the container, not outside.

When the container starts, the only process running is your shell, the cluster doesn't actually start.

So how can I make the cluster run in the background? Do I have to include the cluster creation command on the entrypoint/cmd?

wzshiming commented 2 months ago

Do I have to include the cluster creation command on the entrypoint/cmd?

Yes