search

kubernetes-sigs / kwok

Kubernetes WithOut Kubelet - Simulates thousands of Nodes and Clusters.
https://kwok.sigs.k8s.io
Apache License 2.0
2.59k stars 206 forks source link

Creating kwok cluster during docker build fails to be accessible after running the container #1077

Closed expruc closed 1 month ago

expruc commented 6 months ago

How to use it?

What happened?

I'm trying to create a container containing a cluster created on build time (running kwokctl create cluster on Dockerfile). When running the container, the cluster is shown as created (after running the container and entering it, running kwokctl get clusters returns the proper result), but the cluster is inaccessible (running kwokctl kubectl get pods for example, returns connection refused on port 32764 - the default port kwok opens on cluster creation). Moreover, when creating a new cluster, it creates the cluster on the same port successfully (and new cluster is accessible).

I saw on the examples in this repository that the cluster is served using kubectl proxy. Is it a must to access the cluster? I couldn't find documentation about building an image with the cluster, especially when the kwok is meant to run in the background and the entrypoint is another application.

From what I saw, the cluster creation during the build was fine and all the files under /root/.kwok/clusters/<my-cluster> are fine. I tried also to move the creation of the cluster command to the entrypoint, without success.

What did you expect to happen?

Cluster is running on that port after creating and running the container. Entering the container and running kubectl get pods shouldn't return connection refused

How can we reproduce it (as minimally and precisely as possible)?

The Dockerfile:

FROM golang:1.22.3 as build

RUN apt-get update && \
    apt-get install -y jq

RUN KWOK_REPO=kubernetes-sigs/kwok && \
    KWOK_LATEST_RELEASE=$(curl "https://api.github.com/repos/${KWOK_REPO}/releases/latest" | jq -r '.tag_name') && \
    # kwokctl
    wget -O kwokctl -c "https://github.com/${KWOK_REPO}/releases/download/${KWOK_LATEST_RELEASE}/kwokctl-$(go env GOOS)-$(go env GOARCH)"&& \
    chmod +x kwokctl && \
    # kwok
    wget -O kwok -c "https://github.com/${KWOK_REPO}/releases/download/${KWOK_LATEST_RELEASE}/kwok-$(go env GOOS)-$(go env GOARCH)" && \
    chmod +x kwok && \
    # kubectl
    curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" && \
    chmod +x kubectl && \
    mv kwokctl kwok kubectl /usr/local/bin && \
    kwokctl create cluster --name=background-cluster
docker build -t <image-name> .
docker run --rm -dit -p 8080:8080 <image-name>

docker exec -it <container-name> bash

And inside the container the results of the commands:

root@199b24037cdb:/go# kwokctl get clusters
background-cluster
root@199b24037cdb:/go# kubectl get pods
E0514 18:17:25.890363      99 memcache.go:265] couldn't get current server API group list: Get "https://127.0.0.1:32764/api?timeout=32s": dial tcp 127.0.0.1:32764: connect: connection refused
E0514 18:17:25.890617      99 memcache.go:265] couldn't get current server API group list: Get "https://127.0.0.1:32764/api?timeout=32s": dial tcp 127.0.0.1:32764: connect: connection refused
E0514 18:17:25.891945      99 memcache.go:265] couldn't get current server API group list: Get "https://127.0.0.1:32764/api?timeout=32s": dial tcp 127.0.0.1:32764: connect: connection refused
E0514 18:17:25.892069      99 memcache.go:265] couldn't get current server API group list: Get "https://127.0.0.1:32764/api?timeout=32s": dial tcp 127.0.0.1:32764: connect: connection refused
E0514 18:17:25.905428      99 memcache.go:265] couldn't get current server API group list: Get "https://127.0.0.1:32764/api?timeout=32s": dial tcp 127.0.0.1:32764: connect: connection refused
The connection to the server 127.0.0.1:32764 was refused - did you specify the right host or port?
root@199b24037cdb:/go# kwokctl create cluster --name=new-cluster
Cluster is creating                                                                                                                                      cluster=new-cluster
Cluster is created                                                                                                                          elapsed=0.3s cluster=new-cluster
Cluster is starting                                                                                                                                      cluster=new-cluster
Cluster is started                                                                                                                          elapsed=0.7s cluster=new-cluster
You can now use your cluster with:

        kubectl cluster-info --context kwok-new-cluster

Thanks for using kwok!
root@199b24037cdb:/go# kubectl get pods
No resources found in default namespace.

Anything else we need to know?

I'm not sure if this behavior should be expected or not, though I don't see any reason for it to fail.

Kwok version

```console $ kwok --version kwok version v0.5.2 go1.21.9 (linux/amd64) $ kwokctl --version kwokctl version v0.5.2 go1.21.9 (linux/amd64) ```

OS version

```console # On Linux: $ cat /etc/os-release PRETTY_NAME="Debian GNU/Linux 12 (bookworm)" NAME="Debian GNU/Linux" VERSION_ID="12" VERSION="12 (bookworm)" VERSION_CODENAME=bookworm ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/" $ uname -a Linux 193b14697e0f 6.5.0-28-generic #29-Ubuntu SMP PREEMPT_DYNAMIC Thu Mar 28 23:46:48 UTC 2024 x86_64 GNU/Linux # On Darwin: $ uname -a # paste output here # On Windows: C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture # paste output here ```
network-charles commented 6 months ago

It looks like you’re missing this in your RUN section “go install sigs.k8s.io/kwok/cmd/{kwok,kwokctl}@${KWOK_LATEST_RELEASE}”

wzshiming commented 6 months ago

I saw on the examples in this repository that the cluster is served using kubectl proxy. Is it a must to access the cluster? I couldn't find documentation about building an image with the cluster, especially when the kwok is meant to run in the background and the entrypoint is another application.

Not required, if you don't use kubectl proxy, you need to copy the TLS certificates out of the container when using the cluster outside the container.

When running the container, the cluster is shown as created (after running the container and entering it, running kwokctl get clusters returns the proper result), but the cluster is inaccessible (running kwokctl kubectl get pods for example, returns connection refused on port 32764 - the default port kwok opens on cluster creation).

When the container starts, the only process running is your shell, the cluster doesn't actually start.

expruc commented 6 months ago

It looks like you’re missing this in your RUN section “go install sigs.k8s.io/kwok/cmd/{kwok,kwokctl}@${KWOK_LATEST_RELEASE}”

This didn't help and I'm still experiencing the same error.

I saw on the examples in this repository that the cluster is served using kubectl proxy. Is it a must to access the cluster? I couldn't find documentation about building an image with the cluster, especially when the kwok is meant to run in the background and the entrypoint is another application.

Not required, if you don't use kubectl proxy, you need to copy the TLS certificates out of the container when using the cluster outside the container.

I'm trying to run kubectl commands from inside the container, not outside.

When the container starts, the only process running is your shell, the cluster doesn't actually start.

So how can I make the cluster run in the background? Do I have to include the cluster creation command on the entrypoint/cmd?

wzshiming commented 6 months ago

Do I have to include the cluster creation command on the entrypoint/cmd?

Yes

k8s-triage-robot commented 3 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 2 months ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot commented 1 month ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

k8s-ci-robot commented 1 month ago

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to [this](https://github.com/kubernetes-sigs/kwok/issues/1077#issuecomment-2408619006): >The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. > >This bot triages issues according to the following rules: >- After 90d of inactivity, `lifecycle/stale` is applied >- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied >- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed > >You can: >- Reopen this issue with `/reopen` >- Mark this issue as fresh with `/remove-lifecycle rotten` >- Offer to help out with [Issue Triage][1] > >Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community). > >/close not-planned > >[1]: https://www.kubernetes.dev/docs/guide/issue-triage/ Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.