feat: add vcluster e2e cluster provider

[crandles]

addition of vcluster provider; interested to see if the CI fails. vcluster needs a "host" cluster to launch into, so I've followed an existing example where a real cluster or kind cluster are utilized as necessary: https://github.com/kubernetes-sigs/e2e-framework/blob/9206e6f6082907a1c77750b7cac0fc44ff4dac02/examples/real_cluster/main_test.go#L39

What type of PR is this?

/kind feature

What this PR does / why we need it:

This PR enables a new E2EClusterProvider for vcluster based clusters.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?

NONE

Additional documentation e.g., Usage docs, etc.:

NA

[k8s-ci-robot]

Hi @crandles. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.

[vladimirvivien]

@crandles this is awesome and thanks for the contribution. At first glance it looks Okay, but I will ask @harshanarayana to take a look as well since he's the E2E provider expert 😉

[vladimirvivien]

/retest

[crandles]

Updated copyright, verify-boilerplate passes locally now.

[vladimirvivien]

/retest

[crandles]

golangci appears to be passing now for thew new code I've added, though errirng for some existing LOC:

third_party/helm/helm.go:240:9: SA1006: printf-style function with dynamic format string and no further arguments should use print-style function instead (staticcheck)
        err = fmt.Errorf(missingHelm)
              ^
third_party/flux/flux_setup.go:47:16: SA1006: printf-style function with dynamic format string and no further arguments should use print-style function instead (staticcheck)
            return ctx, fmt.Errorf(NoFluxInstallationFoundMsg)
                        ^
third_party/flux/flux_setup.go:61:16: SA1006: printf-style function with dynamic format string and no further arguments should use print-style function instead (staticcheck)
            return ctx, fmt.Errorf(NoFluxInstallationFoundMsg)
                        ^
pkg/env/env.go:229:11: printf: non-constant format string in call to (*testing.common).Skipf (govet)
        t.Skipf(message)
                ^
pkg/env/env.go:513:22: printf: non-constant format string in call to (*testing.common).Skipf (govet)
                    internalT.Skipf(message)

Assuming that won't block this PR.

[vladimirvivien]

@harshanarayana and @cpanato any idea why golangci failing on previously working code ?

[crandles]

errors were primarily for:

SA1006: printf-style function with dynamic format string and no further arguments should use print-style function instead (staticcheck)

It's possible that I'm using a different toolchain version, but as the warning seemed fair, I've fixed it via https://github.com/kubernetes-sigs/e2e-framework/pull/450/commits/cf9cbf28dfc7a407ed901b2da26b31ef4aabcc35

make verify now passes locally.

[vladimirvivien]

@crandles is this ready to go now ?

[crandles]

Yes, I believe so

[crandles]

rebased due to go.mod conflict

[vladimirvivien]

@harshanarayana @ShwethaKumbla @cpanato PTAL I think we need a second reviewer.

[harshanarayana]

I will check by eod tomorrow.

[crandles]

Seems to fail installing vcluster; can't reproduce. Any tips? I think I'm installing vcluster the same way other tools are (ko|kwok|kind)

[harshanarayana]

Seems to fail installing vcluster; can't reproduce. Any tips? I think I'm installing vcluster the same way other tools are (ko|kwok|kind)

@crandles

Helm and flux are installed slightly in a different way, https://github.com/kubernetes-sigs/e2e-framework/blob/777cbcd59fbbe572b71dde89c8f7b0b1a5ddc4a7/hack/install-helm.sh

test: install-helm install-flux ## Runs golang unit tests
    ./hack/test-go.sh

But you are right about ko or kwok though

[vladimirvivien]

/retest

[crandles]

Is there a recommended way to simulate the prow test runner locally?

This looks relevant: https://docs.prow.k8s.io/docs/build-test-update/#running-a-prowjob-locally (and this job), I'll try to follow it.

---

Trying this out so far, not quite working yet:

# new makefile target
run-pr-tests:
    mkdir -p ./build ./build/out ./build/node_dir
    curl -s -o build/e2e-framework-presubmits.yaml https://raw.githubusercontent.com/kubernetes/test-infra/master/config/jobs/kubernetes-sigs/e2e-framework/e2e-framework-presubmits.yaml
    curl -s -o build/pj-on-kind.sh https://raw.githubusercontent.com/kubernetes-sigs/prow/refs/heads/main/pkg/pj-on-kind.sh
    curl -s -o build/prow-config.yaml https://raw.githubusercontent.com/kubernetes/test-infra/refs/heads/master/config/prow/config.yaml
    NODE_DIR=$(realpath build/node_dir) OUT_DIR=$(realpath build/out) CONFIG_PATH=$(realpath build/prow-config.yaml) JOB_CONFIG_PATH=$(realpath build/e2e-framework-presubmits.yaml) bash ./build/pj-on-kind.sh pull-e2e-framework-test

Output

``` latest: Pulling from k8s-prow/mkpj Digest: sha256:6e71c2d9f675bce23136443cd8093fad59ed3344325d305963e89eadb7d3b6d5 Status: Image is up to date for gcr.io/k8s-prow/mkpj:latest gcr.io/k8s-prow/mkpj:latest What's next: View a summary of image vulnerabilities and recommendations → docker scout quickview gcr.io/k8s-prow/mkpj:latest WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested time="2024-11-13T04:38:37Z" level=warning msg="empty -github-token-path, will use anonymous github client" time="2024-11-13T04:38:37Z" level=info msg="Throttle(0, 0, [])" client=github PR Number: 450 time="2024-11-13T04:39:20Z" level=info msg="GetPullRequest(kubernetes-sigs, e2e-framework, 450)" client=github time="2024-11-13T04:39:20Z" level=debug msg="Using GitHub REST API Version: 2022-11-28" client=github time="2024-11-13T04:39:20Z" level=debug msg="Used token for request" cache-mode= client=github method=GET path=/repos/kubernetes-sigs/e2e-framework/pulls/450 throttled=true time="2024-11-13T04:39:20Z" level=debug msg="GetPullRequest(kubernetes-sigs, e2e-framework, 450) finished" client=github duration=419.387292ms latest: Pulling from k8s-prow/mkpod Digest: sha256:1a95d701f6cafae2c004088e47f15efc0b8e5b8d8fb6a23a2e2ccd94471eda46 Status: Image is up to date for gcr.io/k8s-prow/mkpod:latest gcr.io/k8s-prow/mkpod:latest What's next: View a summary of image vulnerabilities and recommendations → docker scout quickview gcr.io/k8s-prow/mkpod:latest WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested time="2024-11-13T04:39:22Z" level=info msg="Generated build-id for job." build-id=1856557443136360448 time="2024-11-13T04:39:22Z" level=info msg="Pod-utils configured for local mode. Instead of uploading to GCS, files will be copied to an output dir on the node." out-dir=/Users/c/code/e2e-framework/build/out/pull-e2e-framework-test/1856557443136360448 For each of the following volumes specify one of: - 'empty' to use an emptyDir; - a path on the host to use hostPath; - '' (nothing) to use the existing volume source and assume it is available in the cluster Volume "service": empty Applying pod to the mkpod cluster. Configure kubectl for the mkpod cluster with: > export KUBECONFIG='/Users/c/.kube/kind-config-mkpod' NAME READY STATUS RESTARTS AGE aed6b1e8-6ada-4b27-90ad-c7387846b7b0 0/2 Init:0/3 0 0s aed6b1e8-6ada-4b27-90ad-c7387846b7b0 0/2 Init:0/3 0 4s aed6b1e8-6ada-4b27-90ad-c7387846b7b0 0/2 Init:1/3 0 7s aed6b1e8-6ada-4b27-90ad-c7387846b7b0 0/2 Init:2/3 0 10s aed6b1e8-6ada-4b27-90ad-c7387846b7b0 0/2 Init:2/3 0 12s aed6b1e8-6ada-4b27-90ad-c7387846b7b0 0/2 PodInitializing 0 13s aed6b1e8-6ada-4b27-90ad-c7387846b7b0 2/2 Running 0 70s --- % kubectl logs -f aed6b1e8-6ada-4b27-90ad-c7387846b7b0 Defaulted container "test" out of: test, sidecar, clonerefs (init), initupload (init), place-entrypoint (init) Docker in Docker enabled, initializing... ================================================================================ net.ipv6.conf.all.disable_ipv6 = 0 net.ipv6.conf.all.forwarding = 1 Starting Docker: docker. Waiting for docker to be ready, sleeping for 1 seconds. ================================================================================ Done setting up docker in docker. ERROR: (gcloud.auth.activate-service-account) Unable to read file [/etc/service-account/service-account.json]: [Errno 2] No such file or directory: '/etc/service-account/service-account.json' + WRAPPED_COMMAND_PID=271 + wait 271 + make test ./hack/install-helm.sh Downloading https://get.helm.sh/helm-v3.16.2-linux-amd64.tar.gz Verifying checksum... Done. ... ./hack/test-go.sh go: downloading sigs.k8s.io/controller-runtime v0.19.1 go: downloading k8s.io/klog/v2 v2.130.1 go: downloading k8s.io/client-go v0.31.2 go: downloading k8s.io/apimachinery v0.31.2 ... docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: waiting for init preliminary setup: read init-p: connection reset by peer: unknown. ... ```

[crandles]

found/resolved a couple of issues:

• I was mistaken on the vcluster installation path, it is vclusterctl not vcluster
  • Apparently I've been falling back to an alternatively installed vcluster in my CI
• there was a flaw in the version defaulting logic

[cpanato]

/ok-to-test

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cpanato, crandles, vladimirvivien

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files: - ~~[OWNERS](https://github.com/kubernetes-sigs/e2e-framework/blob/main/OWNERS)~~ [cpanato,vladimirvivien] Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment

[crandles]

Thanks for the reviews!

I will try to rebase next time, and I may follow up with an issue about how to run the pr tests locally. I need to spend more time looking at the pj on kind script.