search

kudobuilder / kuttl

KUbernetes Test TooL (kuttl)
https://kuttl.dev
Apache License 2.0
672 stars 86 forks source link

feat: add lazy loading kubeconfigs for Test Steps #540

Closed kumar-mallikarjuna closed 3 months ago

kumar-mallikarjuna commented 3 months ago

What this PR does / why we need it: This PR adds an attribute LazyLoadKubeconfig to the TestStep type which can be used to configure whether the Kubeconfig specified in the TestStep needs to be lazy-loaded, i.e., if the Kubeconfig doesn't need to be loaded at the start of the TestCase eval.

Fixes #539

Tests

.
├── eager
│   └── loading
│       └── 00-teststep.yaml
├── kubeconfig
├── kuttl-test.yaml
└── lazy
    └── loading
        └── 00-teststep.yaml

eager/loading/00-teststep.yaml:

apiVersion: kuttl.dev/v1beta1
kind: TestStep
commands:
- script: echo hi
kubeconfig: test.kubeconfig
kubeconfigLoading: Eager

lazy/loading/00-teststep.yaml:

apiVersion: kuttl.dev/v1beta1
kind: TestStep
commands:
- script: echo hi
kubeconfig: test.kubeconfig
kubeconfigLoading: Lazy

Output on main:

➜  lazy_test git:(main) ✗ go run ../cmd/kubectl-kuttl/main.go test eager
2024/07/04 11:43:22 kutt-test config testdirs is overridden with args: [ eager ]
=== RUN   kuttl
    harness.go:464: starting setup
    harness.go:255: running tests using configured kubeconfig.
    harness.go:278: Successful connection to cluster at: https://127.0.0.1:49639
    harness.go:363: running tests
    harness.go:75: going to run test suite with timeout of 2400 seconds for each step
    harness.go:375: testsuite: eager has 1 tests
=== RUN   kuttl/harness
=== RUN   kuttl/harness/loading
=== PAUSE kuttl/harness/loading
=== CONT  kuttl/harness/loading
    case.go:344: stat /Users/kumarmallikarjuna/Workspace/oss/kuttl/lazy_test/eager/loading/test.kubeconfig: no such file or directory
=== NAME  kuttl
    harness.go:407: run tests finished
    harness.go:515: cleaning up
    harness.go:572: removing temp folder: ""
--- FAIL: kuttl (0.01s)
    --- FAIL: kuttl/harness (0.00s)
        --- FAIL: kuttl/harness/loading (0.00s)
FAIL
exit status 1
➜  lazy_test git:(main) ✗ go run ../cmd/kubectl-kuttl/main.go test lazy
2024/07/04 11:43:27 kutt-test config testdirs is overridden with args: [ lazy ]
=== RUN   kuttl
    harness.go:464: starting setup
    harness.go:255: running tests using configured kubeconfig.
    harness.go:278: Successful connection to cluster at: https://127.0.0.1:49639
    harness.go:363: running tests
    harness.go:75: going to run test suite with timeout of 2400 seconds for each step
    harness.go:375: testsuite: lazy has 1 tests
=== RUN   kuttl/harness
=== RUN   kuttl/harness/loading
=== PAUSE kuttl/harness/loading
=== CONT  kuttl/harness/loading
    case.go:344: stat /Users/kumarmallikarjuna/Workspace/oss/kuttl/lazy_test/lazy/loading/test.kubeconfig: no such file or directory
=== NAME  kuttl
    harness.go:407: run tests finished
    harness.go:515: cleaning up
    harness.go:572: removing temp folder: ""
--- FAIL: kuttl (0.02s)
    --- FAIL: kuttl/harness (0.00s)
        --- FAIL: kuttl/harness/loading (0.00s)
FAIL
exit status 1

Output after these changes:

➜  lazy_test git:(tmp-kubeconfig-lazy-loading) ✗ go run ../cmd/kubectl-kuttl/main.go test eager
2024/07/04 11:41:15 kutt-test config testdirs is overridden with args: [ eager ]
=== RUN   kuttl
    harness.go:464: starting setup
    harness.go:255: running tests using configured kubeconfig.
    harness.go:278: Successful connection to cluster at: https://127.0.0.1:6443
    harness.go:363: running tests
    harness.go:75: going to run test suite with timeout of 2400 seconds for each step
    harness.go:375: testsuite: eager has 1 tests
=== RUN   kuttl/harness
=== RUN   kuttl/harness/loading
=== PAUSE kuttl/harness/loading
=== CONT  kuttl/harness/loading
    case.go:346: stat /Users/kumarmallikarjuna/Workspace/oss/kuttl/lazy_test/eager/loading/test.kubeconfig: no such file or directory
=== NAME  kuttl
    harness.go:407: run tests finished
    harness.go:515: cleaning up
    harness.go:572: removing temp folder: ""
--- FAIL: kuttl (0.01s)
    --- FAIL: kuttl/harness (0.00s)
        --- FAIL: kuttl/harness/loading (0.00s)
FAIL
exit status 1
➜  lazy_test git:(tmp-kubeconfig-lazy-loading) ✗ go run ../cmd/kubectl-kuttl/main.go test lazy
2024/07/04 11:41:18 kutt-test config testdirs is overridden with args: [ lazy ]
=== RUN   kuttl
    harness.go:464: starting setup
    harness.go:255: running tests using configured kubeconfig.
    harness.go:278: Successful connection to cluster at: https://127.0.0.1:6443
    harness.go:363: running tests
    harness.go:75: going to run test suite with timeout of 2400 seconds for each step
    harness.go:375: testsuite: lazy has 1 tests
=== RUN   kuttl/harness
=== RUN   kuttl/harness/loading
=== PAUSE kuttl/harness/loading
=== CONT  kuttl/harness/loading
    logger.go:42: 11:41:18 | loading | Creating namespace: kuttl-test-credible-crow
    case.go:400: failed in step 0-teststep
    case.go:402: failed to lazy-load kubeconfig: stat /Users/kumarmallikarjuna/Workspace/oss/kuttl/lazy_test/lazy/loading/test.kubeconfig: no such file or directory
    logger.go:42: 11:41:18 | loading | loading events from ns kuttl-test-credible-crow:
    logger.go:42: 11:41:18 | loading | Deleting namespace: kuttl-test-credible-crow
=== NAME  kuttl
    harness.go:407: run tests finished
    harness.go:515: cleaning up
    harness.go:572: removing temp folder: ""
--- FAIL: kuttl (5.14s)
    --- FAIL: kuttl/harness (0.00s)
        --- FAIL: kuttl/harness/loading (5.12s)
FAIL
exit status 1

Lazy-loading when the Kubeconfig exists and is same as the global Kubeconfig:

➜  lazy_test git:(main) ✗ go run ../cmd/kubectl-kuttl/main.go test eager
2024/07/04 11:37:44 kutt-test config testdirs is overridden with args: [ eager ]
=== RUN   kuttl
    harness.go:464: starting setup
    harness.go:255: running tests using configured kubeconfig.
    harness.go:278: Successful connection to cluster at: https://127.0.0.1:49639
    harness.go:363: running tests
    harness.go:75: going to run test suite with timeout of 2400 seconds for each step
    harness.go:375: testsuite: eager has 1 tests
=== RUN   kuttl/harness
=== RUN   kuttl/harness/loading
=== PAUSE kuttl/harness/loading
=== CONT  kuttl/harness/loading
    case.go:344: stat /Users/kumarmallikarjuna/Workspace/oss/kuttl/lazy_test/eager/loading/test.kubeconfig: no such file or directory
=== NAME  kuttl
    harness.go:407: run tests finished
    harness.go:515: cleaning up
    harness.go:572: removing temp folder: ""
--- FAIL: kuttl (0.02s)
    --- FAIL: kuttl/harness (0.00s)
        --- FAIL: kuttl/harness/loading (0.00s)
FAIL
exit status 1
➜  lazy_test git:(main) ✗ go run ../cmd/kubectl-kuttl/main.go test lazy
2024/07/04 11:37:50 kutt-test config testdirs is overridden with args: [ lazy ]
=== RUN   kuttl
    harness.go:464: starting setup
    harness.go:255: running tests using configured kubeconfig.
    harness.go:278: Successful connection to cluster at: https://127.0.0.1:49639
    harness.go:363: running tests
    harness.go:75: going to run test suite with timeout of 2400 seconds for each step
    harness.go:375: testsuite: lazy has 1 tests
=== RUN   kuttl/harness
=== RUN   kuttl/harness/loading
=== PAUSE kuttl/harness/loading
=== CONT  kuttl/harness/loading
    logger.go:42: 11:37:50 | loading | Ignoring test.kubeconfig as it does not match file name regexp: ^(\d+)-(?:[^\.]+)(?:\.yaml)?$
    logger.go:42: 11:37:50 | loading | Creating namespace: kuttl-test-needed-ox
    logger.go:42: 11:37:50 | loading | Creating namespace: kuttl-test-needed-ox
    logger.go:42: 11:37:50 | loading/0-teststep | starting test step 0-teststep
    logger.go:42: 11:37:50 | loading/0-teststep | running command: [sh -c echo hi]
    logger.go:42: 11:37:50 | loading/0-teststep | hi
    logger.go:42: 11:37:50 | loading/0-teststep | test step completed 0-teststep
    logger.go:42: 11:37:50 | loading | loading events from ns kuttl-test-needed-ox:
    logger.go:42: 11:37:50 | loading | Deleting namespace: kuttl-test-needed-ox
    logger.go:42: 11:37:55 | loading | Deleting namespace: kuttl-test-needed-ox
kumar-mallikarjuna commented 3 months ago

Hi @porridge , thanks for the quick review. I've incorporated your comments. I found it a little difficult to add tests for this. For now, I've updated the PR description with the outputs comparing the failures to load lazily on main and with these changes. Can you please take a look again?

kumar-mallikarjuna commented 3 months ago

I see the CRDs are in crds/. But I'm unable to re-generate them with make generate. Is this expected?

kumar-mallikarjuna commented 3 months ago

I don't see the failed to lazy-load kubeconfig string in the ouput you pasted? Was the code changed after you copied? Can you please update them?

Apologies, I seem to have mixed up the outputs. I've updated them now.

As for the CRDs, they seem to have gone stale. I didn't even know they were there 😅

Oh, okay.

If you could file a bug to remind us to fix generation and check their freshness in CI, that would be great.

Opened https://github.com/kudobuilder/kuttl/issues/541.

porridge commented 3 months ago

Also please take a look at the failing checks.