search

quintush / helm-unittest

BDD styled unit test framework for Kubernetes Helm charts as a Helm plugin.
MIT License
344 stars 69 forks source link

Unit test fail randomly #131

Open laptua opened 3 years ago

laptua commented 3 years ago

Unit test fail randomly

test data contains array of only one element. Test case override to add more elements. Repeatedly run the same test without change. Testcase occasionally failed. If test data contains multiple elements, i.e. without override with set command in testcase. Problem do not exist.

Testdata

    ingress:
      enable: false
      host: test.com
      path: /test
      port: web
    extensions:
      ingress:
          - paths:
              - /api/building-calculator/health
            port: web
            matchDirective: "PathPrefixStrip"
            rewrite: /actuator/health

Testcase

suite: Test Ingress extensions 2 or more Paths without rewrite
templates:
  - deploymentSpec/extensions/ingress.yaml
values:
  - ./values/ingress_ex_default.yaml
tests:
  - it: ingress documents (one for ingress, one for middleware)
    asserts:
      - hasDocuments:
          count: 4
  - it: ingress kind and path
    set:
      deploymentSpec.app.extensions.ingress[0].matchDirective: PathPrefixStrip
      deploymentSpec.app.extensions.ingress[0].paths:
        - /welcome/api/wiremock
        - /loyalty/api/wiremock
    documentIndex: 0
    asserts:
      - equal:
          path: metadata.annotations.kubernetes\.io/ingress\.class
          value: traefik
      - isKind:
          of: Ingress
      - equal:
          path: spec.rules[0].http.paths[0].path
          value: /welcome/api/wiremock
      - isNull:
          path: spec.rules[0].host
      - equal:
          path: spec.rules[0].http.paths[0].backend.servicePort
          value: 80
      - equal:
          path: spec.rules[0].http.paths[0].backend.serviceName
          value: base-helm-chart-app-svc

Repeat run the same test without change, testcase fail randomly

 PASS  Test Ingress Chain   tests/app-ingress_chain_test.yaml
 PASS  Test Ingress default matchDirective value to PathPrefixStrip tests/app-ingress_default_test.yaml
 PASS  Test Ingress PathPrefix  tests/app-ingress_pathprefix_test.yaml
 PASS  Test Ingress PathPrefixStrip tests/app-ingress_pathprefixstrip_test.yaml
 PASS  Test Ingress ReplacePathRegex    tests/app-ingress_replacepathregex_test.yaml
 PASS  Test Ingress extensions 2 or more Paths without rewrite  tests/ingress_ex_default2_test.yaml
 PASS  Test Ingress extensions 1 path with rewrite which force matchDirective to PathPrefix tests/ingress_ex_default_test.yaml

Charts:      1 passed, 1 total
Test Suites: 7 passed, 7 total
Tests:       29 passed, 29 total
Snapshot:    0 passed, 0 total
Time:        51.38539ms
 PASS  Test Ingress Chain   tests/app-ingress_chain_test.yaml
 PASS  Test Ingress default matchDirective value to PathPrefixStrip tests/app-ingress_default_test.yaml
 PASS  Test Ingress PathPrefix  tests/app-ingress_pathprefix_test.yaml
 PASS  Test Ingress PathPrefixStrip tests/app-ingress_pathprefixstrip_test.yaml
 PASS  Test Ingress ReplacePathRegex    tests/app-ingress_replacepathregex_test.yaml
 FAIL  Test Ingress extensions 2 or more Paths without rewrite  tests/ingress_ex_default2_test.yaml
    - ingress extension middleware

        - asserts[3] `equal` fail
            Template:   base-helm-chart/templates/deploymentSpec/extensions/ingress.yaml
            DocumentIndex:  0
            Path:   spec.stripPrefix.prefixes[0]
            Expected to equal:
                /welcome/api/wiremock
            Actual:
                null
            Diff:
                --- Expected
                +++ Actual
                @@ -1,2 +1,2 @@
                -/welcome/api/wiremock
                +null

        - asserts[4] `equal` fail
            Template:   base-helm-chart/templates/deploymentSpec/extensions/ingress.yaml
            DocumentIndex:  0
            Error:
                [1] :
                - null

 PASS  Test Ingress extensions 1 path with rewrite which force matchDirective to PathPrefix tests/ingress_ex_default_test.yaml

Charts:      1 failed, 0 passed, 1 total
Test Suites: 1 failed, 6 passed, 7 total
Tests:       1 failed, 28 passed, 29 total
Snapshot:    0 passed, 0 total
Time:        52.417587ms
d3adb5 commented 2 years ago

I get the same kind of behavior, but with one particular test that uses contains:

 FAIL  Keycloak StatefulSet     keycloak1/tests/statefulset_test.yaml
        - init container downloads OJDBC driver from archive when DB vendor is oracle

                - asserts[0] `contains` fail
                        Template:       keycloak1/charts/keycloak/templates/statefulset.yaml
                        DocumentIndex:  0
                        Error:
                                expect 'spec.template.spec.initContainers' to be an array, got:
                                null

This works fine most of the time, but sometimes I get the above error.

d3adb5 commented 2 years ago

Also randomly I get the following Go error:

panic: runtime error: index out of range [0] with length 0

goroutine 1 [running]:
github.com/lrills/helm-unittest/pkg/unittest.(*Assertion).Assert(0xc00055a6c0, 0x20?, {0x17bdf40?, 0xc0003dae40}, 0x1, {0x0?, 0x0}, 0xc0006f8be0)
        /root/project/pkg/unittest/assertion.go:53 +0x796
github.com/lrills/helm-unittest/pkg/unittest.(*TestJob).runAssertions(0xc0000c0500, 0xc000ae88c0?, {0x17bdf40, 0xc0003dae40}, 0x8?, {0x0, 0x0}, 0x0)
        /root/project/pkg/unittest/test_job.go:642 +0x139
github.com/lrills/helm-unittest/pkg/unittest.(*TestJob).RunV3(0xc0000c0500, 0xc000ae88c0, 0xc000132e40, 0xc0?, 0xc0006f9400)
        /root/project/pkg/unittest/test_job.go:253 +0x237
github.com/lrills/helm-unittest/pkg/unittest.(*TestSuite).runV3TestJobs(0xc0000dea00, {0x7ffc13386367, 0x7}, 0x40d627?, 0x0)
        /root/project/pkg/unittest/test_suite.go:231 +0x145
github.com/lrills/helm-unittest/pkg/unittest.(*TestSuite).RunV3(0xc0000dea00, {0x7ffc13386367, 0x7}, 0x17bece0?, 0x18?, 0xc0003476c0)
        /root/project/pkg/unittest/test_suite.go:109 +0xad
github.com/lrills/helm-unittest/pkg/unittest.(*TestRunner).runV3SuitesOfChart(0xc000a8fbb8, {0xc0000e2340, 0x7, 0x7ffc13386367?}, {0x7ffc13386367, 0x7})
        /root/project/pkg/unittest/test_runner.go:301 +0x185
github.com/lrills/helm-unittest/pkg/unittest.(*TestRunner).RunV3(0xc000a8fbb8, {0xc0001a61c0, 0xd, 0x0?})
        /root/project/pkg/unittest/test_runner.go:169 +0x1f7
main.glob..func1(0x21c5740?, {0xc0001a61c0, 0xd, 0xe})
        /root/project/cmd/helm-unittest/helm_unittest.go:89 +0x21e
github.com/spf13/cobra.(*Command).execute(0x21c5740, {0xc00003a100, 0xe, 0xe})
        /go/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:876 +0x67b
github.com/spf13/cobra.(*Command).ExecuteC(0x21c5740)
        /go/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:990 +0x3b4
github.com/spf13/cobra.(*Command).Execute(...)
        /go/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:918
main.main()
        /root/project/cmd/helm-unittest/helm_unittest.go:100 +0x25
Error: plugin "unittest" exited with error