search

OpenLiberty / open-liberty-operator

Eclipse Public License 2.0
28 stars 37 forks source link

Run acceptance tests (FVT) with OCP 4.15 #550

Closed leochr closed 7 months ago

leochr commented 7 months ago

Run acceptance tests with OCP 4.15 and WLO 1.3.1 (latest release)

leochr commented 7 months ago

All tests except the license test passed on OCP 4.15.0 on OCP on X, P and Z. license test is expected to fail since WLO 1.3.1 doesn't support this license functionality (it's in the main branch, not released yet).

WLO CI Build 1957

X results:

****** Testing on X so starting scorecard tests...
--------------------------------------------------------------------------------
Image:      quay.io/operator-framework/scorecard-test-kuttl:latest
Labels:
    "suite":"kuttlsuite"
    "test":"basic-test"
Results:
    Name: affinity
    State: pass

    Name: probe
    State: pass

    Name: volume
    State: pass

    Name: trace
    State: pass

    Name: storage
    State: pass

    Name: statefulset-strategy
    State: pass

    Name: sso2-providers
    State: pass

    Name: sso1-social
    State: pass

    Name: sidecar-stateful
    State: pass

    Name: sidecar-deployment
    State: pass

    Name: service-account-name
    State: pass

    Name: semeru
    State: pass

    Name: security-context
    State: pass

    Name: routes
    State: pass

    Name: route-certificate
    State: pass

    Name: pullpolicy
    State: pass

    Name: service-certificate
    State: pass

    Name: service-account2
    State: pass

    Name: service-types
    State: pass

    Name: service-certificate-annotations
    State: pass

    Name: service-binding2
    State: pass

    Name: service-binding1
    State: pass

    Name: init-stateful
make: *** [Makefile:353: test-pipeline-e2e] Error 1
    State: pass

    Name: monitor
    State: pass

    Name: manage-tls
    State: pass

    Name: persistent-storage
    State: pass

    Name: nodeport
    State: pass

    Name: license
    State: fail

    Errors:
        resource matched of kind: /v1, Kind=Pod

    Name: network-policy-multiple-apps
    State: pass

    Name: knative2
    State: pass

    Name: network-policy
    State: pass

    Name: knative
    State: pass

    Name: deployment-strategy
    State: pass

    Name: env
    State: pass

    Name: image-stream
    State: pass

    Name: dump
    State: pass

    Name: init-deployment
    State: pass

    Name: auto1
    State: pass

    Name: basic
    State: pass

    Name: auto2
    State: pass

    Name: auto3
    State: pass

    Name: annotations
    State: pass

P results:

****** Testing on P so starting scorecard tests...
--------------------------------------------------------------------------------
Image:      quay.io/operator-framework/scorecard-test-kuttl:latest
Labels:
    "suite":"kuttlsuite"
    "test":"basic-test"
Results:
    Name: storage
    State: pass

    Name: affinity
    State: pass

    Name: statefulset-strategy
    State: pass

    Name: sso2-providers
    State: pass

    Name: sso1-social
    State: pass

    Name: sidecar-stateful
    State: pass

    Name: sidecar-deployment
    State: pass

    Name: service-types
    State: pass

    Name: service-certificate-annotations
    State: pass

    Name: service-certificate
    State: pass

    Name: service-binding2
    State: pass

    Name: service-binding1
    State: pass

    Name: service-account2
    State: pass

    Name: service-account-name
    State: pass

    Name: semeru
    State: pass

    Name: security-context
    State: pass

    Name: routes
    State: pass

    Name: route-certificate
    State: pass

    Name: pullpolicy
    State: pass

    Name: probe
    State: pass

    Name: persistent-storage
    State: pass

    Name: nodeport
    State: pass

    Name: network-policy-multiple-apps
    State: pass

    Name: network-policy
    State: pass

    Name: monitor
    State: pass

    Name: manage-tls
    State: pass

    Name: license
    State: fail

    Errors:
        resource matched of kind: /v1, Kind=Pod

    Name: knative2
    State: pass

    Name: knative
    State: pass

    Name: init-stateful
    State: pass

    Name: init-deployment
    State: pass

    Name: image-stream
    State: pass

    Name: env
    State: pass

    Name: dump
    State: pass

    Name: deployment-strategy
    State: pass

    Name: basic
    State: pass

    Name: auto3
    State: pass

    Name: auto2
    State: pass

    Name: auto1
    State: pass

    Name: annotations
    State: pass

    Name: volume
    State: pass

    Name: trace
    State: pass

Z results:

--- FAIL: kuttl (1065.64s)
    --- FAIL: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/affinity (36.82s)
        --- PASS: kuttl/harness/volume (47.03s)
        --- PASS: kuttl/harness/trace (31.18s)
        --- PASS: kuttl/harness/storage (51.68s)
        --- PASS: kuttl/harness/statefulset-strategy (12.12s)
        --- PASS: kuttl/harness/sso2-providers (6.31s)
        --- PASS: kuttl/harness/sso1-social (29.27s)
        --- PASS: kuttl/harness/sidecar-stateful (40.69s)
        --- PASS: kuttl/harness/sidecar-deployment (36.81s)
        --- PASS: kuttl/harness/service-types (18.64s)
        --- PASS: kuttl/harness/service-certificate-annotations (3.95s)
        --- PASS: kuttl/harness/service-certificate (3.92s)
        --- PASS: kuttl/harness/service-binding2 (2.10s)
        --- PASS: kuttl/harness/service-binding1 (2.11s)
        --- PASS: kuttl/harness/service-account2 (18.45s)
        --- PASS: kuttl/harness/service-account-name (14.97s)
        --- PASS: kuttl/harness/semeru (54.14s)
        --- PASS: kuttl/harness/security-context (7.49s)
        --- PASS: kuttl/harness/routes (5.49s)
        --- PASS: kuttl/harness/route-certificate (20.70s)
        --- PASS: kuttl/harness/pullpolicy (11.74s)
        --- PASS: kuttl/harness/probe (41.75s)
        --- PASS: kuttl/harness/init-stateful (46.73s)
        --- PASS: kuttl/harness/persistent-storage (26.21s)
        --- PASS: kuttl/harness/nodeport (1.44s)
        --- PASS: kuttl/harness/network-policy-multiple-apps (0.71s)
        --- PASS: kuttl/harness/network-policy (10.51s)
        --- PASS: kuttl/harness/monitor (13.61s)
        --- PASS: kuttl/harness/manage-tls (47.59s)
        --- FAIL: kuttl/harness/license (209.55s)
        --- PASS: kuttl/harness/knative2 (8.87s)
        --- PASS: kuttl/harness/knative (5.00s)
        --- PASS: kuttl/harness/deployment-strategy (6.52s)
        --- PASS: kuttl/harness/init-deployment (9.29s)
        --- PASS: kuttl/harness/image-stream (15.58s)
        --- PASS: kuttl/harness/env (11.47s)
        --- PASS: kuttl/harness/dump (19.24s)
        --- PASS: kuttl/harness/auto2 (13.34s)
        --- PASS: kuttl/harness/basic (24.79s)
        --- PASS: kuttl/harness/auto3 (50.50s)
        --- PASS: kuttl/harness/auto1 (40.82s)
        --- PASS: kuttl/harness/annotations (6.06s)
[PASSED] kind-e2e-run
[FAILED] ocp-e2e-run-P: Exited (2) 47 seconds ago
[FAILED] ocp-e2e-run-X: Exited (2) 37 seconds ago
[FAILED] ocp-e2e-run-Z: Exited (2) 53 seconds ago
leochr commented 7 months ago

Kind tests also passed on Kubernetes 1.28 (used by OCP 4.15). license test is expected to fail since WLO 1.3.1 doesn't support this license functionality (it's in the main branch, not released yet).

WLO CI Build 1961

****** Starting kind scorecard tests...
--------------------------------------------------------------------------------
Image:      [quay.io/operator-framework/scorecard-test-kuttl:latest](http://quay.io/operator-framework/scorecard-test-kuttl:latest)
Labels:
    "suite":"kuttlsuite"
    "test":"basic-test"
Results:
    Name: affinity
    State: pass

    Name: persistent-storage
    State: pass

    Name: volume
    State: pass

    Name: storage
    State: pass

    Name: statefulset-strategy
    State: pass

    Name: sso2-providers
    State: pass

    Name: sso1-social
    State: pass

    Name: sidecar-stateful
    State: pass

    Name: sidecar-deployment
    State: pass

    Name: service-types
    State: pass

    Name: service-certificate-annotations
    State: pass

    Name: service-certificate
    State: pass

    Name: service-binding2
    State: pass

    Name: service-binding1
    State: pass

    Name: service-account2
    State: pass

    Name: service-account-name
    State: pass

    Name: semeru
    State: pass

    Name: security-context
    State: pass

    Name: resource-conflict
    State: pass

    Name: pullpolicy
    State: pass

    Name: probe
    State: pass

    Name: ingress-manage-tls
    State: pass

    Name: nodeport
    State: pass

    Name: monitor
    State: pass

    Name: license
    State: fail

    Errors:
        resource matched of kind: /v1, Kind=Pod

    Name: knative2
    State: pass

    Name: knative
    State: pass

    Name: kind-network-policy-multiple-apps
    State: pass

    Name: kind-network-policy
    State: pass

    Name: init-stateful
    State: pass

    Name: init-deployment
    State: pass

    Name: basic
    State: pass

    Name: ingress-certificate
    State: pass

    Name: ingress
    State: pass

    Name: env
    State: pass

    Name: deployment-strategy
    State: pass

    Name: auto2
    State: pass

    Name: auto3
    State: pass

    Name: auto1
    State: pass

    Name: annotations
    State: pass
leochr commented 7 months ago

Closing as completed