search

knative / serving

Kubernetes-based, scale-to-zero, request-driven compute
https://knative.dev/docs/serving/
Apache License 2.0
5.55k stars 1.16k forks source link

Implement tests with plugable upgrade mechanism #9137

Closed mgencur closed 3 years ago

mgencur commented 4 years ago

Describe the feature

The current upgrade/downgrade tests are orchestrated from Bash. It runs pre-upgrade Golang tests, calls Bash functions to perform the upgrade, runs post-upgrade Golang tests, calls Bash functions for downgrade, runs post-downgrade Golang tests.

This is a proposal for changing the orchestration where the Golang test suite including pre-upgrade, post-upgrade and post-downgrade tests would run as a monolithic test suite, performing upgrades and downgrades using a plugable mechanism.

The scenario would look like this:

Upstream tests would include a default plugin that would call bash scripts for upgrading/downgrading Knative. Downstream tests could provide their own mechanism for upgrades/downgrades, possibly using Golang API. If no extra plugin is provided the test suite would execute the default plugin.

Advantages of this approach:

mgencur commented 4 years ago

/assign

cardil commented 4 years ago

I would go even further. We got upgrade tests in eventing as well, and also merger of those two in knative/operator upgrade tests. That's not ideal.

I would propose to make it fully pluggable test suite. Test suite should be located in some common place, like new knative.dev/pkg/test/upgrade package.

The modified scenario would look like this:

In that way we can do it properly, once. And then reuse that code in Serving, Eventing, Operator and in many places downstream as well.

/cc @houshengbo

coryrc commented 4 years ago

We should use Tekton for orchestration of the various test components.

Supporting a custom-written test orchestrator in Go might be less work than maintaining one in bash, but it's still a custom test orchestrator and we should use a standard tool instead of reinventing our own.

cardil commented 4 years ago

That would require to have working Tekton installation. Additional dependency, additional time to set up.

I think, here we should have a single upgrade test logic, but pluggable, that we can adjust to perform different operations (plugins) on specific steps. In example, we on downstream would like to execute serving & eventing upgrade tests in the same run, go upgrading Serverless Operator and OCP cluster, instead of upgrading serving or eventing separately.

chizhg commented 4 years ago

kntest is under development, we can potentially make it as a subcommand.

Also for a bit more context, we are currently using kubetest2 as the test orchestrator, and it supports plugins just like kn and kubectl. A customer knative-upgrade-tester plugin implementation should also fit in here.

cardil commented 4 years ago

@chizhg PTAL at my refined proposition at knative/pkg#1638

github-actions[bot] commented 3 years ago

This issue is stale because it has been open for 90 days with no activity. It will automatically close after 30 more days of inactivity. Reopen the issue with /reopen. Mark the issue as fresh by adding the comment /remove-lifecycle stale.

cardil commented 3 years ago

/reopen /remove-lifecycle stale

PR #10216 should fix this

knative-prow-robot commented 3 years ago

@cardil: Reopened this issue.

In response to [this](https://github.com/knative/serving/issues/9137#issuecomment-748946407): >/reopen >/remove-lifecycle stale > >PR #10216 should fix this Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.