search

operator-framework / operator-sdk

SDK for building Kubernetes applications. Provides high level APIs, useful abstractions, and project scaffolding.
https://sdk.operatorframework.io
Apache License 2.0
7.1k stars 1.73k forks source link

[SDK Stability] Add plugin-specific scaffolding test interface to decide if a plugin version should be updated or not #6738

Open itroyano opened 2 months ago

itroyano commented 2 months ago

Feature Request

Describe the problem you need a feature to resolve.

Currently a plugin can fail on basic scaffolding operator-sdk init --plugins=<plugin> returning 1 and/or creating no files, or fail partially creating an incomplete operator project which doesn't meet that specific plugin's requirements.

Describe the solution you'd like.

Add a mechanism in CI (e.g. on pull request) to test any current or future plugin for

  1. Basic scaffolding and exit code 0.
  2. Advanced / plugin-specific scaffolding to be implemented or delegated to be run by the plugin itself (?).

If a plugin passes both criteria - update its version in go.mod and continue to test other plugins and components. otherwise reject the plugin version, keep the existing one, and continue to test other plugins and components.

Optionally - bot creates an issue in the repo of the plugin, so maintainers can take a look.

varshaprasad96 commented 2 months ago

@itroyano Thanks for raising the issue. Some info on how this is done today:

When a new plugin is released (for ex. go/) and we would like to pull that into SDK, the developer:

  1. Creates a new branch and tries updating the go.mod.
  2. Updates the template for scaffolding plugin in cli.go - https://github.com/operator-framework/operator-sdk/blob/de40e5091cc33bbd08592aa1da0007d9bfad6eb9/internal/cmd/operator-sdk/cli/cli.go#L84
  3. After which, the testdata (https://github.com/operator-framework/operator-sdk/tree/de40e5091cc33bbd08592aa1da0007d9bfad6eb9/testdata) is updated using make generate. During the updation of testdata, the operator-sdk command then calls in the new plugin version and scaffolds out the files using the operator-sdk <commands> (https://github.com/operator-framework/operator-sdk/blob/de40e5091cc33bbd08592aa1da0007d9bfad6eb9/hack/generate/samples/generate_testdata.go#L35-L60).

With this change if there is any issue during scaffolding the tests do not pass, and CI errors. There is no automation here though, but we do catch scaffolding issues before merging anything.

This works well currently with Helm and Go plugins. But with ansible (recently moved out external plugin), generation of testdata gets a bit complicated since we scaffold the templates in the external repository: https://github.com/operator-framework/ansible-operator-plugins/blob/4e8c78d21b6017ab0a280e3d274d97edf50b8fc4/hack/generate/samples/ansible/generate.go#L107. The scaffolded operator is tested in the plugin repository for any plugin update (https://github.com/operator-framework/ansible-operator-plugins/blob/main/test/e2e/ansible/suite_test.go) before updating it in SDK.

Having improvements in testing external scaffolds would be helpful, however to do so we need to discuss on how we would like to split tests between the external plugin and SDK repos.