[registry-operator]: end-to-end testing framework

[shanduur]

Context and Problem Statement

Ensuring the reliability and functionality of the registry-operator is essential to maintain a seamless experience for developers and DevOps teams utilizing CNCF Distribution Registry instances. End-to-end testing plays a critical role in validating the deployment, scaling, and management capabilities of the registry-operator across various scenarios.

The selection of an appropriate end-to-end testing framework is paramount to establish robust testing standards and practices. Factors such as adherence to language idioms, support for testing complex workflows, community adoption, and extensibility are crucial considerations in this decision-making process.

Considered Options

• Radish (Python)

  • Description: Radish is a behavior-driven development (BDD) framework for Python, offering a structured approach to end-to-end testing.
  • Pros: BDD approach fosters collaboration between stakeholders, human-readable scenarios, support for Gherkin syntax.
  • Cons: Limited adoption and community support compared to other Python testing frameworks.

• pytest-bdd (Python)

  • Description: pytest-bdd is a plugin for pytest that allows for behavior-driven testing using Gherkin syntax.
  • Pros: Seamless integration with pytest, extensive plugin ecosystem, active development and community support.
  • Cons: Potential learning curve for teams unfamiliar with pytest and BDD principles.

• behave (Python)

  • Description: Behave is another Python BDD framework, enabling collaboration between stakeholders through human-readable behavior specifications.
  • Pros: Well-established in the Python BDD community, supports Gherkin syntax, extensive documentation.
  • Cons: Configuration overhead, may be slower compared to pytest-based frameworks.

• Ginkgo and Gomega (Go)

  • Description: Ginkgo is a BDD-style testing framework for Go, and Gomega is its companion matcher library.
  • Pros: Integrates well with Go's testing package, readable and expressive test specifications.
  • Cons: No support for Gherkin syntax, promotes non-idiomatic Go practices.

• testing (Go)

  • Description: The standard testing package in Go provides a simple and effective way to write tests.
  • Pros: Minimalist approach, built-in to the Go language, lightweight and fast test execution.
  • Cons: Lacks BDD-style features, may require more effort to structure tests for complex scenarios compared to BDD frameworks.

[shanduur]

I've done some initial investigation and I really enjoy pytest-bdd. It has huge community, easily integrates with feature files written with Gherkin syntax, and as it is a pytest plugin, it can benefit from multiple other pytest plugins.

Also, dependency management with Poetry is frictionless. It also easily integrates with linting tools, providing single file for configuration management for many, e.g. mypy, black, isort.

tests
├── README.md
├── e2e
│   ├── __init__.py
│   ├── features
│   │   └── sanity.feature
│   ├── framework
│   │   ├── __init__.py
│   │   ├── kubernetes.py
│   │   └── utils.py
│   └── test_sanity.py
├── poetry.lock
└── pyproject.toml

[malgorzatadutka]

I'd like to use testing (Go) for the unit tests, Ginkgo doesn't have integration with feature files. Radish and pytest-bdd are quite similar. I would choose pytest-bdd with the community support.

[purpleturtll]

+1 for pytest-bdd. Seems to be the most "getting out of your way" when writing scenarios.

[shanduur]

Reopening.

One more option worth considering:

• Kyverno Chainsaw (low-code, YAML)
  • Description: Kyverno Chainsaw is a low-code and declarative tool for testing Kyverno policies and running end-to-end tests in Kubernetes clusters.
  • Pros: easy to use (low-code), declarative, advanced assertions, more flexible than KUTTL, built for testing operators;
  • Cons: strongly opinionated.