search

oras-project / oras

OCI registry client - managing content like artifacts, images, packages
https://oras.land
Apache License 2.0
1.44k stars 174 forks source link

Building up E2E Tests for ORAS #523

Closed shizhMSFT closed 1 year ago

shizhMSFT commented 2 years ago

Discussed in https://github.com/oras-project/oras/discussions/485

Originally posted by **qweeah** August 9, 2022 ## What is Needed End-to-end tests that continuously validate the functional correctness of oras features. ## Why it's Needed - As the number of supported commands grows, it's impossible to manually validate all the features before releasing. - Although we have unit tests for CLI, things might still go wrong after integration. Having E2E tests would at least cover high-value use cases and reduce regression. ## How to ### Setup Test Environment The E2E tests should run against oras CLI binary with [oras-distribution](https://github.com/oras-project/distribution) as the backend. ### Choose Test Framework Technically, there are two different directions of tools we use to build E2E test suite: #### 1) Use golang-native test framework to run test based on Cobra command execution calls [ginkgo](https://github.com/onsi/ginkgo) + [gomega](https://github.com/onsi/gomega) - Pros: - Based on golang standard testing package, which matches tech-stack of most active contributors. - No change in github actions. - Applied by many state-of-the-art CLI projects like [kubectl](https://github.com/kubernetes/kubernetes) and [frp](https://github.com/fatedier/frp) so we have a lot of existing good practices to learn from. - Cons: - Use Cobra command execution call to mock user input, not 100% E2E. - Brings new dependencies in only for testing. - Won't be able to be reused by oras-py or other registry CLI tools sharing same UX as oras. #### 2) Use common Behaviour-Driven Development (BDD) tools to test the built out binary Gherkin + [godog](https://github.com/cucumber/godog) - Pros: - Testing oras as an CLI application, 100% E2E. - Idomatic BDD framework. - The latest cucumber binary can be downloaded during test time, which brings no depedency into oras. - Test cases(specs) can be reused by oras-py or other registry CLI tools sharing same UX as oras. - Cons: - Extra one-time setup effort in Github actions. - Not Golang-native. (As far as I know) No expert in the current active contributors. Both options provides expressive assertion that can meet oras' testing need. **Future Works** Below items are valuable but not in scope of this discussion: - Benchmark Test & Profiling. - E2E tests based on registry services from different cloud providers. If you have any other ideas or suggestions on building the E2E test suite for oras, feel free to response below. Any comments are highly appreciated 🍻
qweeah commented 2 years ago

@shizhMSFT Can you assign this to me, thanks.

qweeah commented 2 years ago

The work items of E2E tests are listed out as below:

Infrastructure

Utility package for writing E2E tests

Sample Tests (oras artifact tests will be after OCI artifact is supported)

Documentation