search

kudobuilder / kudo

Kubernetes Universal Declarative Operator (KUDO)
https://kudo.dev
Apache License 2.0
1.18k stars 101 forks source link

Separate E2E and operator tests #1540

Closed nfnt closed 4 years ago

nfnt commented 4 years ago

What this PR does / why we need it: This runs operator tests as a separate test in parallel with the other tests. It makes operator tests independent from E2E test results and their failures distinguishable.

Fixes #1539

nfnt commented 4 years ago

I verified that test logs and artifacts are collected for E2E tests as well as operator tests.

kensipe commented 4 years ago

could we get clarity on what is expected in each of these categories of tests? unit, integration, e2e and operator? isn't operator tests e2e tests? when would I choose to create an integration vs e2e test? are operator tests, different than e2e tests? or are they basically the same but have different audiences and concerns?

If we are going expand the category of testing options... should be document the purpose and guide the community?

nfnt commented 4 years ago

Operator tests have the same setup but different concerns from E2E tests. The motivation is to get earlier feedback when these tests fail. In the past, the operator tests were only run if the E2E tests succeeded. This made it hard for larger PRs to distinguish between changes that fix operators and changes that fix E2E tests.