Open bobcatfish opened 4 years ago
Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten
.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close
.
/lifecycle rotten
Send feedback to tektoncd/plumbing.
This is a roadmap item: https://github.com/tektoncd/pipeline/blob/master/roadmap.md
/lifecycle frozen
People should be able to write "unit tests" for their Tasks and Pipelines (I have no idea what these would actually look like!)
Quick passing thought - now that we have both params and results (and also workspaces), we could provide tools / libs to write "tests" that exercise Tasks and Pipelines with certain inputs and check for expected outputs and workspace contents. Maybe not as much of "unit test" as we'd like but probably a useful test for many cases 🤔
(Also wondering if there might be some overlap with the catalog tiers work @jerop )
It would be great to have some mechanism for testing tasks and pipelines. I think there are two scenarios worth considering:
TaskRunTest
or PipelineRunTest
that would take the TaskRun
or PipelineRun
definition and list of assertions to be tested and at the end indicate whether the assertions are satisfied.TaskRun
and execute task without deploying it into tekton could be useful. Something that will interpolate parameters and run the image and script from each step. Perhaps incorporating some form of tests like bats.It would be great to have some mechanism for testing tasks and pipelines. I think there are two scenarios worth considering:
- testing against the task's or pipeline's results, maybe with a CR like
TaskRunTest
orPipelineRunTest
that would take theTaskRun
orPipelineRun
definition and list of assertions to be tested and at the end indicate whether the assertions are satisfied.- testing tasks outside of tekton. I think having a mechanism that can evaluate a
TaskRun
and execute task without deploying it into tekton could be useful. Something that will interpolate parameters and run the image and script from each step. Perhaps incorporating some form of tests like bats.
For (2), something like buildkit-tekton
could probably help 👼🏼
This would be really helpful when maintaining a catalog with a lot of Tasks and Pipelines. It would be great to be able to actually test the following scenarios in isolation (without a cluster):
Pipelines:
I'm a bit puzzled whether this would be "configuration testing" or not but, something like this would really help automating checks/validations when maintaining a huge catalog of tasks/pipelines. Also, we had a look at https://github.com/tektoncd/catlin and it might cover what I described above (not sure if that's what you are planning to do with that tool or if you'd like to enhance the testing tools separately).
I'm not exactly sure what this looks like but I wanted to start tracking this here if ppl have ideas.
Basically I think that we should have a great testing story around Tekton Pipelines, e.g.:
For inspiration, I really like the way that OPA polices can have tests written for them: https://www.openpolicyagent.org/docs/latest/how-do-i-test-policies/ Unfortunately Tekton Pipelines is totally different b/c it isn't a language (or is it????) and it's mostly side effect driven (the whole point is to execute stuff that causes changes in the state of the world) but I feel confident that if we really spend some time thinking about this we can make something cool!