search

scylladb / scylla-operator

The Kubernetes Operator for ScyllaDB
https://operator.docs.scylladb.com/
Apache License 2.0
324 stars 159 forks source link

Basic end to end test #220

Closed zimnx closed 3 years ago

zimnx commented 3 years ago

Currently we don't have any automatic test in this repo which would validate even basic scenario such as spawning 1 rack with 3 nodes cluster.

For a start, we should have a test which spawns ScyllaCluster on local minikube. Once we have baseline, all new features should come together with tests.

espindola commented 3 years ago

It would be really nice if the basic tests were not integration tests. Since the only thing an operator does is talk http, it is trivial to mock the other side.

We should still have integration tests, not only with Minikube but with any implementation we support (amazon, google, etc), but with good local tests we don't need to run that during the regular development cycle.

yanniszark commented 3 years ago

@zimnx @espindola may I recommend taking a look at k3d? https://github.com/rancher/k3d

It's super-lightweight, works well with GitHub workflows and allows for multi-node clusters. For instance, we use it in our oidc-authservice repo with great success. I've written a small blog post about it in case you are interested: https://www.arrikto.com/devops/k3d-github-actions-kubernetes-e2e-testing-made-easy/

The test is very lightweight and runs in about 3 minutes (sets up k8s cluster, applies YAML, waits and performs checks). I think it would also work great for testing the Scylla operator end-to-end, without slowing down development speed.

espindola commented 3 years ago

Yannis Zarkadas notifications@github.com writes:

@zimnx @espindola may I recommend taking a look at k3d? https://github.com/rancher/k3d

If I understand it correctly, it is basically a lighter version of minikube? If so I don't see why not moving from minikube to it for both development and first pass at integration tests.

I still think there is value in having dedicated "unit" tests by mocking the API server:

As an example, I wrote an independent implementation of the sample controller. It is at

https://github.com/espindola/sample-controller-from-scratch

$ go test  -count=100 ./pkg/controller
ok      github.com/espindola/sample-controller/pkg/controller   0.245s

$ go test ./pkg/controller -coverprofile cover.out -coverpkg ./pkg/controller,./pkg/kubeapi
ok      github.com/espindola/sample-controller/pkg/controller   0.012s
  coverage: 97.3% of statements in ./pkg/controller, ./pkg/kubeapi

Cheers, Rafael

yanniszark commented 3 years ago

If I understand it correctly, it is basically a lighter version of minikube?

More or less. Plus, it supports multi-node setups, using multiple docker containers.

If so I don't see why not moving from minikube to it for both development and first pass at integration tests.

I would recommend doing so.

I still think there is value in having dedicated "unit" tests by mocking the API server:

Of course :) Both are needed (integration and e2e). Let me share the approaches I have seen so far:

I don't think the approaches above can simulate errors, but you can probably achieve that if client is an interface (which it is).

As an example, I wrote an independent implementation of the sample controller. It is at

This is great! You should do a blog post with that in the future, it would be very educational!

espindola commented 3 years ago

Yannis Zarkadas notifications@github.com writes:

If I understand it correctly, it is basically a lighter version of minikube?

More or less. Plus, it supports multi-node setups, using multiple docker containers.

Minikube start has a --nodes option. It is not working?

I still think there is value in having dedicated "unit" tests by mocking the API server:

Of course :) Both are needed (integration and e2e). Let me share the approaches I have seen so far:

I don't think the approaches above can simulate errors, but you can probably achieve that if client is an interface (which it is).

Semantically I think that client-go's fake is similar to what I did, but I did the mock at the http transport level. I will be the first to admit that the tests I have are a mess and need to be refactored, but I hope it is possible to get full coverage of the controller itself (not the low level APIs) without a complicated infrastructure.

As an example, I wrote an independent implementation of the sample controller. It is at

This is great! You should do a blog post with that in the future, it would be very educational!

I am more of mailing list kind of developer, so I wrote

https://groups.google.com/g/kubernetes-dev/c/2GT5T4MzwFk

Cheers, Rafael