We are currently e2e testing typically with only one cluster at a time (exception: remote cluster test). Scale testing has so far been a manual effort (#357 )
While a full blown automated scale test is maybe a lot of work for an unclear return, there is value in testing in automated fashion that ECK can orchestrate more than one cluster at a time. We should therefore:
design a test case that runs multiple clusters (10/100?) at a time
define success criteria/metrics for this test case (no errors, but should we also for example measure reconciliation throughput or memory footprint of the operator?)
define test steps (just a simple deploy, deploy + rolling upgrade?)
implement the test case/job
it should be runnable independently from our regular e2e test suite given the additional resource requirements (e.g. via separate go build constraints)
it needs a CI job/environment that creates a cluster appropriately sized for the larger test case
we need to decide when/how to run it. Could be for example part of a pipeline we run after a release candidate build has finished to test for regressions
pebrc
commented
4 years ago
We are currently e2e testing typically with only one cluster at a time (exception: remote cluster test). Scale testing has so far been a manual effort (#357 )
While a full blown automated scale test is maybe a lot of work for an unclear return, there is value in testing in automated fashion that ECK can orchestrate more than one cluster at a time. We should therefore: