[Epic] Zippy as a Release Qualification Test for the Cloud

[philip-stoev]

Introduction

We need a fully-featured Release Qualification Test for the Cloud that can be used to determine if a given release can go out of the door or not.

Requirements

• a 48-hour sufficiently stressfull test
• dozens of sources, views and sinks
• tens of millions of records
• to run against the cloud staging environment
• perform an upgrade in the middle of the test

Background

We have Zippy, which is a framework that is able to create a stress test that does the following:

• creates tables, kafka upsert, Debezium and Postgres sources
• insertion/ingestion into said sources
• creation of materialized views and validation of their output
• creation of sinks based on those materialized views and validation of the sink output by re-ingesting the output of the sink and validating it

This all runs however in mzcompose/docker, for 1 hour at most and needs to run in K8s, against the staging cloud, for days on end.

Required components

• [x] Allow individual Actions performed by Zippy to be parameterized in a Scenario (e.g. max number of records to be ingested per Action, max number of sources to be created, etc.)
• [x] Tune Zippy to run on a larger machine, e.g. 128Gb of ram. This includes tuning the number of sources, views and sinks as well as the ingestion rate.
• [x] Tune Zippy to perform a smaller number of restarts and kills. Currently, a lot of time is spending restarting Mz
• [ ] Have Zippy exercise the initial snapshot functionality of sources by pre-populating a large snapshot that will be ingested at once at CREATE SOURCE/ CREATE MATERIALIZED VIEW time, as opposed to starting from empty and building up over time; In other words, ensure that a CREATE SOURCE statement can be scheduled and run towards the end of the test, against a Kafka topic that has already accumulated a lot of stuff
• [ ] Tune Zippy to run successfully for 48h by providing a gradually-increasing volume of data on top of the original snapshot
• [ ] Allow Zippy to run in cloudtest as opposed to mzcompose/docker
• [ ] Allow Zippy to perform upgrades in the middle of a Zippy test , by replacing the K8s StatefulSet in KinD
• [ ] Allow cloudtest to run against the cloud staging environment, rather than local KinD
• [ ] Have cloudtest start the Mz cluster using the environment controller API, rather than directly installing a K8s StatefulSet for environmentd
• [ ] Have Zippy perform upgrades via the environment controller API in K8s , rather than by replacing the K8s StatefulSet

[uce]

We're blocking further progress on this epic on getting the current set release qualification tests into a stable state.