[Discussion] Determine strategy for regression testing on old data and long-running systems

[Darwinkel]

As discussed in the retro, it's become clear that a number of subtle bugs occur with long-running production systems. These are mostly either caused by:

• a large (and old) dataset, both in Postgres and XTDB;
  • Example: zcert's suspected mysterious hostname with a trailing dot
• old configuration settings which were not properly adjusted;
  • Example: the new finding type boefjes which are not automatically enabled on existing systems
• the internet being chaotic
  • Example: many instances of unexpected OOI input/output such as uncommon SPF records

Clean installs are generally tested thoroughly by yours truly. Testing on long-running systems with a big, real-world dataset is currently a blind spot in the QA process, for which there is currently no good (automatic) testing flow. We receive some useful feedback from the community when release candidates are published, and when we upgrade our test servers. But it would be nice if we made this a structural part of the QA process.

It's a non-trivial endeavor, and it's necessary to discuss and gather ideas for how we could do this. Feel free to post your thoughts in this ticket :)

[Darwinkel]

In addition: the new Boefje runner design raises similar questions regarding the testability of various container runtimes. This should also be discussed and refined. The general idea is to start with a "simple" unit-test matrix similar to how we currently test multiple Python versions.

[Darwinkel]

Related: https://github.com/minvws/nl-kat-coordination/issues/1757 (and https://github.com/minvws/nl-rt-tim-abang/discussions/864)

[Darwinkel]

Meeting notes 05-10-2023:

Two step approach:

• Phase 1: semi-automatic upgrades/migrations of some running systems (maybe kattest) when we want to test a release (candidate)
  • @dekkers already has some basic scripts in place: maybe we can expand them and publish them in the repository?
• Phase 2: A good DevOps approach where every PR/branch can be easily spinned up on platform X with dataset Y
  • GitLab has facilities for this
  • Can we write some helpers for GitHub Codespaces or Gitpod?
  • After we have implemented Kubernetes support we could use Kubernetes to easily deploy and upgrade KAT environments

Backup/restore flow

• Basically: KAT needs a good method to backup and restore Postgres, XTDB, and Bytes' storage
  • Step 1: Helper scripts for exporting/importing through the CLI