Pre-release test plan

[jml]

We do manual testing before we release new versions of scope. To make that manual testing as quick & effective as possible, we should have a checklist of things to test. This will help us avoid forgetting to test important things and might also help avoid unnecessary duplication of testing.

Plan should include:

• some sort of guidelines on what counts as a release blocker (otherwise what's the point of testing?)
• documentation on how to get a list of changes between this release & the last (currently part of the pre-release PR)
• documentation on how to get the thing to be tested

Edit:

Process should include:

• way of checking off that a particular aspect has been tested. Our current process scales linearly with the number of team members (since everyone in the team is asked to test), and there's little value in everyone checking the same things.

[abuehrle]

It can be useful to make a test case for the bugs that have been submitted and fixed -- as it will at least provide a pretty good baseline, regression test.

[2opremio]

@jml Answering your (very good) questions from https://github.com/weaveworks/scope/pull/1817#issuecomment-240976140

• What counts as a performance regression?

This is still very ad-hoc as you point out. The right way to solve this IMO is https://github.com/weaveworks/scope/issues/1406

• How do I start a two node cluster?

I personally use multiple VMs

• How do I Run a Scope cluster on ECS?

I run the ECS guide: https://www.weave.works/guides/service-discovery-and-load-balancing-with-weave-on-amazon-ecs-2/

What steps do I need to follow for running https://github.com/microservices-demo/microservices-demo/ ?

I haven't actually done it (@pidster has). But the repository has instructions AFAIR.

Deploy & verify on Weave Cloud: How can I tell how the version on Weave Cloud relates to the release candidate?

I don't think you need to know. I think the test should be specific to probes since Weave Could already operates on CD and not discrete releases. To test probes, blackbox testing is enough.

[jml]

What counts as a performance regression?

This is still very ad-hoc as you point out. The right way to solve this IMO is #1406

Then OK if we strike this from the Release Qualification doc? If it's not testable, then we shouldn't ask the whole team to test it.

[2opremio]

Uhm, sure, although I would like people to at the very least look for very obvious performance degradations. We've had them in the past and we only noticed them when we were about to release. For instance: https://github.com/weaveworks/scope/issues/1406#issuecomment-218179574