StackStorm v3.4.0 Pre-release Testing

[blag]

We're not yet ready to prepare the StackStorm v3.4 release and start the pre-release testing, but I'm opening this issue now in preparation. We still need to include a list of major features or areas that saw a large amount of code churn, as well as a complete list of the changes from the changelog so users can test each change. See the previous pre-release testing discussion for an example.

Release Process Preparation

Per Release Management Schedule @blag is the Release Manager and @amanda11 is Assisting for v3.4. They will freeze the master for the major repositories in StackStorm org, follow the StackStorm Release Process which is now available to public, accompanied by the Useful Info for Release managers. Communication is happening in #releasemgmt and #development Slack channels. The first step is pre-release manual user-acceptance testing for v3.4dev.

Why Manual testing?

StackStorm is very serious about testing and has a lot of it:

• Linting, Compile, and Unit tests
• Integration tests
• User Interface tests
• Packaging tests
• Deployment Integrity checks
• Smoke tests
• end-to-end tests when automation spins up new AWS instance for each OS/flavor we support, installs real st2 like user would and runs set of st2tests (for each st2 PR, nightly, periodically, during release), including end-to-end tests ChatOps

See st2ci and st2cd for more examples and workflows about how StackStorm automation is used to test StackStorm (dogfooding).

That's a perfect way to verify what we already know and codify expectations about how StackStorm should function.

However it's not enough. There are always new unknowns to discover, edge cases to experience and tests to add. Hence, manual Exploratory Testing is an exercise where entire team gathers together and starts trying (or breaking) new features before the new release. Because we're all different, perceive software differently and try different things we might find new bugs, improper design, oversights, edge cases and more.

This is how StackStorm previously managed to land less major/critical bugs into production.

TL;DR

See the Testing Process, where it will walk you through:

• Spinning up a StackStorm instance using st2vagrant
• Running StackStorm's self-check script
• Running through the basic manual testing steps

Additionally, try to use that StackStorm instance as you normally would, maybe try to break it in new and interesting ways that you haven't tried before, and report any regressions found comparing to v3.3.

Extra points for PR hotfixes, reporting entirely new bugs, and missing test cases!

Specific things to check

• [ ] Proper integration between st2web and st2flow
  • [ ] The blue + button to create a new workflow is visible in the actions list in the Actions tab
  • [ ] An action that is an Orquesta workflow has a blue EDIT button in the action details pane
• [ ] Python 2 deprecation
  • [ ] ST2 should install packs with Python 3.6 by default
  • [ ] The python3 boolean option should not appear in the web UI in the packs.install action
  • [ ] The st2 pack install command should print a warning when passed the --python3 flag
  • [ ] New sensors are run with Python 3.6
  • [ ] Installation on U16 requires a PPA. A warning and confirmation will be required before it is installed by the ST2 installers, but flags/variables are available to automatically install as well.
• [ ] st2resultstracker was removed
  • [ ] st2ctl status does not show any running st2resultstracker processes
  • [ ] st2ctl restart does not start any st2resultstracker processes
• [ ] RBAC
  • [ ] RBAC not enabled by default by installers
  • [ ] RBAC can be enabled by following latest documentation
• [ ] LDAP
• [ ] Add support for GitLab SSH URLs

[arm4b]

Duplicate of https://github.com/StackStorm/discussions/issues/66