search

StackStorm / community

Async conversation about ideas, planning, roadmap, issues, RFCs, etc around StackStorm
https://stackstorm.com/
Apache License 2.0
8 stars 3 forks source link

StackStorm v3.3.0 Pre-release Testing #52

Closed nmaludy closed 2 years ago

nmaludy commented 3 years ago

We're ready to prepare the StackStorm v3.3 release and starting pre-release testing..

Release Process Preparation

Per Release Management Schedule @nmaludy is the Release Manager and @blag Assisting for v3.3. They will freeze the master for the major repositories in StackStorm org, follow the StackStorm Release Process which is now available to public, accompanied by the Useful Info for Release managers. Communication is happening in #releasemgmt and #development Slack channels. The first step is pre-release manual user-acceptance testing for v3.3dev.

Why Manual testing?

StackStorm is very serious about testing and has a lot of it: Unit tests, Integration, Deployment/Integrity checks, Smoke tests and eventually end-2-end tests when automation spins up new AWS instance for each OS/flavor we support, installs real st2 like user would and runs set of st2tests (for each st2 PR, nightly, periodically, during release).

See st2ci and st2cd for more examples and workflows about how StackStorm automation is used to test StackStorm (dogfooding).

That's a perfect way to verify what we already know and codify expectations about how StackStorm should function.

However it's not enough. There are always new unknowns to discover, edge cases to experience and tests to add. Hence, manual Exploratory Testing is an exercise where entire team gathers together and starts trying (or breaking) new features before the new release. Because we're all different, perceive software differently and try different things we might find new bugs, improper design, oversights, edge cases and more.

This is how StackStorm previously managed to land less major/critical bugs into production.

TL;DR

Install StackStorm v3.3dev unstable packages, try random things in random environments (different OS) and report any regressions found comparing to v3.2:

curl -sSL https://stackstorm.com/packages/install.sh | bash -s -- --user=st2admin --password=Ch@ngeMe --unstable

Extra points for PR hotfixes and adding new or missing test cases.

Major changes

Full Changelog

Changes which are recommended to ack, explore, check and try in a random way.

st2

Added

Changed

Fixed

Removed

orquesta

Fixed

st2chatops

st2web

Conclusion

Please report findings here and bugs/regressions in respective repositories. Depending on severity and importance bugs might be fixed before the release or postponed to the next release if they're very minor and not a release blocker.

Issues Found During Release

PRs Merged for Release

TODOs

amanda11 commented 3 years ago

Raised https://github.com/StackStorm/st2-packages/pull/665 for failing to install previous stable version with one-line installer and -v (--version works fine though). I don't think it's a blocker though, but fix in PR.

nmaludy commented 3 years ago

missing changelog in st2chatops: https://github.com/StackStorm/st2chatops/issues/158 also st2web's changelog appears to be totally unused

amanda11 commented 3 years ago

@nmaludy Some of those st2 web PRs aren't PRs merging into master,but merging into a feature branch. e.g. https://github.com/StackStorm/st2web/pull/807 - they are the merges into the workflow composer feature branch. Not all of them but quite a few...

amanda11 commented 3 years ago

https://github.com/StackStorm/st2-packages/pull/666 raised for fact stable tries to intsall 3.2 with 3.3 scripts..

nmaludy commented 3 years ago

Need to ensure that people migrating form MongoDB 3.4 (previously supported version) follow the upgrade path:

3.4 -> 3.6 -> 4.0

https://docs.mongodb.com/manual/release-notes/3.6-upgrade-standalone/ https://docs.mongodb.com/manual/release-notes/4.0-upgrade-standalone/

submitted issue: https://github.com/StackStorm/st2docs/issues/1026

amanda11 commented 3 years ago

Testing so far on CentOS8 after a bash single line install good. No problems found with UI. Planning to do an upgrade from CentOS 7 3.2.0 bash single line install -> 3.3dev

nmaludy commented 3 years ago

I did a testing of CentOS 7 from 3.2.0 -> 3.3.0 and no issues there (puppet-st2 managed).

amanda11 commented 3 years ago

CentOS 8 ansible install all good.

amanda11 commented 3 years ago

CentOS 7 ansible looking good too.

amanda11 commented 3 years ago

Ubuntu 16.04 ansible install looked good. I found a problem on the web-ui with responding to inquiries and not selecting a field. But have also verified that it is a legacy problem and exists on my 3.2.0 install (https://github.com/StackStorm/st2web/issues/809)

amanda11 commented 3 years ago

Also verified the dig fix on CentOS 8. Reproduced failure on 3.2.0, and passed on 3.3.0dev (after installing bind-utils which wasn't installed as a dependency...). Not sure if we want to install that by default, but had to install manually.

nmaludy commented 3 years ago

Vagrant commands to test various installs (used on my RHEL box so ignore --provider=libvirt if you want to use VirtualBox):

BOX=centos/7 RELEASE=unstable VERSION=3.3dev vagrant up --provider=libvirt
BOX=centos/8 RELEASE=unstable VERSION=3.3dev vagrant up --provider=libvirt
BOX=generic/ubuntu1604 RELEASE=unstable VERSION=3.3dev vagrant up --provider=libvirt
BOX=generic/ubuntu1804 RELEASE=unstable VERSION=3.3dev vagrant up --provider=libvirt

Testing the boxes after they're up:

$ vagrant ssh
[vagrant@st2vagrant ~]$ sudo su -
[root@st2vagrant ~]# sudo ST2_AUTH_TOKEN=$(st2 auth st2admin -p 'Ch@ngeMe' -t) /opt/stackstorm/st2/bin/st2-self-check
amanda11 commented 3 years ago

@nmaludy Did you test on ubuntu 20.04? I thought we had problems with it's newer python 3? (supported ubuntus should be 1604 and 1804 I believe for 3.3).

nmaludy commented 3 years ago

@amanda11 no i haven't yet, sorry copy/paste fail, i'll change that to 1604

nmaludy commented 3 years ago

Self-check passed on all the various boxes!

SELF CHECK SUCCEEDED!
st2-self-check succeeded.

#############################################################
###################################################   #######
###############################################   /~\   #####
############################################   _- `~~~', ####
##########################################  _-~       )  ####
#######################################  _-~          |  ####
####################################  _-~            ;  #####
##########################  __---___-~              |   #####
#######################   _~   ,,                  ;  `,,  ##
#####################  _-~    ;'                  |  ,'  ; ##
###################  _~      '                    `~'   ; ###
############   __---;                                 ,' ####
########   __~~  ___                                ,' ######
#####  _-~~   -~~ _                               ,' ########
##### `-_         _                              ; ##########
#######  ~~----~~~   ;                          ; ###########
#########  /          ;                        ; ############
#######  /             ;                      ; #############
#####  /                `                    ; ##############
###  /                                      ; ###############
#                                            ################
amanda11 commented 3 years ago

Ansible install on Ubuntu 18.04 successful, and quick run-through on UI good.

amanda11 commented 3 years ago

Also did a quick chatops test with slack yesterday on one of the platforms (now forgot which O/S!)

arm4b commented 3 years ago

Did a few manual tests and picked up a couple platforms to try. Here is the report:

nmaludy commented 3 years ago

@armab st2docs PR for the comments above is started here: https://github.com/StackStorm/st2docs/pull/1028/

arms11 commented 3 years ago

Just deployed using st2-docker and able to verify few of the items already like no St2-Api-Key in the header getting logged! Will definitely consider socializing this to other developers as their templated dev environment before the custom pack is released for the K8s Deployment in the prod. Am planning to get the latest images for stand alone K8s envt to see how this goes. Production cluster has been running on 3.3dev for at least 2 months now and so far so good! Appreciate all the efforts!

nmaludy commented 3 years ago

@punkrokk found the issue https://github.com/StackStorm/st2/issues/5057 and i have implemented PRs to fix this:

https://github.com/StackStorm/st2/pull/5058 (into master) https://github.com/StackStorm/st2/pull/5059 (cherry pick for v3.3)

m4dcoder commented 3 years ago

I tested CentOS8. Looks good.

nmaludy commented 3 years ago

Found a bug in st2ci when trying to run the e2e upgrade test for el8. We need to pass -y in order to import the GPG key for the StackStorm/staging-stable repo. PR is here: https://github.com/StackStorm/st2ci/pull/191

nmaludy commented 3 years ago

Found an issue in st2-self-check where it was invoking actions and reporting back "OK" status, but in fact the action failed:

Specifically the action that failed was Attempting Test tests.test_timer_rule...OK! (44s)

The action failed because on the vagrant box for libvirt: generic/ubuntu1604 and generic/1804 the timezone is not set on the box causing the st2timersengine service to not start. Easy fix, simply set the timezone on the vagrant box and restart st2timersengine (this is not a problem on the virtualbox Ubuntu image).

However, the st2-self-check reported success (as seen above), when you check the WebUI the action in fact failed.

TODO: write up an issue for this, investigate and fix