search

StackStorm / community

Async conversation about ideas, planning, roadmap, issues, RFCs, etc around StackStorm
https://stackstorm.com/
Apache License 2.0
8 stars 3 forks source link

StackStorm v3.2dev pre-release Testing #6

Closed arm4b closed 2 years ago

arm4b commented 4 years ago

We're ready to prepare the StackStorm v3.2 release and starting pre-release testing.

This will be the first StackStorm release under the Linux Foundation with new Open Source Governance and Maintainers team. We now rely on our Community more than ever and invite everyone to help with the pre-release testing and development/contribution.

Release Process Preparation

Per Release Management Schedule @armab is the Release Manager and @punkrokk Assisting for v3.2. They will freeze the master for the major repositories in StackStorm org, follow the StackStorm Release Process which is now available to public, accompanied by the Useful Info for Release managers. Communication is happening in #releasemgmt and #development Slack channels. The first step is pre-release manual user-acceptance testing for v3.2dev.

Why Manual testing?

StackStorm is very serious about testing and has a lot of it: Unit tests, Integration, Deployment/Integrity checks, Smoke tests and eventually end-2-end tests when automation spins up new AWS instance for each OS/flavor we support, installs real st2 like user would and runs set of st2tests (for each st2 PR, nightly, periodically, during release).

See st2ci and st2cd for more examples and workflows about how StackStorm automation is used to test StackStorm (dogfooding).

That's a perfect way to verify what we already know and codify expectations about how StackStorm should function.

However it's not enough. There are always new unknowns to discover, edge cases to experience and tests to add. Hence, manual Exploratory Testing is an exercise where entire team gathers together and starts trying (or breaking) new features before the new release. Because we're all different, perceive software differently and try different things we might find new bugs, improper design, oversights, edge cases and more.

This is how StackStorm previously managed to land less major/critical bugs into production.

TL;DR

Install StackStorm v3.2dev staging packages, try random things in random environments (different OS) and report any regressions found comparing to v3.1:

curl -sSL https://stackstorm.com/packages/install.sh | bash -s -- --user=st2admin --password=Ch@ngeMe --unstable

Extra points for PR hotfixes and adding new or missing test cases.

Major changes

Important changes which are highly encouraged to try and test.

Full Changelog

Changes which are recommended to ack, explore, check and try in a random way.

st2

Added

Changed

Fixed

Removed

Full list of changes: https://github.com/StackStorm/st2/blob/master/CHANGELOG.rst

orquesta

Added

Changed

Fixed

Fore more info see https://github.com/StackStorm/orquesta/blob/master/CHANGELOG.rst

st2chatops

st2web

This could be huge. With almost 1 year since v3.1 st2 release codebase accumulated so many new features, changes and fixes.

Please report findings here and bugs/regressions in respective repositories. Depending on severity and importance bugs might be fixed before the release or postponed to the next release if they're very minor and not a release blocker.

arm4b commented 4 years ago

@StackStorm/maintainers Even if you didn't find any bugs, please report back in this discussion which OS/environment you've used and which features/items you've tried or verified. This will help to get an overall idea about OS/distro testing distribution. I may assign OS/flavors to different team members as we did before.

amanda11 commented 4 years ago

Very brief test done on centos8. Installed using single line install which went fine.

Simple chatops configured with slack for the aws ec2 get alias (note: stackstorm_aws 1.2.3 pack used, as 1.3.1 had issues - which I'll report separately - but not to do with stackstorm version). Simple orquesta workflow moved over successfully from stackstorm 3.1 install, which used the aws_boto3 and slack packs.

I did fall foul to the change where it installs the latest tagged pack rather than latest version of the code, on page: https://docs.stackstorm.com/latest/packs.html it states that it installs the latest commit of a pack rather than the tagged, but looking at the change log then that seems to be a known change. Docs state "By default, the latest commit to a pack repository will be installed, but you can specify a particular version, branch, tag, or even a commit hash. Just use =:" yet, it seems to install last tag. If this is by design as per "Install pack with the latest tag version if it exists when branch is not specialized. (improvement) #4743" - then does documentation need updating?

arm4b commented 4 years ago

@amanda11 Awesome, thank you very much for testing and feedback!

Yes indeed, in 3.2dev the behavior of pack install has changed which now installs latest "tag/version" instead of latest commit from master. That's a good catch! :+1:

@amanda11 Documentation indeed needs updating. How about submitting a small PR to fix that? https://github.com/StackStorm/st2docs/blob/master/docs/source/packs.rst

amanda11 commented 4 years ago

@armab I'll submit a PR for the doc change. thanks.

amanda11 commented 4 years ago

@armab I'll submit a PR for the doc change. thanks.

https://github.com/StackStorm/st2docs/pull/973 raised. I also verified the behaviour when you had a local repo that it was working, and what it picked up if you used file:/// towards a local git repo. That seems to still use the latest commit when using a file:/// to a local repo - so I left that section stating latest commit. So I literally just changed one word in the docs in the end.

arm4b commented 4 years ago

FYI for someone testing 3.1 -> 3.2 upgrade scenarios here is how to install 3.1dev from unstable packages first:

curl -sSL https://stackstorm.com/packages/install.sh | bash -s -- \
  --unstable --version=3.1dev \
  --user=st2admin --password=Ch@ngeMe

Then follow the packages upgrade procedure https://docs.stackstorm.com/latest/install/upgrades.html#general-upgrade-procedure to get v3.1dev->v3.2dev upgraded.

arm4b commented 4 years ago

Installation with the following instructions fails on EL6 and EL7:

curl -sSL https://stackstorm.com/packages/install.sh | bash -s --   --unstable --version=3.1dev   --user=st2admin --password=Ch@ngeMe

repoquery -y flag which was added before is non-existent for EL6. Update: -y is non-existent for EL7 as well.

~Reported as https://github.com/StackStorm/st2-packages/issues/641~ (Fixed now)

amanda11 commented 4 years ago

Further positive feedback on testing. Loving the immutable parameters on the aliases. Needed them for some aliases I needed for AWS reporting using the aws_boto3 pack. Integrated those successfully with st2chatops using 3.2dev on Centos8.

arm4b commented 4 years ago

Another 2 minor issues/enhancements were spot:

Both are not release blockers and will be planned for the next st2 v3.3.

amanda11 commented 4 years ago

Probably not an issue - but a problem I encountered. If I already have a config for a pack under /opt/stackstorm/configs, and then try and use st2 pack config then I get an error due to failing to have write permission to that file. It works if I delete the old config file, as it then creates a new file with the owner of st2. But I am guessing that if people have manually configured files they may not have created them with the owner as st2.

amanda11 commented 4 years ago

I had a play with the retry feature in the Orquesta workflow and I was surprised that I didn't have to up the version in the workflow yaml to state it was using version 1.1.0. I have the version as 1.0 and it works fine (documentation example uses version 1.0). So although new functionality it is still the same version of the workflow DSL?

amanda11 commented 4 years ago

I have a Orquesta workflow with: action A - with retry count of 2 (no when clause on the retry - so default to failed) action B - slack post message The when transition on A goes to B if A fails. I was expecting to only goto B once. But I'm seeing that B is being called for all 3 failures. Whereas I expected that it would only do the fail transition once.

So when the first attempt at action A fails, it does two things: 1) trigger action A again 2) goes to the action I have configured to run when A fails

Thus resulting in the failure message being output 3 times rather than as I would have expected it to just get the failure after all attempts had failed.

arm4b commented 4 years ago

@amanda11 For the orquesta with retry, that sounds unexpected to me too. From the description, could be somewhat related to https://github.com/StackStorm/orquesta/issues/138#issuecomment-615338189

Can you please create a new issue in https://github.com/StackStorm/orquesta/ with example workflow for @m4dcoder to reproduce easily, describing what's expected and what went wrong?

m4dcoder commented 4 years ago

I had a play with the retry feature in the Orquesta workflow and I was surprised that I didn't have to up the version in the workflow yaml to state it was using version 1.1.0. I have the version as 1.0 and it works fine (documentation example uses version 1.0). So although new functionality it is still the same version of the workflow DSL?

The version in WF DSL is a placeholder currently and the engine don't support multiple DSL versions. The new features are additive. In the future as we make major changes such as schema change or deprecation, we will definitely need to figure out how to separate support of different DSL versions.

m4dcoder commented 4 years ago

Here's the fix for the task retry reported above. PR @ https://github.com/StackStorm/orquesta/pull/200. Good catch @amanda11, keep them coming. @armab Once PR is merged, I'm thinking of republishing the v1.1.0 release for orquesta. What do you think?

winem commented 4 years ago

Hi,

here is the summary of my tests from tonight:

I hope this helps. Will continue testing in the next days. Let me know if you want anything special tested. Otherwise I will just keep going through the release notes and test the mentioned changes.

amanda11 commented 4 years ago

Whilst testing re-run from task functionality I raised: https://github.com/StackStorm/st2docs/issues/975 on the documentation. Have submitted PR to just remove those that are included, but may want to add new features to the list if any are known.

arm4b commented 4 years ago

@m4dcoder For the Orquesta hotfix I think it's best to publish new v1.1.1 patch to include in st2. Also replied in https://github.com/StackStorm/orquesta/pull/200#discussion_r414570864

arm4b commented 4 years ago

Current 3.2dev pre-release testing status:

amanda11 commented 4 years ago

Documentation issue raised on upgrade docs when running on centos 8 https://github.com/StackStorm/st2docs/issues/977. Proposed st2docs PR to split rhel8/centos 8 so that skip the mistral packages: https://github.com/StackStorm/st2docs/pull/978

arm4b commented 4 years ago

Next stage of release automation process & testing, following the https://github.com/StackStorm/discussions/wiki/Release-Process#process-for-major-release (now open) is that we got staging-stable packages out in https://packagecloud.io/stackstorm/staging-stable repository.

At this stage we can just quickly check the packages like integrity/smoke tests to make sure this version is not different to what we tested last week and has no surprises:

curl -sSL https://stackstorm.com/packages/install.sh | bash -s -- --user=st2admin --password=st2admin --staging

These packages are now candidate for the final stable promotion. If no major issues found, we'll continue the release automation tomorrow, 28 Apr, US morning.

FYI for someone wondering the pipeline is the following: staging-unstable -> unstable -> staging-stable -> stable. See https://github.com/StackStorm/discussions/wiki/Useful-info-for-Release-Managers#packagecloud-repositories for more info about each repository.

winem commented 4 years ago

Probably a bit late to the party but I just ran all the tests from my previous comment on the staging packages again - successfully.

arm4b commented 4 years ago

Thanks everyone! Especially @amanda11 and @winem for extra effort catching and fixing bugs, - that was very helpful.

We now have StackStorm v3.2.0 stable out: https://stackstorm.com/2020/04/30/stackstorm-v3-2-0-released/