search

StackStorm / community

Async conversation about ideas, planning, roadmap, issues, RFCs, etc around StackStorm
https://stackstorm.com/
Apache License 2.0
8 stars 3 forks source link

v3.7.0 release tracker #101

Closed cognifloyd closed 2 years ago

cognifloyd commented 2 years ago

Pre-Release PR Reviews

@cognifloyd reviewed all PRs across all st2 repos involved in the release and merged everything I could. I moved others from the 3.7 project+milestone to 3.8.

st2chatops release prep

We needed to get https://github.com/StackStorm/hubot-stackstorm/pull/214 included in st2chatops which involved release hubot-stackstorm.

@cognifloyd released hubot-stackstorm v0.12.0:

Added that to st2chatops in https://github.com/StackStorm/st2chatops/pull/175

@cognifloyd: I have never made a release on NPM. It would have been simpler to just log in as storminstanley on the command line, and then get the OTP from a senior maintainer. But I didn't know that so, I logged in first on the website. Then, no senior maintainers were available to get me another OTP to login on CLI which is required to publish a release. That's why I created my own account and added myself to hubot-stackstorm so I could make the release.

aside: the st2chatops mess will become much better with the transition from JS based hubot to Python based OpsDroid. I'll be working more on that later this year.

:link: Testing Before Release

volunteers welcome - @cognifloyd will not volun-tell anyone to do this as I cannot know others' time constraints.

steps 4 and 5: create and e2e test the staging-stable packages

These tests highlighted a couple of issues:

  1. We must turn on our EC2 VMs before running workflows: ubuntu-build-itest001, ubuntu-build002 and ubuntu-build003
  2. The upgrade workflow does not automatically handle python upgrades.
    • The RockyLinux 8 tests failed because we didn't run sudo st2ctl reload --register-recreate-virtualenvs.
    • All the tests passed after running that on the test VM.
  3. The upgrade workflow does not cleanup VMs on failure. So, I had to trigger st2cd.destroy_vm for a lot of VMs.
  4. It did not cause issues, but I noticed that st2cd.setup_e2e_tests should be updated to use python3.8 by default on EL8.
  5. The upgrade notes need to be updated to show --register-recreate-virtualenvs instead of --register-setup-recreate-virtualenvs.

:link: step 6 staging-stable testing

BOX=ubuntu/bionic64 REPO_TYPE=staging RELEASE=stable vagrant up
BOX=ubuntu/focal64 REPO_TYPE=staging RELEASE=stable vagrant up
BOX=centos/7 REPO_TYPE=staging RELEASE=stable vagrant up
BOX=rockylinux/8 REPO_TYPE=staging RELEASE=stable vagrant up

volunteers welcome - @cognifloyd will not volun-tell anyone to do this as I cannot know others' time constraints.

:link: step 8 stable testing

BOX=ubuntu/bionic64 RELEASE=stable vagrant up
BOX=ubuntu/focal64 RELEASE=stable vagrant up
BOX=centos/7 RELEASE=stable vagrant up
BOX=rockylinux/8 RELEASE=stable vagrant up

volunteers welcome - @cognifloyd will not volun-tell anyone to do this as I cannot know others' time constraints.

:link: step 9 tag releases

Also merged these changelog PRs:

:link: step 10 update st2cd and st2ci packs

step 12 create the release

step 13 make sure new dev builds succeed

step 14 Check PackageCloud for new packages in unstable channel

remaining steps

blog article

Close v3.7.0

cognifloyd commented 2 years ago

Notes from Testing Before Release:

bionic

bash installer in vagrant box:

st2-self-check:

st2web manual testing:

focal

bash installer in vagrant box:

st2-self-check:

st2web manual testing:

rocky 8

st2-self-check:

st2web manual testing:

nzlosh commented 2 years ago

CentOS7

st2-self-check failed on the enquiry chain test. The response schema expects a string, but an integer is given.

Attempting Test tests.test_inquiry_chain...ERROR! (38s)
Test output: ..................
id: 62705b6b7483bb205ae6b875
action.ref: tests.test_inquiry_chain
parameters: 
  protocol: http
  token: 0521454b412b4c829e2ce14c5272c3ef
status: failed
result_task: valid_response
result: 
  failed: true
  return_code: 1
  stderr: ''
  stdout: 'ERROR: 500 Server Error: Internal Server Error
    MESSAGE: Unable to resume liveaction "6270558d7483bb205ae6b7a6" because it is in "pausing" state and not in "paused" state. for url: http://12
7.0.0.1:9101/inquiries/6270558e93aa732b28abc467
    '
  succeeded: false

  cmd: 'sleep 3 ; st2 inquiry respond -r ''{"secondfactor": 123}'' 6270558e93aa732b28abc467'
  env:
    ST2_API_URL: http://127.0.0.1:9101
    ST2_AUTH_TOKEN: 0521454b412b4c829e2ce14c5272c3ef
    ST2_AUTH_URL: http://127.0.0.1:9100
    ST2_BASE_URL: http://127.0.0.1
status: failed (4s elapsed)
start_timestamp: Mon, 02 May 2022 22:30:19 UTC
end_timestamp: Mon, 02 May 2022 22:30:23 UTC
log: 
  - status: requested
    timestamp: '2022-05-02T22:30:19.253000Z'
  - status: scheduled
    timestamp: '2022-05-02T22:30:19.517000Z'
  - status: running
    timestamp: '2022-05-02T22:30:19.735000Z'
  - status: failed
    timestamp: '2022-05-02T22:30:24.015000Z'
result: 
  failed: true
  return_code: 1
  stderr: ''
  stdout: "ERROR: 400 Client Error: Bad Request
MESSAGE: Response for inquiry "6270558e93aa732b28abc467" did not pass schema validation. 123 is not of type 'string'

Failed validating 'type' in schema['properties']['secondfactor']:
    {'description': 'Please enter second factor for authenticating to '
                    '"foo" service',
     'required': True,
     'type': 'string'}

Update: This issue appeared when using the vagrant repo install script https://raw.githubusercontent.com/StackStorm/st2vagrant/master/scripts/install_st2.sh. When using the officially documented install script https://stackstorm.com/packages/install.sh this issue is not encountered and the st2-self-check passes.

amanda11 commented 2 years ago

@cognifloyd Re web testing:

cognifloyd commented 2 years ago

typing anything in magnifying glass / search field does not do anything (no filtering of results). Amanda: Is it the problem reported here: https://github.com/StackStorm/st2web/issues/925? (follow comments and I found the pull-down filters were ok, but not the edit search bit).

Yes, that's it. And I get the same behavior in installs of 3.4.1 and 3.6.0 so this is an old bug.

these two are interrelated:

Why can't I type a new pack name in the packs drop down of Flow? Amanda: Do you mean when you create a new action in workflow composer? And is this so you can also create a new pack? Or just you can't type in to filter list.

and

when creating a new action if I select something other than default, then I default is removed from the list (prob because nothing is in the default pack). Amanda: I can confirmed that if you did have actions in defaults, then you can still go back and choose it. Verified this is same behaviour as in ST2 3.6. Also you can select default, by the oddity of selecting the blank entry - if you choose the blank entry in the pull-down, it then populates the field with "default".

When I create or edit a workflow, changing the pack doesn't allow me to type in the pack field. If there are no actions/rules etc in the default pack, then you can't select the default pack.

When I create rules in the UI (vs saving in a git repo) I like to use __webui__ as the pack name (or if it's an adhoc thing not created in the webui, I would use some other dunder name) -- that's my personal convention to remember that this resource is only in the database, not preserved in git. The workflow composer would create a file, but that file would still not be committed in git or pushed to the git server, so I would still like to use something like __adhoc__ when creating new actions in the UI.

Then again, I don't really use Flow much, and this is probably a new behavioral request, so it's not a blocker.

for actions and rules env arg does not accept key1=val1,key2=val2 format anymore. It has to be a full JSON object, {"key1":"val2","key2":"val2"}. Amanda: Interesting. Perhaps we should add that to release notes, or to the upgrade notes in st2docs?

Amanda: verified this is same behaviour in ST2 3.6. So not new. Though we should raise an issue to at least change the help for this as the help says to add key=val1

The help text is still valid for CLI usage. We should probably add support for converting this key=val syntax to json in the webui. Since it's not a regression in this release, it is definitely not a blocker.

cognifloyd commented 2 years ago

Dangit. We have a blocker for RockyLinux 8

$ /opt/stackstorm/st2/bin/python --version
Python 3.8.8
$ /opt/stackstorm/virtualenvs/st2/bin/python --version
Python 3.6.8

It's building virtualenvs with the wrong python. That means that the dependencies in /opt/stackstorm/st2 cannot be inherited by /opt/stackstorm/virtualenvs/* which cause a ModeuleNotFoundError: No module named 'orjson'

I'm going to try to duplicate this in the st2vagrant box.

edit: added results

Nope. st2vagrant had the correct version of python in all the virtualenvs.

This might have something to do with the upgrade. Once I rebuilt the virtualenv on st2-pkg-upgrade-staging-stable-el8, the action in question succeeded. So maybe the upgrade workflow does not include updating from python3.6 to python3.8...

cognifloyd commented 2 years ago

Basically, this workflow does not follow the migration guide for RockyLinux8: https://github.com/StackStorm/st2ci/blob/master/actions/workflows/st2_pkg_upgrade_e2e_test.yaml https://docs.stackstorm.com/3.7/install/upgrades.html#version-specific-changes-migration-scripts

It does not run:

sudo st2ctl reload --register-recreate-virtualenvs

Hmm. And the migration guide is wrong, it shows: --register-setup-recreate-virtualenvs instead of --register-recreate-virtualenvs.

So, I manually ran the rebuild command and re-triggered the e2e tests. Everything passed. So, it's not a blocker; we just need a minor docs update.

Also, we need to update https://github.com/StackStorm/st2cd/blob/master/actions/setup_e2e_tests.sh to use python3.8 on EL8

arm4b commented 2 years ago

That's a good catch before shipping it to prod! :+1:

amanda11 commented 2 years ago

Basically, this workflow does not follow the migration guide for RockyLinux8: https://github.com/StackStorm/st2ci/blob/master/actions/workflows/st2_pkg_upgrade_e2e_test.yaml https://docs.stackstorm.com/3.7/install/upgrades.html#version-specific-changes-migration-scripts

It does not run:

sudo st2ctl reload --register-recreate-virtualenvs

Hmm. And the migration guide is wrong, it shows: --register-setup-recreate-virtualenvs instead of --register-recreate-virtualenvs.

So, I manually ran the rebuild command and re-triggered the e2e tests. Everything passed. So, it's not a blocker; we just need a minor docs update.

Also, we need to update https://github.com/StackStorm/st2cd/blob/master/actions/setup_e2e_tests.sh to use python3.8 on EL8

Possibly should have had the st2ctl command in: "st2ci/actions/st2_pkg_upgrade_deps_el8.sh"

Those files are to do version specific things on upgrade. Though I guess we'll never run it again to do an upgrade...

cognifloyd commented 2 years ago

OK. Release announcement blog has been published: https://stackstorm.com/2022/05/10/stackstorm-3-7-0-released/

The only thing we didn't do was fix the packer-st2 build (see https://github.com/StackStorm/community/issues/101#tagged-releases). I'm closing this as complete.