StackStorm / community

Async conversation about ideas, planning, roadmap, issues, RFCs, etc around StackStorm
https://stackstorm.com/
Apache License 2.0
8 stars 3 forks source link

StackStorm v3.4.0 Pre-release Testing #66

Closed blag closed 3 years ago

blag commented 3 years ago

We're ready to prepare the StackStorm v3.4 release and start pre-release testing..

Release Process Preparation

Per Release Management Schedule @blag is the Release Manager and @amanda11 assisting for v3.4. They will freeze the master for the major repositories in StackStorm org, follow the StackStorm Release Process which is now available to public, accompanied by the Useful Info for Release managers. Communication is happening in #releasemgmt and #development Slack channels. The first step is pre-release manual user-acceptance testing for v3.4dev.

Why Manual testing?

StackStorm is very serious about testing and has a lot of it: Unit tests, Integration, Deployment/Integrity checks, Smoke tests and eventually end-2-end tests when automation spins up new AWS instance for each OS/flavor we support, installs real st2 like user would and runs set of st2tests (for each st2 PR, nightly, periodically, during release).

See st2ci and st2cd for more examples and workflows about how StackStorm automation is used to test StackStorm (dogfooding).

That's a perfect way to verify what we already know and codify expectations about how StackStorm should function.

However it's not enough. There are always new unknowns to discover, edge cases to experience and tests to add. Hence, manual Exploratory Testing is an exercise where entire team gathers together and starts trying (or breaking) new features before the new release. Because we're all different, perceive software differently and try different things we might find new bugs, improper design, oversights, edge cases and more.

This is how StackStorm previously managed to land less major/critical bugs into production.

TL;DR

Install StackStorm v3.4dev unstable packages, try random things in random environments (different OS) and report any regressions found comparing to v3.3 (and/or v3.2 if you are still running ST2 Enterprise):

curl -sSL https://stackstorm.com/packages/install.sh | bash -s -- --user=st2admin --password=Ch@ngeMe --unstable

You can also run StackStorm on a specific OS with st2vagrant:

git clone https://github.com/StackStorm/st2vagrant.git
cd st2vagrant
VERSION=3.4dev vagrant up  # Default OS is Ubuntu Xenial (16.04)

BOX=ubuntu/bionic64 VERSION=3.4dev vagrant up  # Ubuntu Bionic 18.04

BOX=centos/7 VERSION=3.4dev vagrant up  # CentOS 7

BOX=centos/8 VERSION=3.4dev vagrant up  # CentOS 8

Extra points for PR hotfixes and adding new or missing test cases.

Specific things to check

If you have successful test results, please post a summary of what all you tested (OSes, what features you tested).

If you run into any bugs, please open them in the respective repositories and link to this issue from there. I will add them to the list at the bottom of this description.

If you have any issues running StackStorm or running the tests, please post down below.

Major changes

Full Changelog

Changes which are recommended to ack, explore, check and try in a random way.

st2

Added

Changed

Fixed

Removed

orquesta 1.3.0

Added

Changed

Fixed

st2chatops

No changes

StackStorm Exchange

Conclusion

Please report findings here and bugs/regressions in respective repositories. Depending on severity and importance bugs might be fixed before the release or postponed to the next release if they're very minor and not a release blocker.

Issues Found During Release

PRs Merged for Release

TODOs

himadriganguly commented 3 years ago

st2ctl status showing

##### st2 components status #####
st2actionrunner PID: 18733
st2actionrunner PID: 18737
st2actionrunner PID: 18740
st2actionrunner PID: 18743
st2actionrunner PID: 18746
st2actionrunner PID: 18749
st2actionrunner PID: 18753
st2actionrunner PID: 18756
st2actionrunner PID: 18762
st2actionrunner PID: 18765
st2api PID: 18635
st2api PID: 18647
st2stream PID: 18583
st2stream PID: 18628
st2auth PID: 18523
st2auth PID: 18535
st2garbagecollector PID: 17552
st2notifier PID: 17555
st2rulesengine PID: 17558
st2sensorcontainer PID: 18653
st2chatops is not running.
st2timersengine PID: 17569
st2workflowengine PID: 18665
st2scheduler PID: 17575

in the result above it is showing st2chatops not running.

sudo service st2chatops status

● st2chatops.service - StackStorm service st2chatops
   Loaded: loaded (/lib/systemd/system/st2chatops.service; enabled; vendor preset: enabled)
   Active: activating (auto-restart) (Result: exit-code) since Sat 2021-02-20 16:04:35 UTC; 3s ago
  Process: 22638 ExecStart=/opt/stackstorm/chatops/bin/hubot $DAEMON_ARGS (code=exited, status=1/FAILURE)
 Main PID: 22638 (code=exited, status=1/FAILURE)
Feb 20 16:04:35 himadri systemd[1]: st2chatops.service: Main process exited, code=exited, status=1/FAILURE
Feb 20 16:04:35 himadri systemd[1]: st2chatops.service: Unit entered failed state.
Feb 20 16:04:35 himadri systemd[1]: st2chatops.service: Failed with result 'exit-code'.

Even after the machine restart, the same thing happens.

blag commented 3 years ago

@himadriganguly Thank you for testing v3.4a!

That's normal when you haven't configured st2chatops by modifying /opt/stackstorm/chatops/st2chatops.env. The default StackStorm installation will not configure ChatOps for you since that is an optional component that requires integration with an external service.

We should try to improve that situation in the future, but for v3.4 I'm not considering that a bug.

If you have configured st2chatops, then that's definitely a bug. In which case, please post your st2chatops.env file (but remember to redact any API keys or Slack tokens) so we can figure out if it's a bug in your configuration or in st2chatops.

himadriganguly commented 3 years ago

Running Self-Verification Result

Click to expand! ``` $ sudo ST2_AUTH_TOKEN=$(st2 auth st2admin -p 'Ch@ngeMe' -t) /opt/stackstorm/st2/bin/st2-self-check ~ ~ Cloning st2tests (branch=master) in /home/vagrant. Cloning into 'st2tests'... remote: Enumerating objects: 323, done. remote: Counting objects: 100% (323/323), done. remote: Compressing objects: 100% (184/184), done. remote: Total 323 (delta 145), reused 251 (delta 121), pack-reused 0 Receiving objects: 100% (323/323), 703.27 KiB | 396.00 KiB/s, done. Resolving deltas: 100% (145/145), done. Checking connectivity... done. Copying asserts, fixtures, tests and examples packs. Installing asserts, fixtures, tests and examples packs. .................. id: 60313654f6df70916a47edb3 action.ref: packs.setup_virtualenv context.user: st2admin parameters: packs: - examples - tests - asserts - fixtures - webui status: succeeded start_timestamp: Sat, 20 Feb 2021 16:18:28 UTC end_timestamp: Sat, 20 Feb 2021 16:19:05 UTC result: exit_code: 0 result: 'Successfully set up virtualenv for the following packs: examples, tests, asserts, fixtures, webui' stderr: 'st2.actions.python.SetupVirtualEnvironmentAction: DEBUG Setting up virtualenv for pack "examples" (/opt/stackstorm/packs/examples) st2.actions.python.SetupVirtualEnvironmentAction: INFO Virtualenv path "/opt/stackstorm/virtualenvs/examples" doesn''t exist st2.actions.python.SetupVirtualEnvironmentAction: DEBUG Creating virtualenv for pack "examples" in "/opt/stackstorm/virtualenvs/examples" st2.actions.python.SetupVirtualEnvironmentAction: DEBUG Creating virtualenv in "/opt/stackstorm/virtualenvs/examples" using Python binary "/opt/stackstorm/st2/bin/python" st2.actions.python.SetupVirtualEnvironmentAction: DEBUG Running command "/opt/stackstorm/st2/bin/virtualenv -p /opt/stackstorm/st2/bin/python --always-copy --verbose --no-download /opt/stackstorm/virtualenvs/examples" to create virtualenv. st2.actions.python.SetupVirtualEnvironmentAction: DEBUG Installing base requirements st2.actions.python.SetupVirtualEnvironmentAction: DEBUG Installing requirement six>=1.9.0,<2.0 with command /opt/stackstorm/virtualenvs/examples/bin/pip install six>=1.9.0,<2.0. st2.actions.python.SetupVirtualEnvironmentAction: DEBUG Installing pack specific requirements from "/opt/stackstorm/packs/examples/requirements.txt" st2.actions.python.SetupVirtualEnvironmentAction: DEBUG Installing requirements from file /opt/stackstorm/packs/examples/requirements.txt with command /opt/stackstorm/virtualenvs/examples/bin/pip install -U -r /opt/stackstorm/packs/examples/requirements.txt. st2.actions.python.SetupVirtualEnvironmentAction: DEBUG Virtualenv for pack "examples" successfully created in "/opt/stackstorm/virtualenvs/examples" st2.actions.python.SetupVirtualEnvironmentAction: DEBUG Setting up virtualenv for pack "tests" (/opt/stackstorm/packs/tests) st2.actions.python.SetupVirtualEnvironmentAction: INFO Virtualenv path "/opt/stackstorm/virtualenvs/tests" doesn''t exist st2.actions.python.SetupVirtualEnvironmentAction: DEBUG Creating virtualenv for pack "tests" in "/opt/stackstorm/virtualenvs/tests" st2.actions.python.SetupVirtualEnvironmentAction: DEBUG Creating virtualenv in "/opt/stackstorm/virtualenvs/tests" using Python binary "/opt/stackstorm/st2/bin/python" st2.actions.python.SetupVirtualEnvironmentAction: DEBUG Running command "/opt/stackstorm/st2/bin/virtualenv -p /opt/stackstorm/st2/bin/python --always-copy --verbose --verbose --no-download /opt/stackstorm/virtualenvs/tests" to create virtualenv. st2.actions.python.SetupVirtualEnvironmentAction: DEBUG Installing base requirements st2.actions.python.SetupVirtualEnvironmentAction: DEBUG Installing requirement six>=1.9.0,<2.0 with command /opt/stackstorm/virtualenvs/tests/bin/pip install six>=1.9.0,<2.0. st2.actions.python.SetupVirtualEnvironmentAction: DEBUG No pack specific requirements found st2.actions.python.SetupVirtualEnvironmentAction: DEBUG Virtualenv for pack "tests" successfully created in "/opt/stackstorm/virtualenvs/tests" st2.actions.python.SetupVirtualEnvironmentAction: DEBUG Setting up virtualenv for pack "asserts" (/opt/stackstorm/packs/asserts) st2.actions.python.SetupVirtualEnvironmentAction: INFO Virtualenv path "/opt/stackstorm/virtualenvs/asserts" doesn''t exist st2.actions.python.SetupVirtualEnvironmentAction: DEBUG Creating virtualenv for pack "asserts" in "/opt/stackstorm/virtualenvs/asserts" st2.actions.python.SetupVirtualEnvironmentAction: DEBUG Creating virtualenv in "/opt/stackstorm/virtualenvs/asserts" using Python binary "/opt/stackstorm/st2/bin/python" st2.actions.python.SetupVirtualEnvironmentAction: DEBUG Running command "/opt/stackstorm/st2/bin/virtualenv -p /opt/stackstorm/st2/bin/python --always-copy --verbose --verbose --verbose --no-download /opt/stackstorm/virtualenvs/asserts" to create virtualenv. st2.actions.python.SetupVirtualEnvironmentAction: DEBUG Installing base requirements st2.actions.python.SetupVirtualEnvironmentAction: DEBUG Installing requirement six>=1.9.0,<2.0 with command /opt/stackstorm/virtualenvs/asserts/bin/pip install six>=1.9.0,<2.0. st2.actions.python.SetupVirtualEnvironmentAction: DEBUG Installing pack specific requirements from "/opt/stackstorm/packs/asserts/requirements.txt" st2.actions.python.SetupVirtualEnvironmentAction: DEBUG Installing requirements from file /opt/stackstorm/packs/asserts/requirements.txt with command /opt/stackstorm/virtualenvs/asserts/bin/pip install -U -r /opt/stackstorm/packs/asserts/requirements.txt. st2.actions.python.SetupVirtualEnvironmentAction: DEBUG Virtualenv for pack "asserts" successfully created in "/opt/stackstorm/virtualenvs/asserts" st2.actions.python.SetupVirtualEnvironmentAction: DEBUG Setting up virtualenv for pack "fixtures" (/opt/stackstorm/packs/fixtures) st2.actions.python.SetupVirtualEnvironmentAction: INFO Virtualenv path "/opt/stackstorm/virtualenvs/fixtures" doesn''t exist st2.actions.python.SetupVirtualEnvironmentAction: DEBUG Creating virtualenv for pack "fixtures" in "/opt/stackstorm/virtualenvs/fixtures" st2.actions.python.SetupVirtualEnvironmentAction: DEBUG Creating virtualenv in "/opt/stackstorm/virtualenvs/fixtures" using Python binary "/opt/stackstorm/st2/bin/python" st2.actions.python.SetupVirtualEnvironmentAction: DEBUG Running command "/opt/stackstorm/st2/bin/virtualenv -p /opt/stackstorm/st2/bin/python --always-copy --verbose --verbose --verbose --verbose --no-download /opt/stackstorm/virtualenvs/fixtures" to create virtualenv. st2.actions.python.SetupVirtualEnvironmentAction: DEBUG Installing base requirements st2.actions.python.SetupVirtualEnvironmentAction: DEBUG Installing requirement six>=1.9.0,<2.0 with command /opt/stackstorm/virtualenvs/fixtures/bin/pip install six>=1.9.0,<2.0. st2.actions.python.SetupVirtualEnvironmentAction: DEBUG Installing pack specific requirements from "/opt/stackstorm/packs/fixtures/requirements.txt" st2.actions.python.SetupVirtualEnvironmentAction: DEBUG Installing requirements from file /opt/stackstorm/packs/fixtures/requirements.txt with command /opt/stackstorm/virtualenvs/fixtures/bin/pip install -U -r /opt/stackstorm/packs/fixtures/requirements.txt. st2.actions.python.SetupVirtualEnvironmentAction: DEBUG Virtualenv for pack "fixtures" successfully created in "/opt/stackstorm/virtualenvs/fixtures" st2.actions.python.SetupVirtualEnvironmentAction: DEBUG Setting up virtualenv for pack "webui" (/opt/stackstorm/packs/webui) st2.actions.python.SetupVirtualEnvironmentAction: INFO Virtualenv path "/opt/stackstorm/virtualenvs/webui" doesn''t exist st2.actions.python.SetupVirtualEnvironmentAction: DEBUG Creating virtualenv for pack "webui" in "/opt/stackstorm/virtualenvs/webui" st2.actions.python.SetupVirtualEnvironmentAction: DEBUG Creating virtualenv in "/opt/stackstorm/virtualenvs/webui" using Python binary "/opt/stackstorm/st2/bin/python" st2.actions.python.SetupVirtualEnvironmentAction: DEBUG Running command "/opt/stackstorm/st2/bin/virtualenv -p /opt/stackstorm/st2/bin/python --always-copy --verbose --verbose --verbose --verbose --verbose --no-download /opt/stackstorm/virtualenvs/webui" to create virtualenv. st2.actions.python.SetupVirtualEnvironmentAction: DEBUG Installing base requirements st2.actions.python.SetupVirtualEnvironmentAction: DEBUG Installing requirement six>=1.9.0,<2.0 with command /opt/stackstorm/virtualenvs/webui/bin/pip install six>=1.9.0,<2.0. st2.actions.python.SetupVirtualEnvironmentAction: DEBUG Installing pack specific requirements from "/opt/stackstorm/packs/webui/requirements.txt" st2.actions.python.SetupVirtualEnvironmentAction: DEBUG Installing requirements from file /opt/stackstorm/packs/webui/requirements.txt with command /opt/stackstorm/virtualenvs/webui/bin/pip install -U -r /opt/stackstorm/packs/webui/requirements.txt. st2.actions.python.SetupVirtualEnvironmentAction: DEBUG Virtualenv for pack "webui" successfully created in "/opt/stackstorm/virtualenvs/webui" ' stdout: '' Registering content...[flags = --config-file /etc/st2/st2.conf --register-all] 2021-02-20 16:19:06,282 INFO [-] Connecting to database "st2" @ "127.0.0.1:27017" as user "stackstorm". 2021-02-20 16:19:06,289 INFO [-] Successfully connected to database "st2" @ "127.0.0.1:27017" as user "stackstorm". 2021-02-20 16:19:06,737 INFO [-] ========================================================= 2021-02-20 16:19:06,738 INFO [-] ############## Registering triggers ##################### 2021-02-20 16:19:06,738 INFO [-] ========================================================= 2021-02-20 16:19:06,984 INFO [-] Registered 1 triggers. 2021-02-20 16:19:06,984 INFO [-] ========================================================= 2021-02-20 16:19:06,984 INFO [-] ############## Registering sensors ###################### 2021-02-20 16:19:06,984 INFO [-] ========================================================= 2021-02-20 16:19:07,182 INFO [-] Registered 7 sensors. 2021-02-20 16:19:07,182 INFO [-] ========================================================= 2021-02-20 16:19:07,183 INFO [-] ############## Registering runners ###################### 2021-02-20 16:19:07,183 INFO [-] ========================================================= 2021-02-20 16:19:07,977 INFO [-] Registered 14 runners. 2021-02-20 16:19:07,977 INFO [-] ========================================================= 2021-02-20 16:19:07,977 INFO [-] ############## Registering actions ###################### 2021-02-20 16:19:07,978 INFO [-] ========================================================= 2021-02-20 16:19:15,491 INFO [-] Registered 230 actions. 2021-02-20 16:19:15,491 INFO [-] ========================================================= 2021-02-20 16:19:15,491 INFO [-] ############## Registering rules ######################## 2021-02-20 16:19:15,492 INFO [-] ========================================================= 2021-02-20 16:19:15,767 INFO [-] Registered 9 rules. 2021-02-20 16:19:15,767 INFO [-] ========================================================= 2021-02-20 16:19:15,767 INFO [-] ############## Registering aliases ###################### 2021-02-20 16:19:15,767 INFO [-] ========================================================= 2021-02-20 16:19:17,329 INFO [-] Registered 134 aliases. 2021-02-20 16:19:17,329 INFO [-] ========================================================= 2021-02-20 16:19:17,329 INFO [-] ############## Registering policy types ################# 2021-02-20 16:19:17,329 INFO [-] ========================================================= 2021-02-20 16:19:17,364 INFO [-] Registered 3 policy types. 2021-02-20 16:19:17,364 INFO [-] ========================================================= 2021-02-20 16:19:17,364 INFO [-] ############## Registering policies ##################### 2021-02-20 16:19:17,364 INFO [-] ========================================================= 2021-02-20 16:19:17,365 INFO [-] Registered 0 policies. 2021-02-20 16:19:17,365 INFO [-] ========================================================= 2021-02-20 16:19:17,365 INFO [-] ############## Registering configs ###################### 2021-02-20 16:19:17,366 INFO [-] ========================================================= 2021-02-20 16:19:17,368 INFO [-] Registered 0 configs. ##### st2 components status ##### st2actionrunner PID: 1330 st2actionrunner PID: 1356 st2actionrunner PID: 1366 st2actionrunner PID: 1392 st2actionrunner PID: 1415 st2actionrunner PID: 1421 st2actionrunner PID: 1427 st2actionrunner PID: 1470 st2actionrunner PID: 1497 st2actionrunner PID: 1539 st2api PID: 1166 st2api PID: 1772 st2stream PID: 1226 st2stream PID: 1764 st2auth PID: 1234 st2auth PID: 1733 st2garbagecollector PID: 1169 st2notifier PID: 1205 st2rulesengine PID: 1220 st2sensorcontainer PID: 1179 st2chatops is not running. st2timersengine PID: 1155 st2workflowengine PID: 1210 st2scheduler PID: 1185 Deleting 'st2tests' directory from /home/vagrant. ~ Attempting Test tests.test_inquiry_chain...OK! (55s) Attempting Test tests.test_key_triggers...OK! (44s) Attempting Test tests.test_pack_install_tool...OK! (17s) Attempting Test tests.test_packs_pack...OK! (130s) Attempting Test tests.test_quickstart...OK! (27s) Attempting Test tests.test_quickstart_key...OK! (7s) Attempting Test tests.test_quickstart_local_script_actions...OK! (13s) Attempting Test tests.test_quickstart_passive_sensor...OK! (7s) Attempting Test tests.test_quickstart_polling_sensor...OK! (5s) Attempting Test tests.test_quickstart_python_actions...OK! (13s) Attempting Test tests.test_quickstart_remote_script_actions...OK! (17s) Attempting Test tests.test_quickstart_rules...OK! (64s) Attempting Test tests.test_quickstart_trace...OK! (21s) Attempting Test tests.test_render_config_context...OK! (5s) Attempting Test tests.test_run_pack_tests_tool...OK! (82s) Attempting Test tests.test_timer_rule...OK! (41s) Skipping tests.test_windows_runners... Skipping tests.test_winrm_large_script... Skipping tests.test_winrm_runners... Attempting Example examples.orquesta-examples...OK! (9s) SELF CHECK SUCCEEDED! st2-self-check succeeded. ############################################################# ################################################### ####### ############################################### /~\ ##### ############################################ _- `~~~', #### ########################################## _-~ ) #### ####################################### _-~ | #### #################################### _-~ ; ##### ########################## __---___-~ | ##### ####################### _~ ,, ; `,, ## ##################### _-~ ;' | ,' ; ## ################### _~ ' `~' ; ### ############ __---; ,' #### ######## __~~ ___ ,' ###### ##### _-~~ -~~ _ ,' ######## ##### `-_ _ ; ########## ####### ~~----~~~ ; ; ########### ######### / ; ; ############ ####### / ; ; ############# ##### / ` ; ############## ### / ; ############### # ################ ```
himadriganguly commented 3 years ago

@blag Thank you for the explanation. I will be testing other things according to the checklist and will post the details. I will try to contribute as much as I can.

winem commented 3 years ago

Just a short update since I ran into an issue with the load_diag workflow in the Linux pack and I'll look into that now:

Update: The issue with the load_diag workflow isn't a regression or issue specific to st2 3.4 and there are already open issues that cover these issues. So I'll proceed.

Kami commented 3 years ago

I run tests on Ubuntu Xenial (16.04) here are the issues I found:

  1. st2 run core.local cmd='python --version' returns Python 2

This may or may not be expected and desired, depending on how we envision whole Python 3 story. It returns python 2 since system default python binary in PATH is used instead of python binary from /opt/stackstorm/st2/bin.

It does appear to be working fine for python runner actions though (which is likely the only thing we care about) - st2 run examples.python_runner_print_python_version, /opt/stackstorm/st2/bin/python --version.

vagrant@kami-desktop:~$ st2 run examples.python_runner_print_python_version
.
id: 60319401b8c38397f0e40066
action.ref: examples.python_runner_print_python_version
context.user: st2admin
parameters: None
status: succeeded
start_timestamp: Sat, 20 Feb 2021 22:58:09 UTC
end_timestamp: Sat, 20 Feb 2021 22:58:09 UTC
result: 
  exit_code: 0
  result: null
  stderr: ''
  stdout: "Using Python executable: /opt/stackstorm/virtualenvs/examples/bin/python
Using Python version: 3.6.13 (default, Feb 19 2021, 22:14:26) 
[GCC 5.4.0 20160609]
"

vagrant@kami-desktop:~$ st2 run examples.python_runner_print_python_environment
.
id: 60319560b8c38397f0e40069
action.ref: examples.python_runner_print_python_environment
context.user: st2admin
parameters: None
status: succeeded
start_timestamp: Sat, 20 Feb 2021 23:04:00 UTC
end_timestamp: Sat, 20 Feb 2021 23:04:00 UTC
result: 
  exit_code: 0
  result: null
  stderr: ''
  stdout: "Using Python executable: /opt/stackstorm/virtualenvs/examples/bin/python
Using Python version: 3.6.13 (default, Feb 19 2021, 22:14:26) 
[GCC 5.4.0 20160609]
Platform: Linux-4.4.0-161-generic-x86_64-with-Ubuntu-16.04-xenial
PYTHONPATH: /opt/stackstorm/virtualenvs/examples/lib/python3.6:/opt/stackstorm/virtualenvs/examples/lib/python3.6/site-packages:/opt/stackstorm/packs/examples/actions/lib::/opt/stackstorm/st2/lib/python3.6/site-packages
sys.path: ['/opt/stackstorm/virtualenvs/examples/lib/python3.6/site-packages', '/opt/stackstorm/st2/lib/python3.6/site-packages/python_runner', '/opt/stackstorm/virtualenvs/examples/lib/python3.6', '/opt/stackstorm/virtualenvs/examples/lib/python3.6/site-packages', '/opt/stackstorm/packs/examples/actions/lib', '/', '/opt/stackstorm/st2/lib/python3.6/site-packages', '/opt/stackstorm/virtualenvs/examples/lib/python36.zip', '/opt/stackstorm/virtualenvs/examples/lib/python3.6/lib-dynload', '/usr/lib/python3.6', '/opt/stackstorm/packs/examples/actions/pythonactions']

As far as core.local "issue" goes, we need to decide what's the desired and expected behavior. I believe we should not add /opt/stackstorm/st2/bin/ to PATH for core.local actions - core local actions should observe same PATH rules as any other script which is manually ran from a shell session on that server.

In short, we only care about python version being used for Python runner actions and StackStorm services.

If that's the case, testing instructions need to be updated.

  1. bash completion for st2 CLI doesn't seem to work out of the box

It could be that this has been broken for a long time already, but I do know it worked at some point in the distant past (perhaps we just don't source source the bash completion script by default, didn't dig in).

arm4b commented 3 years ago

@Kami I agree about 1) /opt/stackstorm/st2/bin/python is something to verify for the new py env. I've updated the instructions so the expectation is to make sure that st2 --version is showing py3.

winem commented 3 years ago

I'm done with the tests of fresh st2 installations on CentOS7 and Ubuntu 16.04 now as well. I created an issue + PR for a small issue with the install script on Ubuntu 16.04 and can also confirm topics reported by @Kami

This means:

I also did a cross-check with st2 v3.3 and the st2 auto-completion does not work, too. So this is not a new issue.

I'll proceed with tests upgrading st2 v3.3 to 3.4 on both OS.

amanda11 commented 3 years ago

Started testing on CentOS 8. Found issue that the ansible install docs page hasn't been updated given the EWC donated features - PR in progress Also should get https://github.com/StackStorm/st2docs/pull/1022 reviewed and merged, as the matching code is merged but not the doc PR.

arm4b commented 3 years ago

@Kami @winem Good find about the auto-completion, fresh perspective always helps :+1: Would be nice to document the bug somewhere so we could fix it in the next release.

amanda11 commented 3 years ago

https://github.com/StackStorm/st2docs/pull/1055 - doc change about EWC components

amanda11 commented 3 years ago

Simulated the E2E tests setup - and found that the ansible-script in the user-data didn't seem to get applied to the instance, hence there was no winrm listener on 5986. Manually setup a windows machine, against a CentOS 8 3.4 dev installed via ansible. Successfully ran the following tests from the st2tests/tests pack

amanda11 commented 3 years ago

Quick play with RBAC and ChatOps (hubot) on CentOS 8 ansible install, all good. !help still worked with rbac enabled.

Ansible install on CentOS 7 - went through UI testing list on Chrome. Couldn't find a "Contact Us" - think its replaced by "Support" - so might want to update https://github.com/StackStorm/discussions/wiki/Release-Process#manual-testing-for-web-and-flow-ui

Followed manual instructions for CentOS 7, all good, and setup ChatOps with manual instructions successfully.

amanda11 commented 3 years ago

Possible upgrade or pythonpath issue.

Upgraded StackStorm 3.3 on CentOS 7-> 3.4.

Got error on aws pack after upgrade:

  File \"/opt/stackstorm/st2/lib/python3.6/site-packages/netaddr/compat.py\", line 93, in <module>
    import importlib_resources as _importlib_resources
  File \"/opt/stackstorm/st2/lib/python3.6/site-packages/importlib_resources/__init__.py\", line 3, in <module>
    from ._common import (
  File \"/opt/stackstorm/st2/lib/python3.6/site-packages/importlib_resources/_common.py\", line 2, in <module>
    import pathlib
ImportError: No module named pathlib
",
  "exit_code": 1,
  "result": "None"
}

Problem is that the virtualenv of the pack from when it was installed on ST2 3.3 has python2.7 files in the lib area. If I remove and then install the pack after upgrade it works fine.

Updated PR https://github.com/StackStorm/st2docs/pull/1056 to include instructions that need to uninstall/re-install packs after upgrade on U16 and EL7.

winem commented 3 years ago

Upgrade on CentOS 7 went fine except for the issue(s) reported by @amanda11

I ran into another issue during the upgrade von Ubuntu 16.04 with --u16-add-insecure-py3-ppa:

20210223T193442+0000 The repository is setup! You can now install packages.
20210223T193442+0000 Reading package lists...
20210223T193442+0000 Building dependency tree...
20210223T193442+0000 Reading state information...
20210223T193442+0000 Some packages could not be installed. This may mean that you have
20210223T193442+0000 requested an impossible situation or if you are using the unstable
20210223T193442+0000 distribution that some required packages have not yet been created
20210223T193442+0000 or been moved out of Incoming.
20210223T193442+0000 The following information may help to resolve the situation:
20210223T193442+0000 
20210223T193442+0000 The following packages have unmet dependencies:
20210223T193442+0000  st2 : PreDepends: python3.6 (>= 3.6) but it is not installable
20210223T193442+0000        Depends: python3.6-dev but it is not installable
20210223T193442+0000 E: Unable to correct problems, you have held broken packages.
20210223T193442+0000 ############### ERROR ###############
20210223T193442+0000 # Failed on Install st2 #
20210223T193442+0000 #####################################

And a second execution of curl -sSL https://stackstorm.com/packages/install.sh | bash -s -- --user=st2admin --password=Ch@ngeMe --unstable --u16-add-insecure-py3-ppa fails due to the already installed mongodb instance that was created during the first try - incl. the users and databases:

20210223T193946+0000 2021-02-23T19:39:46.361+0000 E QUERY    [js] Error: couldn't add user: command createUser requires authentication :
20210223T193946+0000 _getErrorWithCode@src/mongo/shell/utils.js:25:13
20210223T193946+0000 DB.prototype.createUser@src/mongo/shell/db.js:1514:15
20210223T193946+0000 @(shell):1:1
20210223T193946+0000 bye
20210223T193946+0000 ############### ERROR ###############
20210223T193946+0000 # Failed on Install st2 dependencies (MongoDB) #
20210223T193946+0000 #####################################

Digging into it now. It looks like the check if Python3.6 is available or not has some caveats and the ppa was not added.

arm4b commented 3 years ago

@winem StackStorm installer doesn't support curl-bash script re-run and idempotence, eg. resuming installation at this moment. Something that we could improve in the future for sure!

20210223T193442+0000 st2 : PreDepends: python3.6 (>= 3.6) but it is not installable 20210223T193442+0000 Depends: python3.6-dev but it is not installable

I think what happened there is failed apt-get update which happens sometimes due to transient network/upstream issues.

winem commented 3 years ago

@armab yes, it's fine that idempotence is not supported. I just forgot that when I ran it again. :)

Anyway, the deadsnakes sources file in /etc/apt/sources.list.d/ was not created. I'll re-run the whole upgrade path and create a snapshot of running st2 3.3 on 16.04 before I try the update next time. This should speed up my troubleshooting (if it's reproducable and an actual issue). I'll share the results latest tomorrow.

minsis commented 3 years ago

Few things tested. As a note I'm using the docker version. Using Chrome Browser.

Some random feedback:

examples pack missing in container?

root@42118eabf091:/opt/stackstorm# ll -d packs/examples2
lrwxrwxrwx 1 root root 27 Feb 23 21:30 packs/examples2 -> /usr/share/doc/st2/examples/
root@42118eabf091:/opt/stackstorm# ll packs/examples2/*
packs/examples2/actions:
total 40
drwxrwxr-x 10 root st2packs 4096 Feb 23 00:02 ./
drwxrwxr-x  9 root st2packs 4096 Feb 23 00:02 ../
drwxrwxr-x  2 root st2packs 4096 Feb 16 21:30 bash_exit_code/
drwxrwxr-x  2 root st2packs 4096 Feb 16 21:30 bash_ping/
drwxrwxr-x  2 root st2packs 4096 Feb 16 21:30 bash_random/
drwxrwxr-x  2 root st2packs 4096 Feb 16 21:30 chains/
drwxrwxr-x  2 root st2packs 4096 Feb 16 21:30 pythonactions/
drwxrwxr-x  3 root st2packs 4096 Feb 23 00:02 ubuntu_pkg_info/
drwxrwxr-x  2 root st2packs 4096 Feb 16 21:30 windows/
drwxrwxr-x  3 root st2packs 4096 Feb 23 00:02 workflows/

packs/examples2/aliases:
total 8
drwxrwxr-x 2 root st2packs 4096 Feb 16 21:30 ./
drwxrwxr-x 9 root st2packs 4096 Feb 23 00:02 ../

packs/examples2/lib:
total 8
drwxrwxr-x 2 root st2packs 4096 Feb 16 21:30 ./
drwxrwxr-x 9 root st2packs 4096 Feb 23 00:02 ../

packs/examples2/rules:
total 8
drwxrwxr-x 2 root st2packs 4096 Feb 16 21:30 ./
drwxrwxr-x 9 root st2packs 4096 Feb 23 00:02 ../

packs/examples2/sensors:
total 8
drwxrwxr-x 2 root st2packs 4096 Feb 16 21:30 ./
drwxrwxr-x 9 root st2packs 4096 Feb 23 00:02 ../

packs/examples2/tests:
total 8
drwxrwxr-x 2 root st2packs 4096 Feb 16 21:30 ./
drwxrwxr-x 9 root st2packs 4096 Feb 23 00:02 ../

packs/examples2/triggers:
total 8
drwxrwxr-x 2 root st2packs 4096 Feb 16 21:30 ./
drwxrwxr-x 9 root st2packs 4096 Feb 23 00:02 ../
root@42118eabf091:/opt/stackstorm# ll packs/examples2/actions/*
packs/examples2/actions/bash_exit_code:
total 8
drwxrwxr-x  2 root st2packs 4096 Feb 16 21:30 ./
drwxrwxr-x 10 root st2packs 4096 Feb 23 00:02 ../

packs/examples2/actions/bash_ping:
total 8
drwxrwxr-x  2 root st2packs 4096 Feb 16 21:30 ./
drwxrwxr-x 10 root st2packs 4096 Feb 23 00:02 ../

packs/examples2/actions/bash_random:
total 8
drwxrwxr-x  2 root st2packs 4096 Feb 16 21:30 ./
drwxrwxr-x 10 root st2packs 4096 Feb 23 00:02 ../

packs/examples2/actions/chains:
total 8
drwxrwxr-x  2 root st2packs 4096 Feb 16 21:30 ./
drwxrwxr-x 10 root st2packs 4096 Feb 23 00:02 ../

packs/examples2/actions/pythonactions:
total 8
drwxrwxr-x  2 root st2packs 4096 Feb 16 21:30 ./
drwxrwxr-x 10 root st2packs 4096 Feb 23 00:02 ../

packs/examples2/actions/ubuntu_pkg_info:
total 12
drwxrwxr-x  3 root st2packs 4096 Feb 23 00:02 ./
drwxrwxr-x 10 root st2packs 4096 Feb 23 00:02 ../
drwxrwxr-x  2 root st2packs 4096 Feb 16 21:30 lib/

packs/examples2/actions/windows:
total 8
drwxrwxr-x  2 root st2packs 4096 Feb 16 21:30 ./
drwxrwxr-x 10 root st2packs 4096 Feb 23 00:02 ../

packs/examples2/actions/workflows:
total 12
drwxrwxr-x  3 root st2packs 4096 Feb 23 00:02 ./
drwxrwxr-x 10 root st2packs 4096 Feb 23 00:02 ../
drwxrwxr-x  2 root st2packs 4096 Feb 16 21:30 tests/
root@42118eabf091:/opt/stackstorm# ll packs/examples2/actions/*/*
packs/examples2/actions/ubuntu_pkg_info/lib:
total 8
drwxrwxr-x 2 root st2packs 4096 Feb 16 21:30 ./
drwxrwxr-x 3 root st2packs 4096 Feb 23 00:02 ../

packs/examples2/actions/workflows/tests:
total 8
drwxrwxr-x 2 root st2packs 4096 Feb 16 21:30 ./
drwxrwxr-x 3 root st2packs 4096 Feb 23 00:02 ../
amanda11 commented 3 years ago

Upgrade on CentOS 7 went fine except for the issue(s) reported by @amanda11

I ran into another issue during the upgrade von Ubuntu 16.04 with --u16-add-insecure-py3-ppa:

20210223T193442+0000 The repository is setup! You can now install packages.
20210223T193442+0000 Reading package lists...
20210223T193442+0000 Building dependency tree...
20210223T193442+0000 Reading state information...
20210223T193442+0000 Some packages could not be installed. This may mean that you have
20210223T193442+0000 requested an impossible situation or if you are using the unstable
20210223T193442+0000 distribution that some required packages have not yet been created
20210223T193442+0000 or been moved out of Incoming.
20210223T193442+0000 The following information may help to resolve the situation:
20210223T193442+0000 
20210223T193442+0000 The following packages have unmet dependencies:
20210223T193442+0000  st2 : PreDepends: python3.6 (>= 3.6) but it is not installable
20210223T193442+0000        Depends: python3.6-dev but it is not installable
20210223T193442+0000 E: Unable to correct problems, you have held broken packages.
20210223T193442+0000 ############### ERROR ###############
20210223T193442+0000 # Failed on Install st2 #
20210223T193442+0000 #####################################

And a second execution of curl -sSL https://stackstorm.com/packages/install.sh | bash -s -- --user=st2admin --password=Ch@ngeMe --unstable --u16-add-insecure-py3-ppa fails due to the already installed mongodb instance that was created during the first try - incl. the users and databases:

20210223T193946+0000 2021-02-23T19:39:46.361+0000 E QUERY    [js] Error: couldn't add user: command createUser requires authentication :
20210223T193946+0000 _getErrorWithCode@src/mongo/shell/utils.js:25:13
20210223T193946+0000 DB.prototype.createUser@src/mongo/shell/db.js:1514:15
20210223T193946+0000 @(shell):1:1
20210223T193946+0000 bye
20210223T193946+0000 ############### ERROR ###############
20210223T193946+0000 # Failed on Install st2 dependencies (MongoDB) #
20210223T193946+0000 #####################################

Digging into it now. It looks like the check if Python3.6 is available or not has some caveats and the ppa was not added.

I have a fix @winem for this - https://github.com/StackStorm/st2-packages/pull/691

I was looking into it as the E2E tests started failing...

hnanchahal commented 3 years ago

Updated the helm chart https://github.com/StackStorm/stackstorm-ha/pull/182. Incorporated the suggested updates. Have this up and running in my K8s cluster with RBAC and LDAP enabled, So far everything looks good. Just noted some modifications to be made to the cli. Will submit a separate PR for that. Using Ubuntu 18.04 for the K8s cluster. Kubernetes and Ansible pack fail to install with the error: “error: command 'x86_64-linux-gnu-gcc' failed: No such file or directory”

Not sure if this needs to handled at the base image level. fixed it at our end by creating our custom st2actionrunner image but it might make sense to package this dependency in the base image as these are very commonly used packs and pretty there would be other packs which would need this dependency.

FROM stackstorm/st2actionrunner:3.4dev

ARG APP_VERSION=0.4

RUN sudo apt-get update RUN apt-get install -y gcc libkrb5-dev

Regards Harsh Nanchahal

On Tue, Feb 23, 2021 at 2:48 PM Amanda McGuinness notifications@github.com wrote:

Upgrade on CentOS 7 went fine except for the issue(s) reported by @amanda11 https://github.com/amanda11

I ran into another issue during the upgrade von Ubuntu 16.04 with --u16-add-insecure-py3-ppa:

20210223T193442+0000 The repository is setup! You can now install packages. 20210223T193442+0000 Reading package lists... 20210223T193442+0000 Building dependency tree... 20210223T193442+0000 Reading state information... 20210223T193442+0000 Some packages could not be installed. This may mean that you have 20210223T193442+0000 requested an impossible situation or if you are using the unstable 20210223T193442+0000 distribution that some required packages have not yet been created 20210223T193442+0000 or been moved out of Incoming. 20210223T193442+0000 The following information may help to resolve the situation: 20210223T193442+0000 20210223T193442+0000 The following packages have unmet dependencies: 20210223T193442+0000 st2 : PreDepends: python3.6 (>= 3.6) but it is not installable 20210223T193442+0000 Depends: python3.6-dev but it is not installable 20210223T193442+0000 E: Unable to correct problems, you have held broken packages. 20210223T193442+0000 ############### ERROR ############### 20210223T193442+0000 # Failed on Install st2 # 20210223T193442+0000 #####################################

And a second execution of curl -sSL https://stackstorm.com/packages/install.sh | bash -s -- --user=st2admin --password=Ch@ngeMe --unstable --u16-add-insecure-py3-ppa fails due to the already installed mongodb instance that was created during the first try - incl. the users and databases:

20210223T193946+0000 2021-02-23T19:39:46.361+0000 E QUERY [js] Error: couldn't add user: command createUser requires authentication : 20210223T193946+0000 _getErrorWithCode@src/mongo/shell/utils.js:25:13 20210223T193946+0000 DB.prototype.createUser@src/mongo/shell/db.js:1514:15 20210223T193946+0000 @(shell):1:1 20210223T193946+0000 bye 20210223T193946+0000 ############### ERROR ############### 20210223T193946+0000 # Failed on Install st2 dependencies (MongoDB) # 20210223T193946+0000 #####################################

Digging into it now. It looks like the check if Python3.6 is available or not has some caveats and the ppa was not added.

I have a fix @winem https://github.com/winem for this - StackStorm/st2-packages#691 https://github.com/StackStorm/st2-packages/pull/691

I was looking into it as the E2E tests started failing...

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/StackStorm/discussions/issues/66#issuecomment-784568780, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD2PWBSGGXDE44SPM2MJGA3TAQWFNANCNFSM4X5BVMYQ .

cognifloyd commented 3 years ago

Using Ubuntu 18.04 for the K8s cluster. Kubernetes and Ansible pack fail to install with the error: “error: command 'x86_64-linux-gnu-gcc' failed: No such file or directory”

pip reaches for gcc only when it can't resolve a wheel to download so it needs to build a wheel for a package that is not pure-python. Test infra uses a newer version of pip than what we install in production. So we're unlikely to find such errors while testing (before we cut a release) because pip is new enough in testing.

Please, let's merge these PRs to sanitize/make-consistent the pip version we use everywhere. https://github.com/StackStorm/st2packaging-dockerfiles/pull/103 https://github.com/StackStorm/st2-packages/pull/687 https://github.com/StackStorm-Exchange/ci/pull/102 https://github.com/StackStorm/st2/pull/5165

cognifloyd commented 3 years ago

Edited my last post to add one more PR.

This one resolves pip version issues as well: https://github.com/StackStorm/st2/pull/5165

Oddly enough, we have been using pip 21.0.1 during some of the testing because pip-tools (testing-requirements.txt) was pulling in the newer version.

arm4b commented 3 years ago

Thanks @hnanchahal for testing the RBAC/LDAP in K8s/HA environment. Appreciate the feedback and involvement :+1:

Not sure if this needs to handled at the base image level. fixed it at our end by creating our custom st2actionrunner image but it might make sense to package this dependency in the base image as these are very commonly used packs and pretty there would be other packs which would need this dependency.

That's right. Ansible pack per requirements (and some other packs) require additional OS-level dependencies like gcc which we can't package in base Docker images and so we recommend doing it yourself in your infra, which you did and that worked.

@cognifloyd It has nothing to do with the pip version and all those PRs are not bugfixes, but maintenance tasks that are not blockers/requirements for this release. The situation is that the st2+pip version we've packaged now is what we'll ship at this round, if the team won't discover serious blocker reasons last-minute. The master is freezed. Switching the wheels at this moment means we have to restart the testing as those packages will be rebuilt in new environment and rely on new tools which is a risk and should be done slightly beforehand. Thanks for your dedication and help! Can't wait once we'll get the v3.4.0 out and then on-board all the pip + performance improvements :100:

cognifloyd commented 3 years ago

@cognifloyd It has nothing to do with the pip version and all those PRs are not bugfixes, but maintenance tasks that are not blockers/requirements for this release.

I disagree. If we explicitly pin one version of a package, but we test with a different version, then that is a bug.

cognifloyd commented 3 years ago

As far as the ansible pack is concerned, that note in the readme is somewhat incorrect. gcc is only needed because the version of pip we're using in production is broken.

Ansible is a pure-python package. It depends on cryptography and PyYAML which have non-python components. Both cryptography and PyYAML provide wheels, but the old versions of pip can't use them so it tries to build its own wheel instead. If we used the same version of pip in production that we have been running unit/integration tests with, then this would be a non-issue.

amanda11 commented 3 years ago

Tested manual and ansible deployments on U16. All good. Not had chance to confirm upgrade on U16. Quick play in those environments with workflow designer, and the st2-self-check all ran successfully.

amanda11 commented 3 years ago

Some random feedback:

  • The save button in the visualizer is easy to miss as it looks like its more of a navigation button. Maybe put a big button that says "Save"; same goes for the run button
  • I know workflows have to have an action behind it, but it seems like it would be cleaner to maybe have the visualizer in its own tab?
  • When hovering over the anchor point on a task object maybe have the icon change to a move icon.
  • When closing the visualizer after making a change there should be a warning telling you that you haven't saved anything.

@minsis This is great feedback, would you mind (if you haven't already) create a feature request on the st2web repository for these. They wouldn't be regression problems, but good points to improve the UI in future releases.

winem commented 3 years ago

I'm running into another issue with the upgrade from 3.3 to 3.4 on U16 now. It looks like an issue with stdin redirection again. It appears if the st2.conf was modified and does not match the one provided by st2:

20210227T212722+0000 Unpacking st2 (3.4dev-58) over (3.3.0-4) ...
20210227T212730+0000 Processing triggers for libc-bin (2.23-0ubuntu11.2) ...
20210227T212730+0000 Setting up libpython3.6:amd64 (3.6.13-1+xenial2) ...
20210227T212730+0000 Setting up libpython3.6-dev:amd64 (3.6.13-1+xenial2) ...
20210227T212730+0000 Setting up python3.6-dev (3.6.13-1+xenial2) ...
20210227T212730+0000 Setting up st2 (3.4dev-58) ...
20210227T212730+0000 Installing new version of config file /etc/logrotate.d/st2 ...
20210227T212730+0000 
20210227T212730+0000 Configuration file '/etc/st2/st2.conf'
20210227T212730+0000  ==> Modified (by you or by a script) since installation.
20210227T212730+0000  ==> Package distributor has shipped an updated version.
20210227T212730+0000    What would you like to do about it ?  Your options are:
20210227T212730+0000     Y or I  : install the package maintainer's version
20210227T212730+0000     N or O  : keep your currently-installed version
20210227T212730+0000       D     : show the differences between the versions
20210227T212730+0000       Z     : start a shell to examine the situation
20210227T212730+0000  The default action is to keep your current version.
20210227T212730+0000 *** st2.conf (Y/I/N/O/D/Z) [default=N] ? dpkg: error processing package st2 (--configure):
20210227T212730+0000  end of file on stdin at conffile prompt
20210227T212730+0000 Processing triggers for libc-bin (2.23-0ubuntu11.2) ...
20210227T212730+0000 Errors were encountered while processing:
20210227T212730+0000  st2
20210227T212731+0000 E: Sub-process /usr/bin/dpkg returned an error code (1)
20210227T212731+0000 ############### ERROR ###############
20210227T212731+0000 # Failed on Install st2 #
20210227T212731+0000 #####################################

So it does not wait for any user input and fails instead immediately. This seems to be the same as: https://github.com/StackStorm/st2-packages/issues/688

The upgrade seems to work fine when I revert all changes on st2.conf and run the curl-bash-command. At least the self-verification was successful. Still have to do some more tests.

winem commented 3 years ago

It looks like we can avoid the stdin issue as described above and in https://github.com/StackStorm/st2-packages/issues/688 by using the following command:

bash <(curl -sSL https://stackstorm.com/packages/install.sh) --user=st2admin --password=Ch@ngeMe --unstable --u16-add-insecure-py3-ppa

instead of the original one:

curl -sSL https://stackstorm.com/packages/install.sh | bash -s -- --user=st2admin --password=Ch@ngeMe --unstable

The one I suggested does not have any issues with interactions and does wait for the users input. I tested all available options Y/I, N/O, D and Z successfully.

I do not see any additional issue or (security) risk by changing the commands but am happy about additional views on this suggestion.

arm4b commented 3 years ago

That enhancement would be helpful (please submit a PR), but we don't support upgrading st2 versions via curl|bash installer as it doesn't know about the migration path, nor idempotent for that operation. The user needs to follow the https://docs.stackstorm.com/latest/install/upgrades.html#general-upgrade-procedure instead.

winem commented 3 years ago

PR provided. And yes, it's just for testing/development purpose. :+1:

winem commented 3 years ago

I'm done with the tests on different installations. Let me just summarize the tests here for a better overview:

Testing means:

Tested Versions:

~I just found some (minor?) issues:~ ~1. After the manual upgrade on U16 st2resultstracker was still running. The running service was not stopped by the script. The service was disabled and did not start automatically again after a reboot. st2resultstracker was stopped during and no longer running after the upgrade via ansible-st2, so this is just an issue with the bash script.~ ~2. The FlowUI works fine for viewing existing Orquesta workflows but you can't save any changes due to a permission issue. Example: "Error saving workflow.: Unable to write data to "/opt/stackstorm/packs/packs/actions/workflows/install.yaml" (permission denied). Make sure permissions for that pack directory are configured correctly so st2api can write to it." Creating new workflows and updating them works fine. This issue is not limited to Ubuntu 16.04 and was also reproducable on 18.04.~

~So, is 1) an actual issue? Do we expect the same issue when st2 3.4 will be installed via package managers like apt, yum, etc? Do we want to add a note to the migration docs?~ ~Is 2) known & expected and I just missed it in the documentation? I couldn't find a corresponding issue. I'll create one tomorrow unless we figure this out earlier than that here.~

amanda11 commented 3 years ago
  1. After the manual upgrade on U16 st2resultstracker was still running. The running service was not stopped by the script. The service was disabled and did not start automatically again after a reboot. st2resultstracker was stopped during and no longer running after the upgrade via ansible-st2, so this is just an issue with the bash script.

@winem - Did you follow the manual upgrade instructions? On CentOS 7 when following the manual upgrade instructions there is no problem. As the first instruction is to stop all ST2 services, therefore st2resultstracker is stopped at that point, and then never restarted when the ST2 services is restarted. If you used the bash installer for upgrade, then that isn't the documented upgrade procedure - so I think that isn't relevant.

amanda11 commented 3 years ago

2. The FlowUI works fine for viewing existing Orquesta workflows but you can't save any changes due to a permission issue. Example: "Error saving workflow.: Unable to write data to "/opt/stackstorm/packs/packs/actions/workflows/install.yaml" (permission denied). Make sure permissions for that pack directory are configured correctly so st2api can write to it." Creating new workflows and updating them works fine. This issue is not limited to Ubuntu 16.04 and was also reproducable on 18.04.

I've just installed a pack on a CentOS 7 upgraded to 3.4, and I can edit the workflows. But I do need to make sure it has fully loaded everything, so when I click on action I can see that the action has loaded etc. I can't save packs.install workflow but I think that might be due to: https://github.com/StackStorm/st2web/pull/859 or https://github.com/StackStorm/st2web/pull/858. (And was same on EWC). So there are some fields I think workflow composer can't handle yet. (which we should probably document).

I installed one of my own packs after upgrade, and could edit its workflows fine. And I can't see any difference in ownership of files.

@winem Is it possible to test on a workflow other than the packs one, and see if problem exists (one that doesn't have the fields that those two pull requests above mention).

amanda11 commented 3 years ago

Successfully performed U16 upgrade, using the new st2ctl reload register-recreate-virtualenvs. After upgrade, installed a pack of mine that had an Orquesta workflow and was able to edit it.

I couldn't necessarily find a suitable place to document the problems about editing workflows with the designer that had retry or delay parameters.... There are pull requests in progress for those, so could just create an issue in st2web and link the in-progress PRs to that?

winem commented 3 years ago

@winem Is it possible to test on a workflow other than the packs one, and see if problem exists (one that doesn't have the fields that those two pull requests above mention).

The error I saw was not related to these properties. But I was able to confirm that there is a very specific error message for these cases which is good!

To make it short: Both of my the 2 issues are obsolete now. The issue saving the workflow was caused by a permission issue. Investigation has shown that files of packs that were installed during the self-verification are owned by root:root while those of packs installed via the UI or st2 pack install are owned by root:st2packs. The FlowUI process running under the st2 user does not have the required write permission on those workflows installed during the self-verification.

@amanda11 to me, it looks like we're fine with testing CentOS7 and U16 now. Do you see further test cases? If yes, just let me know and I'll proceed with them.

blag commented 3 years ago

@amanda11 The task delay and retry PRs for st2web are StackStorm/st2web#858 and StackStorm/st2web#859, respectively. They will likely go into v3.5. 😕

amanda11 commented 3 years ago

@winem I think we've hit everything we need to, unless I'm missing something.

The only thing of concern to me, is whether the issues mentioned on docker are blocking or not. @armab Do you have a view on that?

Neither myself or @blag are that familiar on the docker use-cases.

Other than that I think we've tested all the scenarios, and fixed the regressions....

amanda11 commented 3 years ago

@amanda11 The task delay and retry PRs for st2web are StackStorm/st2web#858 and StackStorm/st2web#859, respectively. They will likely go into v3.5. 😕

Yeah - I see they were in progress. And those problems existed in EWC - so agree not blocking for 3.4 release.

arm4b commented 3 years ago

Thanks! There is no blocker from the Docker side for starting a release.

In fact, all the deployment methods (Puppet, Ansible, Docker, K8s, Vagrant) really come as a secondary comparing to st2 core, unless there's some terribly wrong that's failing CI and needs fixing.

So we're good from that standpoint :+1:

minsis commented 3 years ago

Ok I'm going to try and explain this the best I can here, sorry if it seems like ramblings.

Something happened with my docker setup when I enabled RBAC with LDAP. I'm still trying to troubleshoot why this is happening but one of the containers is continually trying to authenticate my user and basically locked out my account. My password was reset by our IT team but my account is continually locked out (probably due to caching) and I had to disable LDAP to get this to stop.

Here's the events in order:

  1. LDAP was enabled and working correctly
  2. RBAC was then enabled but wasn't running correctly because I didn't have "backend" set
  3. Set the backend to "default"
  4. My account in LDAP is now locked out.
  5. I restarted containers, rebuilt containers, closed all browsers, etc but I couldn't recover this. Eventually had to disable LDAP all together to get it to stop.

So I'm not exactly sure where or why this happened. I looked at the logs of all the other containers and none of them were trying to auth my account. It was solely the auth container that was trying to auth my account every 5 seconds.

minsis commented 3 years ago

So I have LDAP disabled and then reenabled RBAC only and its now failing again due to invalid login. So something has to be cached in the database and even with LDAP disabled its still trying to reauth me to the LDAP backend

[auth]
mode = standalone
backend = ldap
backend_kwargs = <--some config stuff-->
enable = False
use_ssl = False
logging = /etc/st2/logging.auth.gunicorn.conf
debug = True

[rbac]
enable = True
backend = default

I'm guessing even though enabled = False, its still trying to auth to ldap since backend is still set to ldap. Other than this issue I'm not even sure how to clear the cache to stop it from trying to auth with my old LDAP login.

arm4b commented 3 years ago

Thanks, @minsis. Looks like you already documented the Feature request/Issue in the respective Docker repository: https://github.com/StackStorm/st2-docker/issues/219. There is a somewhat similar WIP PR contribution in the K8s from the Community: https://github.com/StackStorm/stackstorm-ha/pull/182 Though both are not tied to any release.

Because Docker and K8s might need proper implementation in the future, keeping in mind their multi-container specifics, I'd recommend testing LDAP/RBAC with the VM at this moment, following documentation:

minsis commented 3 years ago

The st-docker#219 issue is just for getting a volume added to store the rbac role files.

The current issue here is that something has broken with LADP and RBAC to where its locking out LDAP accounts because it tries to reauth every 5s (which I also put in an issue on that repo) on a cached credential. I'm just not sure where to go from here in terms of troubleshooting. Every time I turn on LDAP/RBAC it tries to auth my old credentials that are cached somewhere and I can't seem to clear them out.

I would love to test directly on a VM, but at work I have limited resources and it takes our provisioning department weeks to get something built out.

blag commented 3 years ago

You can spin up a VM locally with st2vagrant and configure it that way.

amanda11 commented 3 years ago

Since Saturday install on EL8 failing. Looks to be that we are picking up a new version of rabbitmq, which requires a new version of erlang, and that version of erlang is not available in CentOS or the epel repos. https://github.com/rabbitmq/rabbitmq-server/releases/tag/v3.8.13

Also affecting old releases, unless pin rabbitmq version...

guzzijones commented 3 years ago

I upgraded our dev box.

  1. if you use coordination you will need to reinstall redis using the documented instructions. After the upgrade my st2api stopped working with no module named redis stack trace. pip install redis intructions
  2. also if you were using the 'Enterprise RBAC' or 'Enterprise LDAP' you will be forced to upgrade to one of the unstable dev packages for rbac now. The 'enterprise' backend config option will not work anymore.
guzzijones commented 3 years ago

FWIW, I am on Centos 7 which now uses python3. I fired off a few large workflows and they all seem to be working as expected.

guzzijones commented 3 years ago

Probably worth mentioning that if on a distro with python2.7 as the default that all sensor packs will need to be changed to python version 3 in the pack.yaml and reinstalled