Closed blag closed 3 years ago
st2ctl status showing
##### st2 components status #####
st2actionrunner PID: 18733
st2actionrunner PID: 18737
st2actionrunner PID: 18740
st2actionrunner PID: 18743
st2actionrunner PID: 18746
st2actionrunner PID: 18749
st2actionrunner PID: 18753
st2actionrunner PID: 18756
st2actionrunner PID: 18762
st2actionrunner PID: 18765
st2api PID: 18635
st2api PID: 18647
st2stream PID: 18583
st2stream PID: 18628
st2auth PID: 18523
st2auth PID: 18535
st2garbagecollector PID: 17552
st2notifier PID: 17555
st2rulesengine PID: 17558
st2sensorcontainer PID: 18653
st2chatops is not running.
st2timersengine PID: 17569
st2workflowengine PID: 18665
st2scheduler PID: 17575
in the result above it is showing st2chatops not running.
sudo service st2chatops status
● st2chatops.service - StackStorm service st2chatops
Loaded: loaded (/lib/systemd/system/st2chatops.service; enabled; vendor preset: enabled)
Active: activating (auto-restart) (Result: exit-code) since Sat 2021-02-20 16:04:35 UTC; 3s ago
Process: 22638 ExecStart=/opt/stackstorm/chatops/bin/hubot $DAEMON_ARGS (code=exited, status=1/FAILURE)
Main PID: 22638 (code=exited, status=1/FAILURE)
Feb 20 16:04:35 himadri systemd[1]: st2chatops.service: Main process exited, code=exited, status=1/FAILURE
Feb 20 16:04:35 himadri systemd[1]: st2chatops.service: Unit entered failed state.
Feb 20 16:04:35 himadri systemd[1]: st2chatops.service: Failed with result 'exit-code'.
Even after the machine restart, the same thing happens.
@himadriganguly Thank you for testing v3.4a!
That's normal when you haven't configured st2chatops by modifying /opt/stackstorm/chatops/st2chatops.env
. The default StackStorm installation will not configure ChatOps for you since that is an optional component that requires integration with an external service.
We should try to improve that situation in the future, but for v3.4 I'm not considering that a bug.
If you have configured st2chatops, then that's definitely a bug. In which case, please post your st2chatops.env
file (but remember to redact any API keys or Slack tokens) so we can figure out if it's a bug in your configuration or in st2chatops.
Running Self-Verification Result
@blag Thank you for the explanation. I will be testing other things according to the checklist and will post the details. I will try to contribute as much as I can.
Just a short update since I ran into an issue with the load_diag workflow in the Linux pack and I'll look into that now:
Update: The issue with the load_diag workflow isn't a regression or issue specific to st2 3.4 and there are already open issues that cover these issues. So I'll proceed.
I run tests on Ubuntu Xenial (16.04) here are the issues I found:
st2 run core.local cmd='python --version'
returns Python 2This may or may not be expected and desired, depending on how we envision whole Python 3 story. It returns python 2 since system default python binary in PATH
is used instead of python binary from /opt/stackstorm/st2/bin
.
It does appear to be working fine for python runner actions though (which is likely the only thing we care about) - st2 run examples.python_runner_print_python_version
, /opt/stackstorm/st2/bin/python --version
.
vagrant@kami-desktop:~$ st2 run examples.python_runner_print_python_version
.
id: 60319401b8c38397f0e40066
action.ref: examples.python_runner_print_python_version
context.user: st2admin
parameters: None
status: succeeded
start_timestamp: Sat, 20 Feb 2021 22:58:09 UTC
end_timestamp: Sat, 20 Feb 2021 22:58:09 UTC
result:
exit_code: 0
result: null
stderr: ''
stdout: "Using Python executable: /opt/stackstorm/virtualenvs/examples/bin/python
Using Python version: 3.6.13 (default, Feb 19 2021, 22:14:26)
[GCC 5.4.0 20160609]
"
vagrant@kami-desktop:~$ st2 run examples.python_runner_print_python_environment
.
id: 60319560b8c38397f0e40069
action.ref: examples.python_runner_print_python_environment
context.user: st2admin
parameters: None
status: succeeded
start_timestamp: Sat, 20 Feb 2021 23:04:00 UTC
end_timestamp: Sat, 20 Feb 2021 23:04:00 UTC
result:
exit_code: 0
result: null
stderr: ''
stdout: "Using Python executable: /opt/stackstorm/virtualenvs/examples/bin/python
Using Python version: 3.6.13 (default, Feb 19 2021, 22:14:26)
[GCC 5.4.0 20160609]
Platform: Linux-4.4.0-161-generic-x86_64-with-Ubuntu-16.04-xenial
PYTHONPATH: /opt/stackstorm/virtualenvs/examples/lib/python3.6:/opt/stackstorm/virtualenvs/examples/lib/python3.6/site-packages:/opt/stackstorm/packs/examples/actions/lib::/opt/stackstorm/st2/lib/python3.6/site-packages
sys.path: ['/opt/stackstorm/virtualenvs/examples/lib/python3.6/site-packages', '/opt/stackstorm/st2/lib/python3.6/site-packages/python_runner', '/opt/stackstorm/virtualenvs/examples/lib/python3.6', '/opt/stackstorm/virtualenvs/examples/lib/python3.6/site-packages', '/opt/stackstorm/packs/examples/actions/lib', '/', '/opt/stackstorm/st2/lib/python3.6/site-packages', '/opt/stackstorm/virtualenvs/examples/lib/python36.zip', '/opt/stackstorm/virtualenvs/examples/lib/python3.6/lib-dynload', '/usr/lib/python3.6', '/opt/stackstorm/packs/examples/actions/pythonactions']
As far as core.local "issue" goes, we need to decide what's the desired and expected behavior. I believe we should not add /opt/stackstorm/st2/bin/
to PATH
for core.local actions - core local actions should observe same PATH rules as any other script which is manually ran from a shell session on that server.
In short, we only care about python version being used for Python runner actions and StackStorm services.
If that's the case, testing instructions need to be updated.
st2
CLI doesn't seem to work out of the boxIt could be that this has been broken for a long time already, but I do know it worked at some point in the distant past (perhaps we just don't source source the bash completion script by default, didn't dig in).
@Kami I agree about 1)
/opt/stackstorm/st2/bin/python
is something to verify for the new py env.
I've updated the instructions so the expectation is to make sure that st2 --version
is showing py3.
I'm done with the tests of fresh st2 installations on CentOS7 and Ubuntu 16.04 now as well. I created an issue + PR for a small issue with the install script on Ubuntu 16.04 and can also confirm topics reported by @Kami
This means:
I also did a cross-check with st2 v3.3 and the st2 auto-completion does not work, too. So this is not a new issue.
I'll proceed with tests upgrading st2 v3.3 to 3.4 on both OS.
Started testing on CentOS 8. Found issue that the ansible install docs page hasn't been updated given the EWC donated features - PR in progress Also should get https://github.com/StackStorm/st2docs/pull/1022 reviewed and merged, as the matching code is merged but not the doc PR.
@Kami @winem Good find about the auto-completion, fresh perspective always helps :+1: Would be nice to document the bug somewhere so we could fix it in the next release.
https://github.com/StackStorm/st2docs/pull/1055 - doc change about EWC components
Simulated the E2E tests setup - and found that the ansible-script in the user-data didn't seem to get applied to the instance, hence there was no winrm listener on 5986. Manually setup a windows machine, against a CentOS 8 3.4 dev installed via ansible. Successfully ran the following tests from the st2tests/tests pack
Quick play with RBAC and ChatOps (hubot) on CentOS 8 ansible install, all good. !help still worked with rbac enabled.
Ansible install on CentOS 7 - went through UI testing list on Chrome. Couldn't find a "Contact Us" - think its replaced by "Support" - so might want to update https://github.com/StackStorm/discussions/wiki/Release-Process#manual-testing-for-web-and-flow-ui
Followed manual instructions for CentOS 7, all good, and setup ChatOps with manual instructions successfully.
Possible upgrade or pythonpath issue.
Upgraded StackStorm 3.3 on CentOS 7-> 3.4.
Got error on aws pack after upgrade:
File \"/opt/stackstorm/st2/lib/python3.6/site-packages/netaddr/compat.py\", line 93, in <module>
import importlib_resources as _importlib_resources
File \"/opt/stackstorm/st2/lib/python3.6/site-packages/importlib_resources/__init__.py\", line 3, in <module>
from ._common import (
File \"/opt/stackstorm/st2/lib/python3.6/site-packages/importlib_resources/_common.py\", line 2, in <module>
import pathlib
ImportError: No module named pathlib
",
"exit_code": 1,
"result": "None"
}
Problem is that the virtualenv of the pack from when it was installed on ST2 3.3 has python2.7 files in the lib area. If I remove and then install the pack after upgrade it works fine.
Updated PR https://github.com/StackStorm/st2docs/pull/1056 to include instructions that need to uninstall/re-install packs after upgrade on U16 and EL7.
Upgrade on CentOS 7 went fine except for the issue(s) reported by @amanda11
I ran into another issue during the upgrade von Ubuntu 16.04 with --u16-add-insecure-py3-ppa
:
20210223T193442+0000 The repository is setup! You can now install packages.
20210223T193442+0000 Reading package lists...
20210223T193442+0000 Building dependency tree...
20210223T193442+0000 Reading state information...
20210223T193442+0000 Some packages could not be installed. This may mean that you have
20210223T193442+0000 requested an impossible situation or if you are using the unstable
20210223T193442+0000 distribution that some required packages have not yet been created
20210223T193442+0000 or been moved out of Incoming.
20210223T193442+0000 The following information may help to resolve the situation:
20210223T193442+0000
20210223T193442+0000 The following packages have unmet dependencies:
20210223T193442+0000 st2 : PreDepends: python3.6 (>= 3.6) but it is not installable
20210223T193442+0000 Depends: python3.6-dev but it is not installable
20210223T193442+0000 E: Unable to correct problems, you have held broken packages.
20210223T193442+0000 ############### ERROR ###############
20210223T193442+0000 # Failed on Install st2 #
20210223T193442+0000 #####################################
And a second execution of curl -sSL https://stackstorm.com/packages/install.sh | bash -s -- --user=st2admin --password=Ch@ngeMe --unstable --u16-add-insecure-py3-ppa
fails due to the already installed mongodb instance that was created during the first try - incl. the users and databases:
20210223T193946+0000 2021-02-23T19:39:46.361+0000 E QUERY [js] Error: couldn't add user: command createUser requires authentication :
20210223T193946+0000 _getErrorWithCode@src/mongo/shell/utils.js:25:13
20210223T193946+0000 DB.prototype.createUser@src/mongo/shell/db.js:1514:15
20210223T193946+0000 @(shell):1:1
20210223T193946+0000 bye
20210223T193946+0000 ############### ERROR ###############
20210223T193946+0000 # Failed on Install st2 dependencies (MongoDB) #
20210223T193946+0000 #####################################
Digging into it now. It looks like the check if Python3.6 is available or not has some caveats and the ppa was not added.
@winem StackStorm installer doesn't support curl-bash script re-run and idempotence, eg. resuming installation at this moment. Something that we could improve in the future for sure!
20210223T193442+0000 st2 : PreDepends: python3.6 (>= 3.6) but it is not installable 20210223T193442+0000 Depends: python3.6-dev but it is not installable
I think what happened there is failed apt-get update
which happens sometimes due to transient network/upstream issues.
@armab yes, it's fine that idempotence is not supported. I just forgot that when I ran it again. :)
Anyway, the deadsnakes sources file in /etc/apt/sources.list.d/
was not created. I'll re-run the whole upgrade path and create a snapshot of running st2 3.3 on 16.04 before I try the update next time. This should speed up my troubleshooting (if it's reproducable and an actual issue).
I'll share the results latest tomorrow.
Few things tested. As a note I'm using the docker version. Using Chrome Browser.
st2common.exceptions.db.StackStormDBObjectNotFoundError: Resource with a ref or id "webui" not found
Some random feedback:
examples pack missing in container?
root@42118eabf091:/opt/stackstorm# ll -d packs/examples2
lrwxrwxrwx 1 root root 27 Feb 23 21:30 packs/examples2 -> /usr/share/doc/st2/examples/
root@42118eabf091:/opt/stackstorm# ll packs/examples2/*
packs/examples2/actions:
total 40
drwxrwxr-x 10 root st2packs 4096 Feb 23 00:02 ./
drwxrwxr-x 9 root st2packs 4096 Feb 23 00:02 ../
drwxrwxr-x 2 root st2packs 4096 Feb 16 21:30 bash_exit_code/
drwxrwxr-x 2 root st2packs 4096 Feb 16 21:30 bash_ping/
drwxrwxr-x 2 root st2packs 4096 Feb 16 21:30 bash_random/
drwxrwxr-x 2 root st2packs 4096 Feb 16 21:30 chains/
drwxrwxr-x 2 root st2packs 4096 Feb 16 21:30 pythonactions/
drwxrwxr-x 3 root st2packs 4096 Feb 23 00:02 ubuntu_pkg_info/
drwxrwxr-x 2 root st2packs 4096 Feb 16 21:30 windows/
drwxrwxr-x 3 root st2packs 4096 Feb 23 00:02 workflows/
packs/examples2/aliases:
total 8
drwxrwxr-x 2 root st2packs 4096 Feb 16 21:30 ./
drwxrwxr-x 9 root st2packs 4096 Feb 23 00:02 ../
packs/examples2/lib:
total 8
drwxrwxr-x 2 root st2packs 4096 Feb 16 21:30 ./
drwxrwxr-x 9 root st2packs 4096 Feb 23 00:02 ../
packs/examples2/rules:
total 8
drwxrwxr-x 2 root st2packs 4096 Feb 16 21:30 ./
drwxrwxr-x 9 root st2packs 4096 Feb 23 00:02 ../
packs/examples2/sensors:
total 8
drwxrwxr-x 2 root st2packs 4096 Feb 16 21:30 ./
drwxrwxr-x 9 root st2packs 4096 Feb 23 00:02 ../
packs/examples2/tests:
total 8
drwxrwxr-x 2 root st2packs 4096 Feb 16 21:30 ./
drwxrwxr-x 9 root st2packs 4096 Feb 23 00:02 ../
packs/examples2/triggers:
total 8
drwxrwxr-x 2 root st2packs 4096 Feb 16 21:30 ./
drwxrwxr-x 9 root st2packs 4096 Feb 23 00:02 ../
root@42118eabf091:/opt/stackstorm# ll packs/examples2/actions/*
packs/examples2/actions/bash_exit_code:
total 8
drwxrwxr-x 2 root st2packs 4096 Feb 16 21:30 ./
drwxrwxr-x 10 root st2packs 4096 Feb 23 00:02 ../
packs/examples2/actions/bash_ping:
total 8
drwxrwxr-x 2 root st2packs 4096 Feb 16 21:30 ./
drwxrwxr-x 10 root st2packs 4096 Feb 23 00:02 ../
packs/examples2/actions/bash_random:
total 8
drwxrwxr-x 2 root st2packs 4096 Feb 16 21:30 ./
drwxrwxr-x 10 root st2packs 4096 Feb 23 00:02 ../
packs/examples2/actions/chains:
total 8
drwxrwxr-x 2 root st2packs 4096 Feb 16 21:30 ./
drwxrwxr-x 10 root st2packs 4096 Feb 23 00:02 ../
packs/examples2/actions/pythonactions:
total 8
drwxrwxr-x 2 root st2packs 4096 Feb 16 21:30 ./
drwxrwxr-x 10 root st2packs 4096 Feb 23 00:02 ../
packs/examples2/actions/ubuntu_pkg_info:
total 12
drwxrwxr-x 3 root st2packs 4096 Feb 23 00:02 ./
drwxrwxr-x 10 root st2packs 4096 Feb 23 00:02 ../
drwxrwxr-x 2 root st2packs 4096 Feb 16 21:30 lib/
packs/examples2/actions/windows:
total 8
drwxrwxr-x 2 root st2packs 4096 Feb 16 21:30 ./
drwxrwxr-x 10 root st2packs 4096 Feb 23 00:02 ../
packs/examples2/actions/workflows:
total 12
drwxrwxr-x 3 root st2packs 4096 Feb 23 00:02 ./
drwxrwxr-x 10 root st2packs 4096 Feb 23 00:02 ../
drwxrwxr-x 2 root st2packs 4096 Feb 16 21:30 tests/
root@42118eabf091:/opt/stackstorm# ll packs/examples2/actions/*/*
packs/examples2/actions/ubuntu_pkg_info/lib:
total 8
drwxrwxr-x 2 root st2packs 4096 Feb 16 21:30 ./
drwxrwxr-x 3 root st2packs 4096 Feb 23 00:02 ../
packs/examples2/actions/workflows/tests:
total 8
drwxrwxr-x 2 root st2packs 4096 Feb 16 21:30 ./
drwxrwxr-x 3 root st2packs 4096 Feb 23 00:02 ../
Upgrade on CentOS 7 went fine except for the issue(s) reported by @amanda11
I ran into another issue during the upgrade von Ubuntu 16.04 with
--u16-add-insecure-py3-ppa
:20210223T193442+0000 The repository is setup! You can now install packages. 20210223T193442+0000 Reading package lists... 20210223T193442+0000 Building dependency tree... 20210223T193442+0000 Reading state information... 20210223T193442+0000 Some packages could not be installed. This may mean that you have 20210223T193442+0000 requested an impossible situation or if you are using the unstable 20210223T193442+0000 distribution that some required packages have not yet been created 20210223T193442+0000 or been moved out of Incoming. 20210223T193442+0000 The following information may help to resolve the situation: 20210223T193442+0000 20210223T193442+0000 The following packages have unmet dependencies: 20210223T193442+0000 st2 : PreDepends: python3.6 (>= 3.6) but it is not installable 20210223T193442+0000 Depends: python3.6-dev but it is not installable 20210223T193442+0000 E: Unable to correct problems, you have held broken packages. 20210223T193442+0000 ############### ERROR ############### 20210223T193442+0000 # Failed on Install st2 # 20210223T193442+0000 #####################################
And a second execution of
curl -sSL https://stackstorm.com/packages/install.sh | bash -s -- --user=st2admin --password=Ch@ngeMe --unstable --u16-add-insecure-py3-ppa
fails due to the already installed mongodb instance that was created during the first try - incl. the users and databases:20210223T193946+0000 2021-02-23T19:39:46.361+0000 E QUERY [js] Error: couldn't add user: command createUser requires authentication : 20210223T193946+0000 _getErrorWithCode@src/mongo/shell/utils.js:25:13 20210223T193946+0000 DB.prototype.createUser@src/mongo/shell/db.js:1514:15 20210223T193946+0000 @(shell):1:1 20210223T193946+0000 bye 20210223T193946+0000 ############### ERROR ############### 20210223T193946+0000 # Failed on Install st2 dependencies (MongoDB) # 20210223T193946+0000 #####################################
Digging into it now. It looks like the check if Python3.6 is available or not has some caveats and the ppa was not added.
I have a fix @winem for this - https://github.com/StackStorm/st2-packages/pull/691
I was looking into it as the E2E tests started failing...
Updated the helm chart https://github.com/StackStorm/stackstorm-ha/pull/182. Incorporated the suggested updates. Have this up and running in my K8s cluster with RBAC and LDAP enabled, So far everything looks good. Just noted some modifications to be made to the cli. Will submit a separate PR for that. Using Ubuntu 18.04 for the K8s cluster. Kubernetes and Ansible pack fail to install with the error: “error: command 'x86_64-linux-gnu-gcc' failed: No such file or directory”
Not sure if this needs to handled at the base image level. fixed it at our end by creating our custom st2actionrunner image but it might make sense to package this dependency in the base image as these are very commonly used packs and pretty there would be other packs which would need this dependency.
FROM stackstorm/st2actionrunner:3.4dev
ARG APP_VERSION=0.4
RUN sudo apt-get update RUN apt-get install -y gcc libkrb5-dev
Regards Harsh Nanchahal
On Tue, Feb 23, 2021 at 2:48 PM Amanda McGuinness notifications@github.com wrote:
Upgrade on CentOS 7 went fine except for the issue(s) reported by @amanda11 https://github.com/amanda11
I ran into another issue during the upgrade von Ubuntu 16.04 with --u16-add-insecure-py3-ppa:
20210223T193442+0000 The repository is setup! You can now install packages. 20210223T193442+0000 Reading package lists... 20210223T193442+0000 Building dependency tree... 20210223T193442+0000 Reading state information... 20210223T193442+0000 Some packages could not be installed. This may mean that you have 20210223T193442+0000 requested an impossible situation or if you are using the unstable 20210223T193442+0000 distribution that some required packages have not yet been created 20210223T193442+0000 or been moved out of Incoming. 20210223T193442+0000 The following information may help to resolve the situation: 20210223T193442+0000 20210223T193442+0000 The following packages have unmet dependencies: 20210223T193442+0000 st2 : PreDepends: python3.6 (>= 3.6) but it is not installable 20210223T193442+0000 Depends: python3.6-dev but it is not installable 20210223T193442+0000 E: Unable to correct problems, you have held broken packages. 20210223T193442+0000 ############### ERROR ############### 20210223T193442+0000 # Failed on Install st2 # 20210223T193442+0000 #####################################
And a second execution of curl -sSL https://stackstorm.com/packages/install.sh | bash -s -- --user=st2admin --password=Ch@ngeMe --unstable --u16-add-insecure-py3-ppa fails due to the already installed mongodb instance that was created during the first try - incl. the users and databases:
20210223T193946+0000 2021-02-23T19:39:46.361+0000 E QUERY [js] Error: couldn't add user: command createUser requires authentication : 20210223T193946+0000 _getErrorWithCode@src/mongo/shell/utils.js:25:13 20210223T193946+0000 DB.prototype.createUser@src/mongo/shell/db.js:1514:15 20210223T193946+0000 @(shell):1:1 20210223T193946+0000 bye 20210223T193946+0000 ############### ERROR ############### 20210223T193946+0000 # Failed on Install st2 dependencies (MongoDB) # 20210223T193946+0000 #####################################
Digging into it now. It looks like the check if Python3.6 is available or not has some caveats and the ppa was not added.
I have a fix @winem https://github.com/winem for this - StackStorm/st2-packages#691 https://github.com/StackStorm/st2-packages/pull/691
I was looking into it as the E2E tests started failing...
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/StackStorm/discussions/issues/66#issuecomment-784568780, or unsubscribe https://github.com/notifications/unsubscribe-auth/AD2PWBSGGXDE44SPM2MJGA3TAQWFNANCNFSM4X5BVMYQ .
Using Ubuntu 18.04 for the K8s cluster. Kubernetes and Ansible pack fail to install with the error: “error: command 'x86_64-linux-gnu-gcc' failed: No such file or directory”
pip reaches for gcc only when it can't resolve a wheel to download so it needs to build a wheel for a package that is not pure-python. Test infra uses a newer version of pip than what we install in production. So we're unlikely to find such errors while testing (before we cut a release) because pip is new enough in testing.
Please, let's merge these PRs to sanitize/make-consistent the pip version we use everywhere. https://github.com/StackStorm/st2packaging-dockerfiles/pull/103 https://github.com/StackStorm/st2-packages/pull/687 https://github.com/StackStorm-Exchange/ci/pull/102 https://github.com/StackStorm/st2/pull/5165
Edited my last post to add one more PR.
This one resolves pip version issues as well: https://github.com/StackStorm/st2/pull/5165
Oddly enough, we have been using pip 21.0.1 during some of the testing because pip-tools
(testing-requirements.txt) was pulling in the newer version.
Thanks @hnanchahal for testing the RBAC/LDAP in K8s/HA environment. Appreciate the feedback and involvement :+1:
Not sure if this needs to handled at the base image level. fixed it at our end by creating our custom st2actionrunner image but it might make sense to package this dependency in the base image as these are very commonly used packs and pretty there would be other packs which would need this dependency.
That's right. Ansible pack per requirements (and some other packs) require additional OS-level dependencies like gcc
which we can't package in base Docker images and so we recommend doing it yourself in your infra, which you did and that worked.
@cognifloyd It has nothing to do with the pip
version and all those PRs are not bugfixes, but maintenance tasks that are not blockers/requirements for this release. The situation is that the st2+pip version we've packaged now is what we'll ship at this round, if the team won't discover serious blocker reasons last-minute. The master is freezed
. Switching the wheels at this moment means we have to restart the testing as those packages will be rebuilt in new environment and rely on new tools which is a risk and should be done slightly beforehand.
Thanks for your dedication and help! Can't wait once we'll get the v3.4.0
out and then on-board all the pip + performance improvements :100:
@cognifloyd It has nothing to do with the pip version and all those PRs are not bugfixes, but maintenance tasks that are not blockers/requirements for this release.
I disagree. If we explicitly pin one version of a package, but we test with a different version, then that is a bug.
As far as the ansible pack is concerned, that note in the readme is somewhat incorrect. gcc is only needed because the version of pip we're using in production is broken.
Ansible is a pure-python package. It depends on cryptography and PyYAML which have non-python components. Both cryptography and PyYAML provide wheels, but the old versions of pip can't use them so it tries to build its own wheel instead. If we used the same version of pip in production that we have been running unit/integration tests with, then this would be a non-issue.
Tested manual and ansible deployments on U16. All good. Not had chance to confirm upgrade on U16. Quick play in those environments with workflow designer, and the st2-self-check all ran successfully.
Some random feedback:
- The save button in the visualizer is easy to miss as it looks like its more of a navigation button. Maybe put a big button that says "Save"; same goes for the run button
- I know workflows have to have an action behind it, but it seems like it would be cleaner to maybe have the visualizer in its own tab?
- When hovering over the anchor point on a task object maybe have the icon change to a move icon.
- When closing the visualizer after making a change there should be a warning telling you that you haven't saved anything.
@minsis This is great feedback, would you mind (if you haven't already) create a feature request on the st2web repository for these. They wouldn't be regression problems, but good points to improve the UI in future releases.
I'm running into another issue with the upgrade from 3.3 to 3.4 on U16 now. It looks like an issue with stdin redirection again. It appears if the st2.conf was modified and does not match the one provided by st2:
20210227T212722+0000 Unpacking st2 (3.4dev-58) over (3.3.0-4) ...
20210227T212730+0000 Processing triggers for libc-bin (2.23-0ubuntu11.2) ...
20210227T212730+0000 Setting up libpython3.6:amd64 (3.6.13-1+xenial2) ...
20210227T212730+0000 Setting up libpython3.6-dev:amd64 (3.6.13-1+xenial2) ...
20210227T212730+0000 Setting up python3.6-dev (3.6.13-1+xenial2) ...
20210227T212730+0000 Setting up st2 (3.4dev-58) ...
20210227T212730+0000 Installing new version of config file /etc/logrotate.d/st2 ...
20210227T212730+0000
20210227T212730+0000 Configuration file '/etc/st2/st2.conf'
20210227T212730+0000 ==> Modified (by you or by a script) since installation.
20210227T212730+0000 ==> Package distributor has shipped an updated version.
20210227T212730+0000 What would you like to do about it ? Your options are:
20210227T212730+0000 Y or I : install the package maintainer's version
20210227T212730+0000 N or O : keep your currently-installed version
20210227T212730+0000 D : show the differences between the versions
20210227T212730+0000 Z : start a shell to examine the situation
20210227T212730+0000 The default action is to keep your current version.
20210227T212730+0000 *** st2.conf (Y/I/N/O/D/Z) [default=N] ? dpkg: error processing package st2 (--configure):
20210227T212730+0000 end of file on stdin at conffile prompt
20210227T212730+0000 Processing triggers for libc-bin (2.23-0ubuntu11.2) ...
20210227T212730+0000 Errors were encountered while processing:
20210227T212730+0000 st2
20210227T212731+0000 E: Sub-process /usr/bin/dpkg returned an error code (1)
20210227T212731+0000 ############### ERROR ###############
20210227T212731+0000 # Failed on Install st2 #
20210227T212731+0000 #####################################
So it does not wait for any user input and fails instead immediately. This seems to be the same as: https://github.com/StackStorm/st2-packages/issues/688
The upgrade seems to work fine when I revert all changes on st2.conf and run the curl-bash-command. At least the self-verification was successful. Still have to do some more tests.
It looks like we can avoid the stdin issue as described above and in https://github.com/StackStorm/st2-packages/issues/688 by using the following command:
bash <(curl -sSL https://stackstorm.com/packages/install.sh) --user=st2admin --password=Ch@ngeMe --unstable --u16-add-insecure-py3-ppa
instead of the original one:
curl -sSL https://stackstorm.com/packages/install.sh | bash -s -- --user=st2admin --password=Ch@ngeMe --unstable
The one I suggested does not have any issues with interactions and does wait for the users input. I tested all available options Y/I, N/O, D and Z successfully.
I do not see any additional issue or (security) risk by changing the commands but am happy about additional views on this suggestion.
That enhancement would be helpful (please submit a PR), but we don't support upgrading st2 versions via curl|bash
installer as it doesn't know about the migration path, nor idempotent for that operation.
The user needs to follow the https://docs.stackstorm.com/latest/install/upgrades.html#general-upgrade-procedure instead.
PR provided. And yes, it's just for testing/development purpose. :+1:
I'm done with the tests on different installations. Let me just summarize the tests here for a better overview:
Testing means:
Tested Versions:
~I just found some (minor?) issues:~ ~1. After the manual upgrade on U16 st2resultstracker was still running. The running service was not stopped by the script. The service was disabled and did not start automatically again after a reboot. st2resultstracker was stopped during and no longer running after the upgrade via ansible-st2, so this is just an issue with the bash script.~ ~2. The FlowUI works fine for viewing existing Orquesta workflows but you can't save any changes due to a permission issue. Example: "Error saving workflow.: Unable to write data to "/opt/stackstorm/packs/packs/actions/workflows/install.yaml" (permission denied). Make sure permissions for that pack directory are configured correctly so st2api can write to it." Creating new workflows and updating them works fine. This issue is not limited to Ubuntu 16.04 and was also reproducable on 18.04.~
~So, is 1) an actual issue? Do we expect the same issue when st2 3.4 will be installed via package managers like apt, yum, etc? Do we want to add a note to the migration docs?~ ~Is 2) known & expected and I just missed it in the documentation? I couldn't find a corresponding issue. I'll create one tomorrow unless we figure this out earlier than that here.~
- After the manual upgrade on U16 st2resultstracker was still running. The running service was not stopped by the script. The service was disabled and did not start automatically again after a reboot. st2resultstracker was stopped during and no longer running after the upgrade via ansible-st2, so this is just an issue with the bash script.
@winem - Did you follow the manual upgrade instructions? On CentOS 7 when following the manual upgrade instructions there is no problem. As the first instruction is to stop all ST2 services, therefore st2resultstracker is stopped at that point, and then never restarted when the ST2 services is restarted. If you used the bash installer for upgrade, then that isn't the documented upgrade procedure - so I think that isn't relevant.
2. The FlowUI works fine for viewing existing Orquesta workflows but you can't save any changes due to a permission issue. Example: "Error saving workflow.: Unable to write data to "/opt/stackstorm/packs/packs/actions/workflows/install.yaml" (permission denied). Make sure permissions for that pack directory are configured correctly so st2api can write to it." Creating new workflows and updating them works fine. This issue is not limited to Ubuntu 16.04 and was also reproducable on 18.04.
I've just installed a pack on a CentOS 7 upgraded to 3.4, and I can edit the workflows. But I do need to make sure it has fully loaded everything, so when I click on action I can see that the action has loaded etc. I can't save packs.install workflow but I think that might be due to: https://github.com/StackStorm/st2web/pull/859 or https://github.com/StackStorm/st2web/pull/858. (And was same on EWC). So there are some fields I think workflow composer can't handle yet. (which we should probably document).
I installed one of my own packs after upgrade, and could edit its workflows fine. And I can't see any difference in ownership of files.
@winem Is it possible to test on a workflow other than the packs one, and see if problem exists (one that doesn't have the fields that those two pull requests above mention).
Successfully performed U16 upgrade, using the new st2ctl reload register-recreate-virtualenvs. After upgrade, installed a pack of mine that had an Orquesta workflow and was able to edit it.
I couldn't necessarily find a suitable place to document the problems about editing workflows with the designer that had retry or delay parameters.... There are pull requests in progress for those, so could just create an issue in st2web and link the in-progress PRs to that?
@winem Is it possible to test on a workflow other than the packs one, and see if problem exists (one that doesn't have the fields that those two pull requests above mention).
The error I saw was not related to these properties. But I was able to confirm that there is a very specific error message for these cases which is good!
To make it short: Both of my the 2 issues are obsolete now.
The issue saving the workflow was caused by a permission issue. Investigation has shown that files of packs that were installed during the self-verification are owned by root:root while those of packs installed via the UI or st2 pack install
are owned by root:st2packs. The FlowUI process running under the st2 user does not have the required write permission on those workflows installed during the self-verification.
@amanda11 to me, it looks like we're fine with testing CentOS7 and U16 now. Do you see further test cases? If yes, just let me know and I'll proceed with them.
@amanda11 The task delay and retry PRs for st2web are StackStorm/st2web#858 and StackStorm/st2web#859, respectively. They will likely go into v3.5. 😕
@winem I think we've hit everything we need to, unless I'm missing something.
The only thing of concern to me, is whether the issues mentioned on docker are blocking or not. @armab Do you have a view on that?
Neither myself or @blag are that familiar on the docker use-cases.
Other than that I think we've tested all the scenarios, and fixed the regressions....
@amanda11 The task delay and retry PRs for st2web are StackStorm/st2web#858 and StackStorm/st2web#859, respectively. They will likely go into v3.5. 😕
Yeah - I see they were in progress. And those problems existed in EWC - so agree not blocking for 3.4 release.
Thanks! There is no blocker from the Docker side for starting a release.
In fact, all the deployment methods (Puppet, Ansible, Docker, K8s, Vagrant) really come as a secondary comparing to st2
core, unless there's some terribly wrong that's failing CI and needs fixing.
So we're good from that standpoint :+1:
Ok I'm going to try and explain this the best I can here, sorry if it seems like ramblings.
Something happened with my docker setup when I enabled RBAC with LDAP. I'm still trying to troubleshoot why this is happening but one of the containers is continually trying to authenticate my user and basically locked out my account. My password was reset by our IT team but my account is continually locked out (probably due to caching) and I had to disable LDAP to get this to stop.
Here's the events in order:
So I'm not exactly sure where or why this happened. I looked at the logs of all the other containers and none of them were trying to auth my account. It was solely the auth container that was trying to auth my account every 5 seconds.
So I have LDAP disabled and then reenabled RBAC only and its now failing again due to invalid login. So something has to be cached in the database and even with LDAP disabled its still trying to reauth me to the LDAP backend
[auth]
mode = standalone
backend = ldap
backend_kwargs = <--some config stuff-->
enable = False
use_ssl = False
logging = /etc/st2/logging.auth.gunicorn.conf
debug = True
[rbac]
enable = True
backend = default
I'm guessing even though enabled = False, its still trying to auth to ldap since backend is still set to ldap. Other than this issue I'm not even sure how to clear the cache to stop it from trying to auth with my old LDAP login.
Thanks, @minsis. Looks like you already documented the Feature request/Issue in the respective Docker repository: https://github.com/StackStorm/st2-docker/issues/219. There is a somewhat similar WIP PR contribution in the K8s from the Community: https://github.com/StackStorm/stackstorm-ha/pull/182 Though both are not tied to any release.
Because Docker and K8s might need proper implementation in the future, keeping in mind their multi-container specifics, I'd recommend testing LDAP/RBAC with the VM at this moment, following documentation:
The st-docker#219 issue is just for getting a volume added to store the rbac role files.
The current issue here is that something has broken with LADP and RBAC to where its locking out LDAP accounts because it tries to reauth every 5s (which I also put in an issue on that repo) on a cached credential. I'm just not sure where to go from here in terms of troubleshooting. Every time I turn on LDAP/RBAC it tries to auth my old credentials that are cached somewhere and I can't seem to clear them out.
I would love to test directly on a VM, but at work I have limited resources and it takes our provisioning department weeks to get something built out.
You can spin up a VM locally with st2vagrant and configure it that way.
Since Saturday install on EL8 failing. Looks to be that we are picking up a new version of rabbitmq, which requires a new version of erlang, and that version of erlang is not available in CentOS or the epel repos. https://github.com/rabbitmq/rabbitmq-server/releases/tag/v3.8.13
Also affecting old releases, unless pin rabbitmq version...
I upgraded our dev box.
no module named redis
stack trace. pip install redis intructionsFWIW, I am on Centos 7 which now uses python3. I fired off a few large workflows and they all seem to be working as expected.
Probably worth mentioning that if on a distro with python2.7 as the default that all sensor packs will need to be changed to python version 3 in the pack.yaml and reinstalled
We're ready to prepare the StackStorm
v3.4
release and start pre-release testing..Release Process Preparation
Per Release Management Schedule @blag is the Release Manager and @amanda11 assisting for v3.4. They will freeze the
master
for the major repositories in StackStorm org, follow the StackStorm Release Process which is now available to public, accompanied by the Useful Info for Release managers. Communication is happening in#releasemgmt
and#development
Slack channels. The first step is pre-release manual user-acceptance testing forv3.4dev
.Why Manual testing?
StackStorm is very serious about testing and has a lot of it: Unit tests, Integration, Deployment/Integrity checks, Smoke tests and eventually end-2-end tests when automation spins up new AWS instance for each OS/flavor we support, installs real st2 like user would and runs set of st2tests (for each st2 PR, nightly, periodically, during release).
That's a perfect way to verify what we already know and codify expectations about how StackStorm should function.
However it's not enough. There are always new unknowns to discover, edge cases to experience and tests to add. Hence, manual Exploratory Testing is an exercise where entire team gathers together and starts trying (or breaking) new features before the new release. Because we're all different, perceive software differently and try different things we might find new bugs, improper design, oversights, edge cases and more.
This is how StackStorm previously managed to land less major/critical bugs into production.
TL;DR
Install StackStorm
v3.4dev
unstable packages, try random things in random environments (different OS) and report any regressions found comparing tov3.3
(and/orv3.2
if you are still running ST2 Enterprise):You can also run StackStorm on a specific OS with st2vagrant:
Extra points for PR hotfixes and adding new or missing test cases.
Specific things to check
st2 --version
actions/lib
, please ensure that those actions still work. In the Python 2.7 removal, we adjusted how we handle this code, so we want to ensure that we do not have any regressions.actions/lib
issue, we haven't troubleshooted this enough to be sure).!help
(with st2chatops/hubot-stackstorm) or!st2help
(with err-stackstorm) commands work. See StackStorm/st2-rbac-backend#47 for more information.If you have successful test results, please post a summary of what all you tested (OSes, what features you tested).
If you run into any bugs, please open them in the respective repositories and link to this issue from there. I will add them to the list at the bottom of this description.
If you have any issues running StackStorm or running the tests, please post down below.
Major changes
timeout
parameter topacks.install
st2 action-alias execute
commandFull Changelog
Changes which are recommended to ack, explore, check and try in a random way.
st2
Added
Added support for GitLab SSH URLs on pack install and download actions. (improvement) #5050 Contributed by @asthLucas
Added st2-rbac-backend pip requirements for RBAC integration. (new feature) #5086 Contributed by @hnanchahal
Added notification support for err-stackstorm. (new feature) #5051
Added st2-auth-ldap pip requirements for LDAP auth integartion. (new feature) #5082 Contributed by @hnanchahal
Changed
Updated deprecation warning for python 2 pack installs, following python 2 support removal. #5099 Contributed by @amanda11
Improve the st2-self-check script to echo to stderr and exit if it isn't run with a ST2_AUTH_TOKEN or ST2_API_KEY environment variable. (improvement) #5068
Added timeout parameter for packs.install action to help with long running installs that exceed the default timeout of 600 sec which is defined by the python_script action runner (improvement) #5084
Contributed by @hnanchahal
Upgraded cryptography version to 3.2 to avoid CVE-2020-25659 (security) #5095
Converted most CI jobs from Travis to GitHub Actions (all except Integration tests).
Contributed by @nmaludy, @winem, and @blag
Updated cryptography dependency to version 3.3.2 to avoid CVE-2020-36242 (security) #5151
Fixed
Pin chardet version as newest version was incompatible with pinned requests version #5101 Contributed by @amanda11
Fixed issue were st2tests was not getting installed using pip because no version was specified. Contributed by @anirudhbagri
Added monkey patch fix to st2stream to enable it to work with mongodb via SSL. (bug fix) #5078 #5091
Fix nginx buffering long polling stream to client. Instead of waiting for closed connection wait for final event to be sent to client. (bug fix) #4842 #5042
Contributed by @guzzijones
StackStorm now explicitly decodes pack files as utf-8 instead of implicitly as ascii (bug fix)
5106, #5107
Fix incorrect array parameter value casting when executing action via chatops or using
POST /aliasexecution/match_and_execute
API endpoint. The code would incorrectly assume the value is always a string, but that may not be the cast - they value could already be a list and in this case we don't want any casting to be performed. (bug fix) #5141Contributed by @Kami.
Fix
@parameter_name=/path/to/file/foo.json
notation in thest2 run
command which didn't work correctly because it didn't convert read bytes to string / unicode type. (bug fix) #5140Contributed by @Kami.
Fix broken
st2 action-alias execute
command and make sure it works correctly. (bug fix) #5138Contributed by @Kami.
Removed
Removed --python3 pack install option #5100 Contributed by @amanda11
Removed submit-debug-info tool and the st2debug component #5103
Removed check-licence script (cleanup) #5092
Contributed by @kroustou
Updated Makefile and CI to use Python 3 only, removing Python 2 (cleanup) #5090
Contributed by @blag
Remove st2resultstracker from st2ctl, the development environment and the st2actions setup.py (cleanup) #5108
Contributed by @winem
orquesta 1.3.0
Added
Changed
Fixed
st2chatops
No changes
StackStorm Exchange
Conclusion
Please report findings here and bugs/regressions in respective repositories. Depending on severity and importance bugs might be fixed before the release or postponed to the next release if they're very minor and not a release blocker.
Issues Found During Release
PRs Merged for Release
TODOs