Closed blag closed 3 years ago
@blag Either way, if this is going in independantly of the pip upgrade we need to remove the call to the legacyresolver as it's failing CircleCI. Did we want to make this PR off master branch rather than the one we're using for pip upgrade?
Also don't we end up with virtualenv being 16.6.0 as that's the version that is specifed in the st2 packages.
When I was testing on CentOS and updated the virtualenv version I found that the version of virtualenv in /opt/stackstorm/st2/bin/virtualenv was the version specified by the st2/fixed-reuquirements. This is different to the virtualenv version on the system. So I'm not sure if this is required.
Can we check if we end up with the right version in /opt/stacktorm/st2/bin/virtualenv. I think perhaps this is just the venv
From what I saw in the packaging tests for Ubuntu Bionic, the virtualenv module just shells out to the system's virtualenv binary to create a virtualenv.
I believe that Python 3's venv
module is different, and actually creates the virtualenv directly. I have not verified this though.
From when I tested on a real system then I thought we ended up with the virtualenv module from st2 fixed-requirements get placed into /opt/stackstorm/st2/bin/virtualenv and then that was what was used to create the package's virtualenv environments. (Note you have to look as the tests are running as it seems to clean up afterwards...) @armab may know more - as I have only found these things from trial and error.
So something weird is going on...
st2.actions.python.SetupVirtualEnvironmentAction: DEBUG Creating virtualenv for pack "hubot" in "/opt/stackstorm/virtualenvs/hubot"
Traceback (most recent call last):
File "/opt/stackstorm/st2/lib/python3.6/site-packages/python_runner/python_action_wrapper.py", line 334, in <module>
obj.run()
File "/opt/stackstorm/st2/lib/python3.6/site-packages/python_runner/python_action_wrapper.py", line 193, in run
output = action.run(**self._parameters)
File "/opt/stackstorm/packs/packs/actions/pack_mgmt/setup_virtualenv.py", line 89, in run
no_download=no_download)
File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/util/virtualenvs.py", line 94, in setup_pack_virtualenv
no_download=no_download)
File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/util/virtualenvs.py", line 154, in create_virtualenv
raise Exception('Virtualenv binary "%s" doesn\\'t exist.' % (virtualenv_binary))
Exception: Virtualenv binary "/usr/bin/virtualenv" doesn't exist.
So can confirm there is a problem but it shouldn't need to be resolved this way I think...
I think the problem on the PRs is that for some reason /opt/stackstorm/st2/bin/python is a link to /usr/bin/python3.6 - and that's why its trying to find /usr/bin/virtualenv. In the master branch the xenial /opt/stackstorm/st2/bin/python is a link to the python3.6 in the /opt/stackstorm/st2/bin directory.
st2common works out the path to virtualenv based on taking the path of sys.executable, and as it's linked to /usr/bin that's the problem.
So problem in the PRs we are working on to up the virtualenv, rather than on master...
This PR is a nasty hack to get the packaging tests to pass - adding python-virtualenv as a dependency to ensure that the virtualenv binary is available to StackStorm when it goes to create a virtualenv when it's installing a non-system pack.
This is a nasty, nasty hack, because the version of virtualenv that it installs is 15.0.1 on Ubuntu Xenial and 15.1.0 [look closely at those version strings, they are different], but only 15.1.0 officially supports Python 3.6 virtualenvs. While this "works" and passes tests, there is no guarantee that it will work in production for ST2 v3.4 on Ubuntu Xenial, so I opened this in a separate PR so we can decide if we are okay with merging this hack.
The proper fix is to refactor StackStorm to use Python 3's built-in
venv
module to create pack virtualenvs. Once that is done, then the python-virtualenv package dependency should be removed. I have not investigated this yet, but we may need to install the python3-venv package to ensure that StackStorm has thevenv
module available. I'm not sure that that module is available on PyPI/installable via pip. More investigation is needed.