StackStorm (aka "IFTTT for Ops") is event-driven automation for auto-remediation, incident responses, troubleshooting, deployments, and more for DevOps and SREs. Includes rules engine, workflow, 160 integration packs with 6000+ actions (see https://exchange.stackstorm.org) and ChatOps. Installer at https://docs.stackstorm.com/install/index.html
In our standard st2 virtualenv there are a variety of .pth files that add modules to the sys.path when python starts up (via the site module: https://docs.python.org/3/library/site.html):
Today, however, none of those .pth files affect pack venvs because the parent st2 venv is not in the standard list of site-packages locations.
We probably haven't encountered issues with the lack of .pth file loading in pack venvs because oslo.config and repoze.lru are the only things that could have an impact, and the impact would be minimal at best.
This becomes more pronounced when using a virtualenv generated by pants+pex (instead of via our Makefile) because all of our first party sources get installed with PEP 660 wheels. The legacy calls to python setup.py develop trigger creating a symlink to st2common, st2auth, st2* and the runner directories under the virtualenv's site-packages directory. PEP 660 wheels use .pth files instead of symlinks to achieve the same thing.
So, we have to ensure .pth files get loaded whenever interacting with a pack venv. This PR does that by injecting a .pth file into the pack venv that uses the site module to load any .pth files from the parent st2 venv.
Also, I adjusted the GHA CI workflow to work when our sources are added to the venv via PEP 660 editable wheels. Apparently, PEP 660 only installs entry point scripts, but most of our st2*/bin/* scripts are just scripts not entry_points. https://peps.python.org/pep-0660/#limitations
Finally, I adjusted how st2-run-pack-tests so that it can handle hermetic scripts. A hermetic script has a python shebang with the -sE args that do not load user site-packages (-s) and ignore all PYTHON* vars (-E) like PYTHONPATH. Once we update to a version of pants that includes https://github.com/pantsbuild/pants/pull/20763, we can just disable how pants+pex changes the virtualenv shebang lines. Until then, we need to run nosetests in a way that lets PYTHONPATH continue to affect it.
I extracted this from #6130 where I was testing what changes are required to use a pants-built venv to run tests.
In our standard st2 virtualenv there are a variety of
.pthfiles that add modules to thesys.pathwhen python starts up (via thesitemodule: https://docs.python.org/3/library/site.html):Today, however, none of those
.pthfiles affect pack venvs because the parent st2 venv is not in the standard list of site-packages locations.We probably haven't encountered issues with the lack of
.pthfile loading in pack venvs becauseoslo.configandrepoze.lruare the only things that could have an impact, and the impact would be minimal at best.This becomes more pronounced when using a virtualenv generated by pants+pex (instead of via our Makefile) because all of our first party sources get installed with PEP 660 wheels. The legacy calls to
python setup.py developtrigger creating a symlink to st2common, st2auth, st2* and the runner directories under the virtualenv's site-packages directory. PEP 660 wheels use.pthfiles instead of symlinks to achieve the same thing.So, we have to ensure
.pthfiles get loaded whenever interacting with a pack venv. This PR does that by injecting a.pthfile into the pack venv that uses thesitemodule to load any.pthfiles from the parent st2 venv.Also, I adjusted the GHA CI workflow to work when our sources are added to the venv via PEP 660 editable wheels. Apparently, PEP 660 only installs entry point scripts, but most of our
st2*/bin/*scripts are justscriptsnotentry_points. https://peps.python.org/pep-0660/#limitationsFinally, I adjusted how
st2-run-pack-testsso that it can handle hermetic scripts. A hermetic script has a python shebang with the-sEargs that do not load user site-packages (-s) and ignore allPYTHON*vars (-E) likePYTHONPATH. Once we update to a version of pants that includes https://github.com/pantsbuild/pants/pull/20763, we can just disable how pants+pex changes the virtualenv shebang lines. Until then, we need to runnosetestsin a way that letsPYTHONPATHcontinue to affect it.I extracted this from #6130 where I was testing what changes are required to use a pants-built venv to run tests.