search

StackStorm / st2

StackStorm (aka "IFTTT for Ops") is event-driven automation for auto-remediation, incident responses, troubleshooting, deployments, and more for DevOps and SREs. Includes rules engine, workflow, 160 integration packs with 6000+ actions (see https://exchange.stackstorm.org) and ChatOps. Installer at https://docs.stackstorm.com/install/index.html
https://stackstorm.com/
Apache License 2.0
6.07k stars 747 forks source link

Upgrade pip to latest 20.x version #5060

Closed dor-abu closed 3 years ago

dor-abu commented 4 years ago

SUMMARY

A few days ago packs that had aio-pika in requirements failed in "install pack dependencies" phase.

We found out that aio-pika uses yarl as a requirement without a fixed version. yarl pushed changes to master in recent days, and also uses multidict who also created changes in recent days.

We resolved the issue when we manually updated the version of pip to latest - 20.2.3.

STACKSTORM VERSION

st2 3.1.0, on Python 3.6.9

OS, environment, install method

We uses Docker and custom docker-compose.

Steps to reproduce the problem

Insert yarl, aio-pika, or multidict to requirements with pip version 19.1.1 in the virtual environment of the packages

Expected Results

We expect to pass the phase of "install pack dependencies".

Actual Results

This error will be raised:

Collecting yarl
  Using cached https://files.pythonhosted.org/packages/6d/0b/44006eb5b3b7898e54df4b7481839d22d15cf3fc4f0420394ad9e3bb214e/yarl-1.6.2.tar.gz
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Requirement already satisfied: idna>=2.0 in ./lib/python3.6/site-packages (from yarl) (2.10)
Collecting typing-extensions>=3.7.4; python_version < "3.8" (from yarl)
  Using cached https://files.pythonhosted.org/packages/60/7a/e881b5abb54db0e6e671ab088d079c57ce54e8a01a3ca443f561ccadb37e/typing_extensions-3.7.4.3-py3-none-any.whl
Collecting multidict>=4.0 (from yarl)
  Using cached https://files.pythonhosted.org/packages/d2/5a/e95b0f9ebacd42e094e229a9a0a9e44d02876abf64969d0cb07dadcf3c4a/multidict-5.0.0.tar.gz
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Building wheels for collected packages: yarl, multidict
  Building wheel for yarl (PEP 517) ... error
  ERROR: Complete output from command /opt/stackstorm/virtualenvs/ibm_xforce/bin/python /opt/stackstorm/virtualenvs/ibm_xforce/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py build_wheel /tmp/tmpgt60fpqm:
  ERROR: **********************
  * Accellerated build *
  **********************
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.6
  creating build/lib.linux-x86_64-3.6/yarl
  copying yarl/_quoting_py.py -> build/lib.linux-x86_64-3.6/yarl
  copying yarl/_quoting.py -> build/lib.linux-x86_64-3.6/yarl
  copying yarl/_url.py -> build/lib.linux-x86_64-3.6/yarl
  copying yarl/__init__.py -> build/lib.linux-x86_64-3.6/yarl
  running egg_info
  writing yarl.egg-info/PKG-INFO
  writing dependency_links to yarl.egg-info/dependency_links.txt
  writing requirements to yarl.egg-info/requires.txt
  writing top-level names to yarl.egg-info/top_level.txt
  reading manifest file 'yarl.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  warning: no previously-included files matching '*.pyc' found anywhere in distribution
  warning: no previously-included files matching '*.cache' found anywhere in distribution
  warning: no previously-included files found matching 'yarl/*.html'
  warning: no previously-included files found matching 'yarl/*.so'
  warning: no previously-included files found matching 'yarl/*.pyd'
  no previously-included directories found matching 'docs/_build'
  writing manifest file 'yarl.egg-info/SOURCES.txt'
  copying yarl/__init__.pyi -> build/lib.linux-x86_64-3.6/yarl
  copying yarl/_quoting_c.c -> build/lib.linux-x86_64-3.6/yarl
  copying yarl/_quoting_c.pyi -> build/lib.linux-x86_64-3.6/yarl
  copying yarl/_quoting_c.pyx -> build/lib.linux-x86_64-3.6/yarl
  copying yarl/py.typed -> build/lib.linux-x86_64-3.6/yarl
  running build_ext
  building 'yarl._quoting_c' extension
  creating build/temp.linux-x86_64-3.6
  creating build/temp.linux-x86_64-3.6/yarl
  x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.6m -I/opt/stackstorm/virtualenvs/ibm_xforce/include/python3.6m -c yarl/_quoting_c.c -o build/temp.linux-x86_64-3.6/yarl/_quoting_c.o
  error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for yarl
  Running setup.py clean for yarl
  Building wheel for multidict (PEP 517) ... error
  ERROR: Complete output from command /opt/stackstorm/virtualenvs/ibm_xforce/bin/python /opt/stackstorm/virtualenvs/ibm_xforce/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py build_wheel /tmp/tmpmr9a0erc:
  ERROR: **********************
  * Accellerated build *
  **********************
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.6
  creating build/lib.linux-x86_64-3.6/multidict
  copying multidict/_multidict_py.py -> build/lib.linux-x86_64-3.6/multidict
  copying multidict/_multidict_base.py -> build/lib.linux-x86_64-3.6/multidict
  copying multidict/_abc.py -> build/lib.linux-x86_64-3.6/multidict
  copying multidict/_compat.py -> build/lib.linux-x86_64-3.6/multidict
  copying multidict/__init__.py -> build/lib.linux-x86_64-3.6/multidict
  running egg_info
  writing multidict.egg-info/PKG-INFO
  writing dependency_links to multidict.egg-info/dependency_links.txt
  writing top-level names to multidict.egg-info/top_level.txt
  reading manifest file 'multidict.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  warning: no previously-included files matching '*.pyc' found anywhere in distribution
  warning: no previously-included files found matching 'multidict/_multidict.html'
  warning: no previously-included files found matching 'multidict/*.so'
  warning: no previously-included files found matching 'multidict/*.pyd'
  warning: no previously-included files found matching 'multidict/*.pyd'
  no previously-included directories found matching 'docs/_build'
  writing manifest file 'multidict.egg-info/SOURCES.txt'
  copying multidict/__init__.pyi -> build/lib.linux-x86_64-3.6/multidict
  copying multidict/_multidict.c -> build/lib.linux-x86_64-3.6/multidict
  copying multidict/py.typed -> build/lib.linux-x86_64-3.6/multidict
  creating build/lib.linux-x86_64-3.6/multidict/_multilib
  copying multidict/_multilib/defs.h -> build/lib.linux-x86_64-3.6/multidict/_multilib
  copying multidict/_multilib/dict.h -> build/lib.linux-x86_64-3.6/multidict/_multilib
  copying multidict/_multilib/istr.h -> build/lib.linux-x86_64-3.6/multidict/_multilib
  copying multidict/_multilib/iter.h -> build/lib.linux-x86_64-3.6/multidict/_multilib
  copying multidict/_multilib/pair_list.h -> build/lib.linux-x86_64-3.6/multidict/_multilib
  copying multidict/_multilib/views.h -> build/lib.linux-x86_64-3.6/multidict/_multilib
  running build_ext
  building 'multidict._multidict' extension
  creating build/temp.linux-x86_64-3.6
  creating build/temp.linux-x86_64-3.6/multidict
  x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.6m -I/opt/stackstorm/virtualenvs/ibm_xforce/include/python3.6m -c multidict/_multidict.c -o build/temp.linux-x86_64-3.6/multidict/_multidict.o -O2 -std=c99 -Wall -Wsign-compare -Wconversion -fno-strict-aliasing -pedantic
  error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for multidict
  Running setup.py clean for multidict
Failed to build yarl multidict
ERROR: Could not build wheels for yarl, multidict which use PEP 517 and cannot be installed directly
WARNING: You are using pip version 19.1.1, however version 20.2.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

image

arm4b commented 4 years ago

From what I understand the actionable here is to upgrade pip to latest version. I updated the Issue title to reflect it.

This is a st2 maintenance periodic task to update dependencies to latest versions and we'll try to use never pip in the next st2 release.

dor-abu commented 3 years ago

@armab As you mentioned in slack, we updated virtualenv to 20.1.0 in st2actionrunner container

It solves the issue. When a pack is installed, it is installed with pip 20.2.4.

We are still checking for side effects.

amanda11 commented 3 years ago

Resolved by by changes committed to master under PR #5123. and related PRs.

amanda11 commented 3 years ago

Closing as upgraded pip in v3.5.0.

rrahman-nv commented 3 years ago

how can we update pip for the kubernetes deployment of ST2?