search

StackStorm-Exchange / stackstorm-kubernetes

st2 content pack containing Kubernetes sensors
https://exchange.stackstorm.org/
Apache License 2.0
19 stars 17 forks source link

Installation fails: Failed to build http-parser #47

Closed javierruizjimenez closed 2 years ago

javierruizjimenez commented 2 years ago

Trying to install st2 kubernetes pack.

Environment:

I have been able to install the Terraform pack successfully

But I can't install the kubernetes pack, it fails with the name and also with the githib repo:

st2 pack install kubernetes
st2 pack install https://github.com/StackStorm-Exchange/stackstorm-kubernetes

Error (full is below):

Failed to build http-parser
Installing collected packages: pycparser, PyYAML, cffi, validate-email, pyaml, MarkupSafe, cryptography, pyswagger, pyopenssl, jinja2, http-parser, backports.ssl
    Running setup.py install for http-parser: started
    Running setup.py install for http-parser: finished with status 'error'
 (stderr:   ERROR: Command errored out with exit status 1:
   command: /opt/stackstorm/virtualenvs/kubernetes/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-x8deomew/http-parser_c54c856d3d4e49dfa9384758a074c458/setup.py'"'"'; __file__='"'"'/tmp\pip-install-x8deomew/http-parser_c54c856d3d4e49dfa9384758a074c458/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\
'"'"', '"'"'\
'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-hjnao655
       cwd: /tmp/pip-install-x8deomew/http-parser_c54c856d3d4e49dfa9384758a074c458/
  Complete output (19 lines):

Full error:

root@6dabf9353482:/opt/stackstorm# st2 pack install kubernetes

For the "kubernetes" pack, the following content will be registered:

actions   |  366
rules     |  2
sensors   |  33
aliases   |  0
triggers  |  0

Installation may take a while for packs with many items.

        [ succeeded ] init_task
        [ succeeded ] download_pack
        [ succeeded ] make_a_prerun
        [ succeeded ] get_pack_dependencies
        [ succeeded ] check_dependency_and_conflict_list
        [  failed   ] install_pack_requirements

id: 626ee0bea88a5e5f10a80dd3
action.ref: packs.install
parameters: 
  packs:
  - kubernetes
status: failed
start_timestamp: Sun, 01 May 2022 19:34:22 UTC
end_timestamp: Sun, 01 May 2022 19:34:44 UTC
log: 
  - status: requested
    timestamp: '2022-05-01T19:34:22.716000Z'
  - status: scheduled
    timestamp: '2022-05-01T19:34:22.907000Z'
  - status: running
    timestamp: '2022-05-01T19:34:23.363000Z'
  - status: failed
    timestamp: '2022-05-01T19:34:44.268000Z'
result: 
  errors:
  - message: Execution failed. See result for details.
    result:
      exit_code: 1
      result: None
      stderr: "st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Setting up virtualenv for pack "kubernetes" (/opt/stackstorm/packs/kubernetes)
st2.actions.python.SetupVirtualEnvironmentAction: INFO     Virtualenv path "/opt/stackstorm/virtualenvs/kubernetes" doesn't exist
st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Creating virtualenv for pack "kubernetes" in "/opt/stackstorm/virtualenvs/kubernetes"
st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Creating virtualenv in "/opt/stackstorm/virtualenvs/kubernetes" using Python binary "/opt/stackstorm/st2/bin/python"
st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Running command "/opt/stackstorm/st2/bin/virtualenv -p /opt/stackstorm/st2/bin/python --always-copy --verbose --no-download /opt/stackstorm/virtualenvs/kubernetes" to create virtualenv.
st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Installing base requirements
st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Installing requirement six>=1.9.0,<2.0 with command /opt/stackstorm/virtualenvs/kubernetes/bin/pip install six>=1.9.0,<2.0.
st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Installing pack specific requirements from "/opt/stackstorm/packs/kubernetes/requirements.txt"
st2.actions.python.SetupVirtualEnvironmentAction: DEBUG    Installing requirements from file /opt/stackstorm/packs/kubernetes/requirements.txt with command /opt/stackstorm/virtualenvs/kubernetes/bin/pip install -U -r /opt/stackstorm/packs/kubernetes/requirements.txt.
Traceback (most recent call last):
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/python_runner/python_action_wrapper.py", line 395, in <module>
    obj.run()
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/python_runner/python_action_wrapper.py", line 214, in run
    output = action.run(**self._parameters)
  File "/opt/stackstorm/packs/packs/actions/pack_mgmt/setup_virtualenv.py", line 98, in run
    no_download=no_download,
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/util/virtualenvs.py", line 131, in setup_pack_virtualenv
    logger=logger,
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/util/virtualenvs.py", line 297, in install_requirements
    % (requirements_file_path, stdout, stderr)
Exception: Failed to install requirements from "/opt/stackstorm/packs/kubernetes/requirements.txt": Collecting backports.ssl==0.0.9
  Downloading backports.ssl-0.0.9.tar.gz (11 kB)
Collecting pyopenssl==17.5.0
  Downloading pyOpenSSL-17.5.0-py2.py3-none-any.whl (53 kB)
Requirement already satisfied: six>=1.5.2 in ./virtualenvs/kubernetes/lib/python3.6/site-packages (from pyopenssl==17.5.0->-r /opt/stackstorm/packs/kubernetes/requirements.txt (line 5)) (1.16.0)
Collecting cryptography>=2.1.4
  Downloading cryptography-37.0.1-cp36-abi3-manylinux_2_24_x86_64.whl (4.0 MB)
Collecting cffi>=1.12
  Downloading cffi-1.15.0-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.whl (405 kB)
Collecting http_parser
  Downloading http-parser-0.9.0.tar.gz (167 kB)
Collecting jinja2
  Downloading Jinja2-3.0.3-py3-none-any.whl (133 kB)
Collecting MarkupSafe>=2.0
  Downloading MarkupSafe-2.0.1-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (30 kB)
Collecting pyswagger
  Downloading pyswagger-0.8.39-py3-none-any.whl (78 kB)
Collecting pyaml>=15.03.1
  Downloading pyaml-21.10.1-py2.py3-none-any.whl (24 kB)
Collecting pycparser
  Downloading pycparser-2.21-py2.py3-none-any.whl (118 kB)
Collecting PyYAML
  Downloading PyYAML-6.0-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (603 kB)
Collecting validate-email
  Downloading validate_email-1.3.tar.gz (4.7 kB)
Building wheels for collected packages: backports.ssl, http-parser, validate-email
  Building wheel for backports.ssl (setup.py): started
  Building wheel for backports.ssl (setup.py): finished with status 'done'
  Created wheel for backports.ssl: filename=backports.ssl-0.0.9-py3-none-any.whl size=12430 sha256=a488d815bddf8586b0fa5b36683e40c1a4e5c9e26811952c9981f65cbcf20329
  Stored in directory: /root/.cache/pip/wheels/92/bc/e1/ec0f4a69e827f0df3beb6603593bdf83a4058626fe17f2794c
  Building wheel for http-parser (setup.py): started
  Building wheel for http-parser (setup.py): finished with status 'error'
  Running setup.py clean for http-parser
  Building wheel for validate-email (setup.py): started
  Building wheel for validate-email (setup.py): finished with status 'done'
  Created wheel for validate-email: filename=validate_email-1.3-py3-none-any.whl size=5461 sha256=32805077de227af799885a2f31bfec9d1330cb07aed99fffd087bdf002d35802
  Stored in directory: /root/.cache/pip/wheels/f1/59/bd/ab906153650f2141bc12f08130d78341f3cdaa8abed92b422d
Successfully built backports.ssl validate-email
Failed to build http-parser
Installing collected packages: pycparser, PyYAML, cffi, validate-email, pyaml, MarkupSafe, cryptography, pyswagger, pyopenssl, jinja2, http-parser, backports.ssl
    Running setup.py install for http-parser: started
    Running setup.py install for http-parser: finished with status 'error'
 (stderr:   ERROR: Command errored out with exit status 1:
   command: /opt/stackstorm/virtualenvs/kubernetes/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-x8deomew/http-parser_c54c856d3d4e49dfa9384758a074c458/setup.py'"'"'; __file__='"'"'/tmp\pip-install-x8deomew/http-parser_c54c856d3d4e49dfa9384758a074c458/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\
'"'"', '"'"'\
'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-hjnao655
       cwd: /tmp/pip-install-x8deomew/http-parser_c54c856d3d4e49dfa9384758a074c458/
  Complete output (19 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.6
  creating build/lib.linux-x86_64-3.6/http_parser
  copying http_parser/_socketio.py -> build/lib.linux-x86_64-3.6/http_parser
  copying http_parser/util.py -> build/lib.linux-x86_64-3.6/http_parser
  copying http_parser/http.py -> build/lib.linux-x86_64-3.6/http_parser
  copying http_parser/__init__.py -> build/lib.linux-x86_64-3.6/http_parser
  copying http_parser/pyparser.py -> build/lib.linux-x86_64-3.6/http_parser
  copying http_parser/reader.py -> build/lib.linux-x86_64-3.6/http_parser
  running build_ext
  building 'http_parser.parser' extension
  creating build/temp.linux-x86_64-3.6
  creating build/temp.linux-x86_64-3.6/http_parser
  x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -Iparser -I/opt/stackstorm/virtualenvs/kubernetes/include -I/usr/include/python3.6m -c http_parser/http_parser.c -o build/temp.linux-x86_64-3.6/http_parser/http_parser.o
  unable to execute 'x86_64-linux-gnu-gcc': No such file or directory
  error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for http-parser
    ERROR: Command errored out with exit status 1:
     command: /opt/stackstorm/virtualenvs/kubernetes/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-x8deomew/http-parser_c54c856d3d4e49dfa9384758a074c458/setup.py'"'"'; __file__='"'"'/t\p/pip-install-x8deomew/http-parser_c54c856d3d4e49dfa9384758a074c458/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\
'"'"', '"'"'\
'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-ffu9aluy/install-record.txt --single-version-externally-managed --compile --install-headers /opt/stackstorm/virtualenvs/kubernetes/include/site/python3.6/http-parser
         cwd: /tmp/pip-install-x8deomew/http-parser_c54c856d3d4e49dfa9384758a074c458/
    Complete output (19 lines):
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.6
    creating build/lib.linux-x86_64-3.6/http_parser
    copying http_parser/_socketio.py -> build/lib.linux-x86_64-3.6/http_parser
    copying http_parser/util.py -> build/lib.linux-x86_64-3.6/http_parser
    copying http_parser/http.py -> build/lib.linux-x86_64-3.6/http_parser
    copying http_parser/__init__.py -> build/lib.linux-x86_64-3.6/http_parser
    copying http_parser/pyparser.py -> build/lib.linux-x86_64-3.6/http_parser
    copying http_parser/reader.py -> build/lib.linux-x86_64-3.6/http_parser
    running build_ext
    building 'http_parser.parser' extension
    creating build/temp.linux-x86_64-3.6
    creating build/temp.linux-x86_64-3.6/http_parser
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -Iparser -I/opt/stackstorm/virtualenvs/kubernetes/include -I/usr/include/python3.6m -c http_parser/http_parser.c -o build/temp.linux-x86_64-3.6/http_parser/http_parser.o
    unable to execute 'x86_64-linux-gnu-gcc': No such file or directory
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /opt/stackstorm/virtualenvs/kubernetes/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-x8deomew/http-parser_c54c856d3d4e49dfa9384758a074c45\/setup.py'"'"'; __file__='"'"'/tmp/pip-install-x8deomew/http-parser_c54c856d3d4e49dfa9384758a074c458/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\
'"'"', '"'"'\
'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-ffu9aluy/install-record.txt --single-version-externally-managed --compile --install-headers /opt/stackstorm/virtualenvs/kubernetes/include/site/python3.6/http-parser Check the logs for full command output.
WARNING: You are using pip version 20.3.3; however, version 21.3.1 is available.
You should consider upgrading via the '/opt/stackstorm/virtualenvs/kubernetes/bin/python -m pip install --upgrade pip' command.
)
"
      stdout: ''
    task_id: install_pack_requirements
    type: error
  output:
    conflict_list: []
    message: ''
    packs_list:
    - kubernetes
    warning_list: null
+--------------------------+------------------------+------------------------------------+-----------------------------+-------------------------------+
| id                       | status                 | task                               | action                      | start_timestamp               |
+--------------------------+------------------------+------------------------------------+-----------------------------+-------------------------------+
| 626ee0c0cf84daeaf792f93d | succeeded (0s elapsed) | init_task                          | core.noop                   | Sun, 01 May 2022 19:34:24 UTC |
| 626ee0c1cf84daeaf792f94d | succeeded (2s elapsed) | download_pack                      | packs.download              | Sun, 01 May 2022 19:34:25 UTC |
| 626ee0c3cf84daeaf792f95d | succeeded (1s elapsed) | make_a_prerun                      | packs.virtualenv_prerun     | Sun, 01 May 2022 19:34:27 UTC |
| 626ee0c4cf84daeaf792f96d | succeeded (1s elapsed) | get_pack_dependencies              | packs.get_pack_dependencies | Sun, 01 May 2022 19:34:28 UTC |
| 626ee0c6cf84daeaf792f97d | succeeded (0s elapsed) | check_dependency_and_conflict_list | core.noop                   | Sun, 01 May 2022 19:34:30 UTC |
| 626ee0c6cf84daeaf792f98d | failed (13s elapsed)   | install_pack_requirements          | packs.setup_virtualenv      | Sun, 01 May 2022 19:34:30 UTC |
+--------------------------+------------------------+------------------------------------+-----------------------------+-------------------------------+

Please let me know if I can provide more info. Thanks

arm4b commented 2 years ago

unable to execute 'x86_64-linux-gnu-gcc': No such file or directory error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

Try to install gcc OS package in your system. Looks like one of the pip dependencies is trying to compile some stuff.

javierruizjimenez commented 2 years ago

Thanks @armab

I did apt update and apt install gcc.

It is now available at the system (in st2client):

# x86_64-linux-gnu-gcc -v
Using built-in specs.
COLLECT_GCC=x86_64-linux-gnu-gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/7/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 7.5.0-3ubuntu1~18.04' --with-bugurl=file:///usr/share/doc/gcc-7/README.Bugs --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++ --prefix=/usr --with-gcc-major-version-only --program-suffix=-7 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-bootstrap --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --enable-plugin --enable-default-pie --with-system-zlib --with-target-system-zlib --enable-objc-gc=auto --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04)

Installation still fails with the same error, somehow x86_64-linux-gnu-gcc is not available in the python env:

    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -Iparser -I/opt/stackstorm/virtualenvs/kubernetes/include -I/usr/include/python3.6m -c http_parser/http_parser.c -o build/temp.linux-x86_64-3.6/http_parser/http_parser.o
    unable to execute 'x86_64-linux-gnu-gcc': No such file or directory
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

Installing other dev libraries didn't solve the issue either.

arm4b commented 2 years ago

Are you using HA-based deployment? You need to make sure gcc is available for the st2actionrunners.

javierruizjimenez commented 2 years ago

Are you using HA-based deployment? You need to make sure gcc is available for the st2actionrunners.

No, the Docker Compose official installation

Thanks for your help. I have been able to "almost install" the pack after installing GCC in the st2actionrunner container. It still fails at register_pack for other reasons (probably it is not able to access the Kubernetes API).

I will close the issue.

INSTALL GCC in st2actionrunner

docker compose exec st2actionrunner bash
apt update
apt install gcc

INSTALL Kubernete pack in st2client

docker compose exec st2client bash
st2ctl reload --register-all
st2 pack install kubernetes
arm4b commented 2 years ago

Thanks for confirming.

I'd recommend opening a PR to document the OS package requirements in the pack README https://github.com/StackStorm-Exchange/stackstorm-kubernetes/blob/master/README.md

Ansible pack requirements (https://github.com/StackStorm-Exchange/stackstorm-ansible/#requirements) might be an example.

That will surely help someone in the future.