duo-labs / cloudmapper

CloudMapper helps you analyze your Amazon Web Services (AWS) environments.
BSD 3-Clause "New" or "Revised" License
5.9k stars 800 forks source link

Cloudmapper fails to install on macos #843

Open serialp opened 3 years ago

serialp commented 3 years ago

Hello, whatever the solution I came up with to setup cloudmapper, it fails at the pyjq install...

pip3 install --user pipenv
cd cloudmapper/; pipenv install --skip-lock

is it related to the below issue ?

https://github.com/doloopwhile/pyjq/issues/54 https://bugs.python.org/issue42504

Please find hereunder my logs:

Installing dependencies from Pipfile... An error occurred while installing pyjq==2.3.1! Will try again. An error occurred while installing seaborn==0.10.1! Will try again. 🐍 ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 37/37 — 00:00:44 Installing initially failed dependencies... InstallError: File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/cli/command.py", line 233, in install InstallError: retcode = do_install( InstallError: File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/core.py", line 2052, in do_install

InstallError: File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/core.py", line 1304, in do_init

InstallError: File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/core.py", line 899, in do_install_dependencies

InstallError: File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/core.py", line 796, in batch_install InstallError: _cleanup_procs(procs, failed_deps_queue, retry=retry) InstallError: File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/core.py", line 703, in _cleanup_procs InstallError: raise exceptions.InstallError(c.dep.name, extra=err_lines) pipenv.exceptions.InstallError: Collecting pyjq==2.3.1 pipenv.exceptions.InstallError: Using cached pyjq-2.3.1.tar.gz (2.1 MB) pipenv.exceptions.InstallError: Requirement already satisfied: six in /Users/hiyaa/.local/share/virtualenvs/cloudmapper-WzNltrOW/lib/python3.8/site-packages (from pyjq==2.3.1->-r /var/folders/26/pmmcdlhx727ffbxy2ld7m86h0000gn/T/pipenv-8bez_ooa-requirements/pipenv-r8czubvu-requirement.txt (line 1)) (1.14.0) pipenv.exceptions.InstallError: Building wheels for collected packages: pyjq pipenv.exceptions.InstallError: Building wheel for pyjq (setup.py): started pipenv.exceptions.InstallError: Building wheel for pyjq (setup.py): finished with status 'error' pipenv.exceptions.InstallError: ERROR: Command errored out with exit status 1: pipenv.exceptions.InstallError: command: /Users/hiyaa/.local/share/virtualenvs/cloudmapper-WzNltrOW/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/26/pmmcdlhx727ffbxy2ld7m86h0000gn/T/pip-install-7m4e96ks/pyjq_abce72f03b2c422d9a98ef71102e0034/setup.py'"'"'; file='"'"'/private/var/folders/26/pmmcdlhx727ffbxy2ld7m86h0000gn/T/pip-install-7m4e96ks/pyjq_abce72f03b2c422d9a98ef71102e0034/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' bdist_wheel -d /private/var/folders/26/pmmcdlhx727ffbxy2ld7m86h0000gn/T/pip-wheel-dbvkaevy pipenv.exceptions.InstallError: cwd: /private/var/folders/26/pmmcdlhx727ffbxy2ld7m86h0000gn/T/pip-install-7m4e96ks/pyjq_abce72f03b2c422d9a98ef71102e0034/ pipenv.exceptions.InstallError: Complete output (44 lines): pipenv.exceptions.InstallError: running bdist_wheel pipenv.exceptions.InstallError: running build pipenv.exceptions.InstallError: running build_py pipenv.exceptions.InstallError: creating build pipenv.exceptions.InstallError: creating build/lib.macosx-11-x86_64-3.8 pipenv.exceptions.InstallError: copying pyjq.py -> build/lib.macosx-11-x86_64-3.8 pipenv.exceptions.InstallError: warning: build_py: byte-compiling is disabled, skipping. pipenv.exceptions.InstallError: pipenv.exceptions.InstallError: running build_ext pipenv.exceptions.InstallError: Traceback (most recent call last): pipenv.exceptions.InstallError: File "", line 1, in pipenv.exceptions.InstallError: File "/private/var/folders/26/pmmcdlhx727ffbxy2ld7m86h0000gn/T/pip-install-7m4e96ks/pyjq_abce72f03b2c422d9a98ef71102e0034/setup.py", line 93, in

pipenv.exceptions.InstallError: File "/Users/hiyaa/.local/share/virtualenvs/cloudmapper-WzNltrOW/lib/python3.8/site-packages/setuptools/init.py", line 153, in setup pipenv.exceptions.InstallError: return distutils.core.setup(**attrs) pipenv.exceptions.InstallError: File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/core.py", line 148, in setup

pipenv.exceptions.InstallError: File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/dist.py", line 966, in run_commands

pipenv.exceptions.InstallError: File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/dist.py", line 985, in run_command

pipenv.exceptions.InstallError: File "/Users/hiyaa/.local/share/virtualenvs/cloudmapper-WzNltrOW/lib/python3.8/site-packages/wheel/bdist_wheel.py", line 299, in run

pipenv.exceptions.InstallError: File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/cmd.py", line 313, in run_command

pipenv.exceptions.InstallError: File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/dist.py", line 985, in run_command

pipenv.exceptions.InstallError: File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/command/build.py", line 135, in run

pipenv.exceptions.InstallError: File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/cmd.py", line 313, in run_command

pipenv.exceptions.InstallError: File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/dist.py", line 985, in run_command

pipenv.exceptions.InstallError: File "/private/var/folders/26/pmmcdlhx727ffbxy2ld7m86h0000gn/T/pip-install-7m4e96ks/pyjq_abce72f03b2c422d9a98ef71102e0034/setup.py", line 30, in run

pipenv.exceptions.InstallError: File "/private/var/folders/26/pmmcdlhx727ffbxy2ld7m86h0000gn/T/pip-install-7m4e96ks/pyjq_abce72f03b2c422d9a98ef71102e0034/setup.py", line 39, in _build_oniguruma

pipenv.exceptions.InstallError: File "/private/var/folders/26/pmmcdlhx727ffbxy2ld7m86h0000gn/T/pip-install-7m4e96ks/pyjq_abce72f03b2c422d9a98ef71102e0034/setup.py", line 67, in _build_lib pipenv.exceptions.InstallError: os.environ['MACOSX_DEPLOYMENT_TARGET'] = macosx_deployment_target pipenv.exceptions.InstallError: File "/usr/local/opt/python@3.8/bin/../Frameworks/Python.framework/Versions/3.8/lib/python3.8/os.py", line 680, in setitem pipenv.exceptions.InstallError: value = self.encodevalue(value) pipenv.exceptions.InstallError: File "/usr/local/opt/python@3.8/bin/../Frameworks/Python.framework/Versions/3.8/lib/python3.8/os.py", line 750, in encode pipenv.exceptions.InstallError: raise TypeError("str expected, not %s" % type(value).name) pipenv.exceptions.InstallError: TypeError: str expected, not int

pipenv.exceptions.InstallError: ERROR: Failed building wheel for pyjq pipenv.exceptions.InstallError: Running setup.py clean for pyjq pipenv.exceptions.InstallError: Failed to build pyjq pipenv.exceptions.InstallError: Installing collected packages: pyjq pipenv.exceptions.InstallError: Running setup.py install for pyjq: started pipenv.exceptions.InstallError: Running setup.py install for pyjq: finished with status 'error' pipenv.exceptions.InstallError: ERROR: Command errored out with exit status 1: pipenv.exceptions.InstallError: command: /Users/hiyaa/.local/share/virtualenvs/cloudmapper-WzNltrOW/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/26/pmmcdlhx727ffbxy2ld7m86h0000gn/T/pip-install-7m4e96ks/pyjq_abce72f03b2c422d9a98ef71102e0034/setup.py'"'"'; file='"'"'/private/var/folders/26/pmmcdlhx727ffbxy2ld7m86h0000gn/T/pip-install-7m4e96ks/pyjq_abce72f03b2c422d9a98ef71102e0034/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /private/var/folders/26/pmmcdlhx727ffbxy2ld7m86h0000gn/T/pip-record-1eezjp50/install-record.txt --single-version-externally-managed --compile --install-headers /Users/hiyaa/.local/share/virtualenvs/cloudmapper-WzNltrOW/include/site/python3.8/pyjq pipenv.exceptions.InstallError: cwd: /private/var/folders/26/pmmcdlhx727ffbxy2ld7m86h0000gn/T/pip-install-7m4e96ks/pyjq_abce72f03b2c422d9a98ef71102e0034/ pipenv.exceptions.InstallError: Complete output (46 lines): pipenv.exceptions.InstallError: running install pipenv.exceptions.InstallError: running build pipenv.exceptions.InstallError: running build_py pipenv.exceptions.InstallError: creating build pipenv.exceptions.InstallError: creating build/lib.macosx-11-x86_64-3.8 pipenv.exceptions.InstallError: copying pyjq.py -> build/lib.macosx-11-x86_64-3.8 pipenv.exceptions.InstallError: warning: build_py: byte-compiling is disabled, skipping. pipenv.exceptions.InstallError: pipenv.exceptions.InstallError: running build_ext pipenv.exceptions.InstallError: Traceback (most recent call last): pipenv.exceptions.InstallError: File "", line 1, in pipenv.exceptions.InstallError: File "/private/var/folders/26/pmmcdlhx727ffbxy2ld7m86h0000gn/T/pip-install-7m4e96ks/pyjq_abce72f03b2c422d9a98ef71102e0034/setup.py", line 93, in

pipenv.exceptions.InstallError: File "/Users/hiyaa/.local/share/virtualenvs/cloudmapper-WzNltrOW/lib/python3.8/site-packages/setuptools/init.py", line 153, in setup pipenv.exceptions.InstallError: return distutils.core.setup(**attrs) pipenv.exceptions.InstallError: File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/core.py", line 148, in setup

pipenv.exceptions.InstallError: File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/dist.py", line 966, in run_commands

pipenv.exceptions.InstallError: File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/dist.py", line 985, in run_command

pipenv.exceptions.InstallError: File "/Users/hiyaa/.local/share/virtualenvs/cloudmapper-WzNltrOW/lib/python3.8/site-packages/setuptools/command/install.py", line 61, in run pipenv.exceptions.InstallError: return orig.install.run(self) pipenv.exceptions.InstallError: File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/command/install.py", line 545, in run

pipenv.exceptions.InstallError: File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/cmd.py", line 313, in run_command

pipenv.exceptions.InstallError: File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/dist.py", line 985, in run_command

pipenv.exceptions.InstallError: File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/command/build.py", line 135, in run

pipenv.exceptions.InstallError: File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/cmd.py", line 313, in run_command

pipenv.exceptions.InstallError: File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/dist.py", line 985, in run_command

pipenv.exceptions.InstallError: File "/private/var/folders/26/pmmcdlhx727ffbxy2ld7m86h0000gn/T/pip-install-7m4e96ks/pyjq_abce72f03b2c422d9a98ef71102e0034/setup.py", line 30, in run

pipenv.exceptions.InstallError: File "/private/var/folders/26/pmmcdlhx727ffbxy2ld7m86h0000gn/T/pip-install-7m4e96ks/pyjq_abce72f03b2c422d9a98ef71102e0034/setup.py", line 39, in _build_oniguruma

pipenv.exceptions.InstallError: File "/private/var/folders/26/pmmcdlhx727ffbxy2ld7m86h0000gn/T/pip-install-7m4e96ks/pyjq_abce72f03b2c422d9a98ef71102e0034/setup.py", line 67, in _build_lib pipenv.exceptions.InstallError: os.environ['MACOSX_DEPLOYMENT_TARGET'] = macosx_deployment_target pipenv.exceptions.InstallError: File "/usr/local/opt/python@3.8/bin/../Frameworks/Python.framework/Versions/3.8/lib/python3.8/os.py", line 680, in setitem pipenv.exceptions.InstallError: value = self.encodevalue(value) pipenv.exceptions.InstallError: File "/usr/local/opt/python@3.8/bin/../Frameworks/Python.framework/Versions/3.8/lib/python3.8/os.py", line 750, in encode pipenv.exceptions.InstallError: raise TypeError("str expected, not %s" % type(value).name) pipenv.exceptions.InstallError: TypeError: str expected, not int

pipenv.exceptions.InstallError: ERROR: Command errored out with exit status 1: /Users/hiyaa/.local/share/virtualenvs/cloudmapper-WzNltrOW/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/26/pmmcdlhx727ffbxy2ld7m86h0000gn/T/pip-install-7m4e96ks/pyjq_abce72f03b2c422d9a98ef71102e0034/setup.py'"'"'; file='"'"'/private/var/folders/26/pmmcdlhx727ffbxy2ld7m86h0000gn/T/pip-install-7m4e96ks/pyjq_abce72f03b2c422d9a98ef71102e0034/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /private/var/folders/26/pmmcdlhx727ffbxy2ld7m86h0000gn/T/pip-record-1eezjp50/install-record.txt --single-version-externally-managed --compile --install-headers /Users/hiyaa/.local/share/virtualenvs/cloudmapper-WzNltrOW/include/site/python3.8/pyjq Check the logs for full command output. ERROR: Couldn't install package: pyjq Package installation failed... ☤ ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 1/2 —

Many thanks !

w0rmr1d3r commented 3 years ago

It happened the same to me (same error), I tried to install the requirements using: pip install -r requirements.txt and with: pip3 install -r requirements.txt And it fails with the very same error.

Could it be due to higher versions of Python than 3.7 ? I was using 3.9 and @serialp here was using 3.8.

0xdabbad00 commented 3 years ago

Is this on macOS Big Sur? Could be this Python issue: https://bugs.python.org/issue42504

I installed CloudMapper without issues on an Ubuntu system last week.

w0rmr1d3r commented 3 years ago

Is this on macOS Big Sur? Could be this Python issue: https://bugs.python.org/issue42504

I installed CloudMapper without issues on an Ubuntu system last week.

Yes, in my case it is. macOS Big Sur 11.1 I'll wait for that to get fixed! Thanks @0xdabbad00 ! Good catch!

serialp commented 3 years ago

@w0rmr1d3r I used the same install method as you then switch to pipenv I also tried with each version of python util 3.9. That's why I have mentioned this issue to @0xdabbad00 and yes Scott I am using MacOs big sur 11.2.2 which is why I mentioned the related issue if it can help. For linux have you tried on Ubuntu 20.04 LTS ? So will you able to deliver a fix on your side or pyjq will handle it ?

Thanks again for your feedback !

w0rmr1d3r commented 3 years ago

@w0rmr1d3r I used the same install method as you then switch to pipenv I also tried with each version of python util 3.9. That's why I have mentioned this issue to @0xdabbad00 and yes Scott I am using MacOs big sur 11.2.2 which is why I mentioned the related issue if it can help. For linux have you tried on Ubuntu 20.04 LTS ? So will you able to deliver a fix on your side or pyjq will handle it ?

Thanks again for your feedback !

Sorry @serialp , I went too fast on your logs and didn't see the links to the very same issue @0xdabbad00 said. I belive that, regarding to Python issues, we might need to wait for pyjq to handle it.

w0rmr1d3r commented 3 years ago

@serialp the python issue has been resolved! It should work now, have you tried it again?

serialp commented 3 years ago

@w0rmr1d3r Many thank, but finally went to the docker version, which works fine. We'll try to install it on my MacOs next time i need it !

cvdv87 commented 3 years ago

I had the same issue on mac os, removing the version numbers from the file fixed it for me, pip sorted out everything.

w0rmr1d3r commented 3 years ago

@cvdv87 which file? The requirements.txt file?

cvdv87 commented 3 years ago

@w0rmr1d3r correct, my file looks like this: astroid autoflake autopep8 boto3 botocore certifi chardet coverage docutils idna isort Jinja2 jmespath lazy-object-proxy MarkupSafe matplotlib mccabe mock netaddr nose pandas parliament policyuniverse pycodestyle pyflakes pyjq pylint python-dateutil PyYAML requests s3transfer seaborn six toml typed-ast urllib3 wrapt

I have to say, I had another dependency issue, different packages, but perhaps this works for you too.

w0rmr1d3r commented 3 years ago

There are a lot of issues in this repo regarding versions and some incompatibilities (and some PRs opened as well). We should take a look at those too! Question: If we delete fixed versions, are we installing latest versions for all of those dependencies? Does Cloudmapper work as usual?

Which dependency issue you say you were having?

cvdv87 commented 3 years ago

So here is my approach, keep in mind I use a newer python version as advised in the installation page: Python version 3.8

1)- Clone repo, install pre-reqs, start venv 2)- Update pip: pip install --upgrade pip 3)- pip install -r requirements.txt

Error: ERROR: Cannot install -r requirements.txt (line 5) and urllib3==1.26.5 because these package versions have conflicting dependencies.

The conflict is caused by: The user requested urllib3==1.26.5 botocore 1.17.6 depends on urllib3<1.26 and >=1.20; python_version != "3.4"

Solution provided by pip: To fix this you could try to:

  1. loosen the range of package versions you've specified
  2. remove package versions to allow pip attempt to solve the dependency conflict

Solution 2 is what I've tested and afaik everything seems to be working fine for me.

w0rmr1d3r commented 3 years ago

That happens to you the same way it happened to other people, because urllib3 got updated and it conflicted with botocore versions and other dependencies. Exists this PR which fixes the issue that you have encountered -> https://github.com/duo-labs/cloudmapper/pull/876 Which, at the same time, it fixes this issue -> https://github.com/duo-labs/cloudmapper/issues/875

That's why I was saying that there are issues and PRs opened regarding dependencies and problems installing them.

Sean12697 commented 3 years ago

Sounds similar to #301 (I've just opened #881, based on Debian)

w0rmr1d3r commented 3 years ago

I can see in https://github.com/duo-labs/cloudmapper/issues/881 that you installed dependencies without a fixed version, which is similar to other PRs that basically upgrade dependencies, am I wrong?

Thanks @Sean12697 !

Sean12697 commented 3 years ago

You're not wrong @w0rmr1d3r, I did manually implement #876 (which I see you reviewed a couple of weeks before I opened my issue), although it was running sudo apt install autoconf libtool from doloopwhile/pyjq#50 which finally fixed the issue for me.

Ideally I would reset my environment to test how much of an impact manually implementing #876 has before installed the additional apt packages.

ishtylerc commented 11 months ago

@w0rmr1d3r correct, my file looks like this: astroid autoflake autopep8 boto3 botocore certifi chardet coverage docutils idna isort Jinja2 jmespath lazy-object-proxy MarkupSafe matplotlib mccabe mock netaddr nose pandas parliament policyuniverse pycodestyle pyflakes pyjq pylint python-dateutil PyYAML requests s3transfer seaborn six toml typed-ast urllib3 wrapt

I have to say, I had another dependency issue, different packages, but perhaps this works for you too.

This is what fixed my issue, removing the verision numbers

danieltanfh95 commented 6 months ago

issues fixed by updating typed-ast==1.5.5 and pyjq==2.6.0