search

duo-labs / cloudmapper

CloudMapper helps you analyze your Amazon Web Services (AWS) environments.
BSD 3-Clause "New" or "Revised" License
5.9k stars 797 forks source link

Fix requierments.txt file #889

Open cgaudit opened 2 years ago

cgaudit commented 2 years ago

Hi All, Does someone has a fix file of requierments.txt which works? BR

GastroGee commented 2 years ago

I switched requests to 2.25.0 and urllib3 to 1.25.6

cgaudit commented 2 years ago

Hi @GastroGee Thank's I change it but pyjq is still erroring... https://github.com/duo-labs/cloudmapper/issues/888#issuecomment-900388571

OlaSegha commented 2 years ago

Ah i see, so i was building a docker image and here is some adjustment with my dockerfile Note, i installed pyjq with my dockerfile ...

# Dockerize https://github.com/duo-labs/cloudmapper
# This project pulls Cloudmapper and initializes it in a docker container.

FROM python:3.7-slim-buster

EXPOSE 8000
WORKDIR /opt/cloudmapper

## basic installs
RUN apt-get update
RUN apt-get install autoconf automake build-essential libtool python3.7-dev python3-tk jq awscli -y
RUN apt-get install git jq -y

## Clone cloudmapper from duo release page
RUN git clone https://github.com/duo-labs/cloudmapper.git .

## The default requirements file has lots of dependency issues, removing it. 
RUN rm requirements.txt 

## Passing in our own tested requirement file that works 
COPY requirements.txt ./requirements.txt

## Install pip and others 
RUN pip install --upgrade pip setuptools wheel
RUN pip install PEP517
RUN pip install pipenv && pipenv clean && pipenv lock --clear && pipenv sync && pipenv install
RUN pip install --upgrade awscli
RUN pip install pyjq

VOLUME /opt/cloudmapper/web 
VOLUME /opt/cloudmapper/account-data 

COPY entrypoint.sh ./entrypoint.sh

# COPY .env ./.env

##Pass in my config file 
COPY config.json ./config.json

CMD ["bash","entrypoint.sh"]

and then my requirements file is this 

astroid==2.4.1
autoflake==1.3.1
autopep8==1.5.2
boto3==1.14.6
botocore==1.17.6
certifi==2020.4.5.1
chardet==3.0.4
coverage==5.1
docutils==0.15.2
idna==2.8
isort==4.3.21
Jinja2==2.11.3
jmespath==0.9.5
lazy-object-proxy==1.4.3
MarkupSafe==1.1.1
matplotlib==3.2.2
mccabe==0.6.1
mock==4.0.2
netaddr==0.7.19
nose==1.3.7
pandas==1.1.3
parliament==0.5.0
policyuniverse==1.3.2.20201103
pycodestyle==2.5.0
pyflakes==2.2.0
pyjq==2.3.1
pylint==2.5.2
python-dateutil==2.8.1
PyYAML==4.2b4
requests==2.25.0
s3transfer==0.3.0
seaborn==0.10.1
six==1.14.0
toml==0.10.0
typed-ast==1.4.1
urllib3==1.25.6
wrapt==1.12.1

hope that helps

w0rmr1d3r commented 2 years ago

Thank you @OlaSegha , can you open a PR with that fix so that we can all take a look?

Cheers!

cgaudit commented 2 years ago

Hi @OlaSegha, Thank's!

I tried to reinstalled it on my local Linux VM after I updated the requierments.txt file with you code and run the pip install pipenv && pipenv clean && pipenv lock --clear && pipenv sync && pipenv install

l still got the pyjq error:

An error occurred while installing pyjq==2.3.1 --hash=sha256:5f3cd45b856ae4b42a1058d2e91e13d84df5ea188aedc3ae397dc0e01c270db9 --hash=sha256:45f7688d8191a4043e7c4839d6cabff8f16abc98fe48c3a4a0cc2da05c62096d! Will try again.

Is it necessary to install it on a docker and how would you suggest to do it the quickest way?

Any Idea why?

@w0rmr1d3r did it worked for you? Is it install locally on your Linux?

BR

w0rmr1d3r commented 2 years ago

Hello @cgaudit ! I'm not running a Linux 😅 , however, since this project gave me a lot of issues installing it and the maintainers aren't reviewing the PRs with the fixes proposed, I switched to another tool which works.

This one -> https://github.com/toniblyx/prowler

See if it helps with what you want to scan or do.

Cheers!

oichokabu commented 2 years ago

Hi All,

I'm sorry for the similar question. The requirements.txt is not correct, could you please tell me the correct file or how to fix it?

I have run this command.

pip install -r requirements.txt

However, I encountered the following error.

ERROR: botocore 1.17.6 has requirement urllib3<1.26,>=1.20; python_version != "3.4", but you'll have urllib3 1.26.5 which is incompatible.
ERROR: requests 2.22.0 has requirement urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1, but you'll have urllib3 1.26.5 which is incompatible.

I followed @OlaSegha's advice to check requirements.txt but it did not lead me to a solution😢

Please help me from this error.

OlaSegha commented 2 years ago

@oicho014 @cgaudit

So i just attempted this on a linux VM, my virtual env is python3.7

First the requirements file is wrong, i removed pyjq 2.1.0 from the list .... and updated urllib3 to 1.25.6.

astroid==2.4.1
autoflake==1.3.1
autopep8==1.5.2
boto3==1.14.6
botocore==1.17.6
certifi==2020.4.5.1
chardet==3.0.4
coverage==5.1
docutils==0.15.2
idna==2.8
isort==4.3.21
Jinja2==2.11.3
jmespath==0.9.5
lazy-object-proxy==1.4.3
MarkupSafe==1.1.1
matplotlib==3.2.2
mccabe==0.6.1
mock==4.0.2
netaddr==0.7.19
nose==1.3.7
pandas==1.1.3
parliament==0.5.0
policyuniverse==1.3.2.20201103
pycodestyle==2.5.0
pyflakes==2.2.0
pyjq==2.3.1
pylint==2.5.2
python-dateutil==2.8.1
PyYAML==4.2b4
requests==2.25.0
s3transfer==0.3.0
seaborn==0.10.1
six==1.14.0
toml==0.10.0
typed-ast==1.4.1
urllib3==1.25.6
wrapt==1.12.1

after which i just ran 

pip install pyjq==2.1.0
oichokabu commented 2 years ago

I tried what you said, and the build went through. Now I can sleep soundly! Thanks, @OlaSegha🎉

cgaudit commented 2 years ago

Hi @OlaSegha

I tried to rerun it but unfortunately got an error:

ERROR: Failed building wheel for pyjq Running setup.py clean for pyjq Failed to build matplotlib pyjq Installing collected packages: botocore, s3transfer, PyYAML, pyflakes, pycodestyle, pandas, mccabe, matplotlib, isort, idna, chardet, certifi, boto3, astroid, typed-ast, seaborn, requests, pylint, pyjq, policyuniverse, parliament, nose, mock, coverage, autopep8, autoflake Attempting uninstall: botocore Found existing installation: botocore 1.21.6 Uninstalling botocore-1.21.6: Successfully uninstalled botocore-1.21.6 Attempting uninstall: s3transfer Found existing installation: s3transfer 0.5.0 Uninstalling s3transfer-0.5.0: Successfully uninstalled s3transfer-0.5.0 Attempting uninstall: PyYAML Found existing installation: PyYAML 5.3.1 ERROR: Cannot uninstall 'PyYAML'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.

I have python 3.92 installed.

Not sure why..

BR

OlaSegha commented 2 years ago

@cgaudit Well the toplevel README says that the installation is known to work with python3.7, so that might be contributing to the issue.

w0rmr1d3r commented 2 years ago

@cgaudit are you running on a Mac or Linux? I had problems with the Mac installation regarding Python 3.9

cgaudit commented 2 years ago

Hi @w0rmr1d3r sorry for the delay... I'm running on Linux

cgaudit commented 2 years ago

Hi I run it using ubuntu with 3.7 on the demo and it succeeded, thank's!

  1. Is it nessecery to add external CIDR names to the config.json file or it can be ignored? If not how can I get it?
  2. Should I truncate any existing folder or it will build new folder? BR
GastroGee commented 2 years ago

To question 1; It is not necessary... if u do not have external CIDRs then u dont have to add it to ur config.json. External CIDRs wld be CIDRs assigned to internal networks within ur organization like HQ CIDR, datacenter etc

cgaudit commented 2 years ago

It seems that there are external CIDR's. Do I need all IPv4 and their names? Is their names also required or it is sufficient to add only the IP's "1.1.1.1/32": {"name": "SF Office"} ?

Anyway...

  1. I don't have MFA
  2. Got error when running without CIDR's: python3 cloudmapper.py prepare --config config.json.xxx --account xxx Traceback (most recent call last): File "cloudmapper.py", line 72, in <module> main() File "cloudmapper.py", line 66, in main commands[command].run(arguments) File "/home/hunter/cloudmapper-main/commands/prepare.py", line 833, in run prepare(account, config, outputfilter) File "/home/hunter/cloudmapper-main/commands/prepare.py", line 662, in prepare cytoscape_json = build_data_structure(account, config, outputfilter) File "/home/hunter/cloudmapper-main/commands/prepare.py", line 420, in build_data_structure for region_json in get_regions(account, outputfilter): File "/home/hunter/cloudmapper-main/shared/common.py", line 161, in get_regions regions = pyjq.all(".Regions[]{}".format(region_filter), region_data) File "/home/hunter/cloudmapper-main/venv/lib/python3.8/site-packages/pyjq.py", line 49, in all return compile(script, vars, library_paths).all(_get_value(value, url, opener)) File "_pyjq.pyx", line 209, in _pyjq.Script.all (_pyjq.c:2561) _pyjq.ScriptRuntimeError: Cannot iterate over null (null)