x4v13r64
commented
4 years ago @paurisa @JJmako could you please review this? Generally these issues are FPs due to the local environment.
Closed berney closed 4 years ago
x4v13r64
commented
4 years ago @paurisa @JJmako could you please review this? Generally these issues are FPs due to the local environment.
prisas
commented
4 years ago I just created a new virtual environment and installed the required libraries with pip install -r requirements.txt. This command installed both, azure-cli-core version 2.1.0 and humanfriendly version 7.1.1 although I also get the incompatibility error. I am able to run scout successfully with both of them in their latest version.
Full console output:
$ python3 -m venv scout
$ source scout/bin/activate
$ pip install -r requirements.txt
<snippet>
azure-cli-core 2.1.0 has requirement humanfriendly~=4.7, but you'll have humanfriendly 7.1.1 which is incompatible.
<snippet>
$ pip list
Package Version
----------------------------- ----------
adal 1.2.2
aliyun-python-sdk-actiontrail 2.0.1
aliyun-python-sdk-core 2.13.14
aliyun-python-sdk-core-v3 2.13.11
aliyun-python-sdk-ecs 4.18.3
aliyun-python-sdk-kms 2.9.0
aliyun-python-sdk-ocs 0.0.4
aliyun-python-sdk-ram 3.1.0
aliyun-python-sdk-rds 2.4.4
aliyun-python-sdk-sts 3.0.1
aliyun-python-sdk-vpc 3.0.8
antlr4-python3-runtime 4.8
applicationinsights 0.11.9
argcomplete 1.11.1
asyncio-throttle 0.1.1
azure-cli-core 2.1.0
azure-cli-nspkg 3.0.4
azure-cli-telemetry 1.0.4
azure-common 1.1.24
azure-core 1.2.2
azure-graphrbac 0.61.1
azure-mgmt-authorization 0.60.0
azure-mgmt-compute 10.0.0
azure-mgmt-keyvault 2.1.1
azure-mgmt-monitor 0.7.0
azure-mgmt-network 9.0.0
azure-mgmt-redis 6.0.0
azure-mgmt-resource 8.0.1
azure-mgmt-security 0.3.0
azure-mgmt-sql 0.16.0
azure-mgmt-storage 7.2.0
azure-mgmt-web 0.44.0
azure-nspkg 3.0.2
bcrypt 3.1.7
boto3 1.12.3
botocore 1.15.3
cachetools 4.0.0
certifi 2019.11.28
cffi 1.14.0
chardet 3.0.4
cheroot 8.3.0
CherryPy 18.5.0
cherrypy-cors 1.6
colorama 0.4.3
coloredlogs 14.0
configparser 4.0.2
crcmod 1.7
cryptography 2.8
docutils 0.15.2
google-api-core 1.16.0
google-api-python-client 1.7.11
google-auth 1.11.2
google-auth-httplib2 0.0.3
google-cloud-container 0.4.0
google-cloud-core 1.3.0
google-cloud-iam 0.2.1
google-cloud-kms 1.3.0
google-cloud-logging 1.14.0
google-cloud-monitoring 0.34.0
google-cloud-resource-manager 0.30.0
google-cloud-storage 1.26.0
google-resumable-media 0.5.0
googleapis-common-protos 1.51.0
grpc-google-iam-v1 0.12.3
grpcio 1.27.2
httpagentparser 1.9.0
httplib2 0.17.0
humanfriendly 7.1.1
idna 2.9
importlib-metadata 1.5.0
isodate 0.6.0
jaraco.classes 3.1.0
jaraco.collections 3.0.0
jaraco.functools 3.0.0
jaraco.text 3.2.0
jmespath 0.9.4
knack 0.6.3
more-itertools 8.2.0
msrest 0.6.11
msrestazure 0.6.2
netaddr 0.7.19
oauth2client 4.1.3
oauthlib 3.1.0
oci 2.10.5
oss2 2.9.1
paramiko 2.7.1
pip 18.1
pkg-resources 0.0.0
policyuniverse 1.3.2.1
portalocker 1.5.2
portend 2.6
protobuf 3.11.3
pyasn1 0.4.8
pyasn1-modules 0.2.8
pycparser 2.19
pycryptodome 3.9.6
Pygments 2.5.2
PyJWT 1.7.1
PyNaCl 1.3.0
pyOpenSSL 19.1.0
python-dateutil 2.8.0
pytz 2019.3
PyYAML 5.3
requests 2.23.0
requests-oauthlib 1.3.0
rsa 4.0
s3transfer 0.3.3
setuptools 40.8.0
six 1.14.0
sqlitedict 1.6.0
tabulate 0.8.6
tempora 2.1.0
uritemplate 3.0.1
urllib3 1.25.8
wheel 0.30.0
zc.lockfile 2.0
zipp 3.0.0
$ python3 scout.py -h
usage: scout.py [-h] [-v] {aws,gcp,azure,aliyun,oci} ...
optional arguments:
-h, --help show this help message and exit
-v, --version show program's version number and exit
The provider you want to run scout against:
{aws,gcp,azure,aliyun,oci}
aws Run Scout against an Amazon Web Services account
gcp Run Scout against a Google Cloud Platform account
azure Run Scout against a Microsoft Azure account
aliyun Run Scout against an Alibaba Cloud account
oci Run Scout against an Oracle Cloud Infrastructure
accountCould you please provide the output of the pip list command?
Are you reusing the same virtual environment that you previously used for an older version of scout or for another project? Sometimes pip caching can lead to this type of errors for what I've read.
x4v13r64
commented
4 years ago It would be good if Azure didn't use just an outdated version of the library so that it wouldn't cause problems for the rest of the ecosystem that isn't in snooze-ville.
Sure but that's an issue for Azure. We're using the latest version of the library.
It would be good is scout would try to load Azure dependencies dynamically so that if they are severely broken they don't interfere with AWS etc.
That shouldn't be the case unless, as @paurisa mentioned, you're reusing a "contaminated" venv.
berney
commented
4 years ago [user@work ScoutSuite]$ git branch
* 5.7.0
master
[user@work ScoutSuite]$ git rev-parse HEAD
300a76b2c1bf4aaf0c8ae5cb6836f4e00c15a392
[user@work ScoutSuite]$ python3 --version
Python 3.7.6
[user@work ScoutSuite]$ python3 -m venv venv2
[user@work ScoutSuite]$ . venv2/bin/activate
(venv2) [user@work ScoutSuite]$ type python
python is /home/user/co/git/ScoutSuite/venv2/bin/python
(venv2) [user@work ScoutSuite]$ python --version
Python 3.7.6
(venv2) [user@work ScoutSuite]$ python setup.py install
running install
<snip>
Installed /home/user/co/git/ScoutSuite/venv2/lib/python3.7/site-packages/docutils-0.15.2-py3.7.egg
error: humanfriendly 4.18 is installed but humanfriendly>=7.1 is required by {'coloredlogs'} (venv2) [user@work ScoutSuite]$ type scout
scout is /home/user/co/git/ScoutSuite/venv2/bin/scout
(venv2) [user@work ScoutSuite]$ scout aws -h
Traceback (most recent call last): File "/home/user/co/git/ScoutSuite/venv2/lib64/python3.7/site-packages/pkg_resources/__init__.py", line 583, in _build_master
ws.require(__requires__)
File "/home/user/co/git/ScoutSuite/venv2/lib64/python3.7/site-packages/pkg_resources/__init__.py", line 900, in require
needed = self.resolve(parse_requirements(requirements)) File "/home/user/co/git/ScoutSuite/venv2/lib64/python3.7/site-packages/pkg_resources/__init__.py", line 791, in resolve
raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (humanfriendly 4.18 (/home/user/co/git/ScoutSuite/venv2/lib/python3.7/site-packages/humanfriendly-4.18-py3.7.egg), Requirement.parse('humanfriendly>=7.1'), {'coloredlogs'})
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/user/co/git/ScoutSuite/venv2/bin/scout", line 6, in <module>
from pkg_resources import load_entry_point
File "/home/user/co/git/ScoutSuite/venv2/lib64/python3.7/site-packages/pkg_resources/__init__.py", line 3191, in <module>
@_call_aside
File "/home/user/co/git/ScoutSuite/venv2/lib64/python3.7/site-packages/pkg_resources/__init__.py", line 3175, in _call_aside
f(*args, **kwargs)
File "/home/user/co/git/ScoutSuite/venv2/lib64/python3.7/site-packages/pkg_resources/__init__.py", line 3204, in _initialize_master_working_set
working_set = WorkingSet._build_master()
File "/home/user/co/git/ScoutSuite/venv2/lib64/python3.7/site-packages/pkg_resources/__init__.py", line 585, in _build_master
return cls._build_from_requirements(__requires__)
File "/home/user/co/git/ScoutSuite/venv2/lib64/python3.7/site-packages/pkg_resources/__init__.py", line 598, in _build_from_requirements
dists = ws.resolve(reqs, Environment())
File "/home/user/co/git/ScoutSuite/venv2/lib64/python3.7/site-packages/pkg_resources/__init__.py", line 791, in resolve
raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (humanfriendly 4.18 (/home/user/co/git/ScoutSuite/venv2/lib/python3.7/site-packages/humanfriendly-4.18-py3.7.egg), Requirement.parse('humanfriendly>=7.1'), {'coloredlogs'})
(venv2) [user@work ScoutSuite]$ pip list
Package Version
----------------------------- ----------
adal 1.2.2 aliyun-python-sdk-actiontrail 2.0.1
aliyun-python-sdk-core 2.13.14
aliyun-python-sdk-core-v3 2.13.11
aliyun-python-sdk-ecs 4.18.3 aliyun-python-sdk-kms 2.9.0
aliyun-python-sdk-ocs 0.0.4
aliyun-python-sdk-ram 3.1.0
aliyun-python-sdk-rds 2.4.4
aliyun-python-sdk-sts 3.0.1
aliyun-python-sdk-vpc 3.0.8
antlr4-python3-runtime 4.8
argcomplete 1.11.1
asyncio-throttle 0.1.1
azure-cli-core 2.1.0
azure-cli-nspkg 3.0.4
azure-cli-telemetry 1.0.4
azure-common 1.1.24
azure-core 1.2.2
azure-graphrbac 0.61.1
azure-mgmt-authorization 0.60.0
azure-mgmt-compute 10.0.0
azure-mgmt-keyvault 2.1.1
azure-mgmt-monitor 0.7.0
azure-mgmt-network 9.0.0
azure-mgmt-redis 7.0.0rc1
azure-mgmt-resource 8.0.1
azure-mgmt-security 0.3.0
azure-mgmt-sql 0.16.0
azure-mgmt-storage 7.2.0
azure-mgmt-web 0.44.0
boto3 1.12.3
botocore 1.15.3
certifi 2019.11.28
CherryPy 18.5.0
cherrypy-cors 1.6
colorama 0.4.3
coloredlogs 14.0
configparser 4.0.2
crcmod 1.7
cryptography 2.8
docutils 0.15.2
google-api-core 1.16.0
google-api-python-client 1.7.11
google-auth 1.11.2
google-auth-httplib2 0.0.3
google-cloud-container 0.4.0
google-cloud-core 1.3.0
google-cloud-iam 0.2.1
google-cloud-kms 1.3.0
google-cloud-logging 1.14.0
google-cloud-monitoring 0.34.0
google-cloud-resource-manager 0.30.0
google-cloud-storage 1.26.0
google-resumable-media 0.5.0
grpc-google-iam-v1 0.12.3
grpcio 1.27.2
httplib2 0.17.0
humanfriendly 4.18
jmespath 0.9.4
knack 0.6.3
msrest 0.6.11
msrestazure 0.6.2
netaddr 0.7.19
oauth2client 4.1.3
oci 2.10.5
oss2 2.9.1
paramiko 2.7.1
pip 19.0.3
policyuniverse 1.3.2.1
pyasn1 0.4.8
pyasn1-modules 0.2.8
pycryptodome 3.9.6
PyJWT 1.7.1
pyOpenSSL 19.1.0
python-dateutil 2.8.0
pytz 2019.3
PyYAML 5.3
requests 2.23.0
rsa 4.0
s3transfer 0.3.3
ScoutSuite 5.7.0
setuptools 40.8.0
six 1.14.0
sqlitedict 1.6.0
uritemplate 3.0.1
urllib3 1.25.8
wheel 0.30.0
You are using pip version 19.0.3, however version 20.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command. lsb_release -a
LSB Version: :core-4.1-amd64:core-4.1-noarch
Distributor ID: Fedora
Description: Fedora release 30 (Thirty)
Release: 30
Codename: ThirtyIf I build the Dockerfile I see the warning error: humanfriendly 4.18 is installed but humanfriendly>=7.1 is required by {'coloredlogs'} but docker run --rm -it berne/scout aws -h runs without crashing, unlike native Fedora 30.
x4v13r64
commented
4 years ago I don't see an appropriate fix on our end, seeing how the "old" version is a requirement of azure-cli-core.
Does pip install -r requirements.txt also make it fail in your base system?
[user@work ScoutSuite]$ python3 -m venv venv2
What does this do?
prisas
commented
4 years ago @j4v The command python3 -m venv venv2 creates a virtual environments called venv2.
Venv is now part of Python 3 (since version 3.6 I believe) and it's not necessary to install any other virtual environments tools (like pyenv or venv standalone). Furthermore, this is now the recommended way of creating a virtual environment on Python 3.
berney
commented
4 years ago # create a new python3 virtual environment (e.g. clean)
python3 -m venv venv3
# use python virtual environment venv3
. venv3/bin/activate
# Try the command you asked
pip install -r requirements.txt
<snip>
azure-cli-core 2.1.0 has requirement humanfriendly~=4.7, but you'll have humanfriendly 7.1.1 which is incompatible.python ./scout.py aws -h works without an error (similar to Docker).
pip list
(venv3) [user@work ScoutSuite]$ pip list [58/489]
Package Version
----------------------------- ----------
adal 1.2.2
aliyun-python-sdk-actiontrail 2.0.1
aliyun-python-sdk-core 2.13.14
aliyun-python-sdk-core-v3 2.13.11
aliyun-python-sdk-ecs 4.18.3
aliyun-python-sdk-kms 2.9.0
aliyun-python-sdk-ocs 0.0.4
aliyun-python-sdk-ram 3.1.0
aliyun-python-sdk-rds 2.4.4
aliyun-python-sdk-sts 3.0.1
aliyun-python-sdk-vpc 3.0.8
antlr4-python3-runtime 4.8
applicationinsights 0.11.9
argcomplete 1.11.1
asyncio-throttle 0.1.1
azure-cli-core 2.1.0
azure-cli-nspkg 3.0.4
azure-cli-telemetry 1.0.4
azure-common 1.1.24
azure-core 1.2.2
azure-graphrbac 0.61.1
azure-mgmt-authorization 0.60.0
azure-mgmt-compute 10.0.0
azure-mgmt-keyvault 2.1.1
azure-mgmt-monitor 0.7.0
azure-mgmt-network 9.0.0
azure-mgmt-redis 6.0.0
azure-mgmt-resource 8.0.1
azure-mgmt-security 0.3.0
azure-mgmt-sql 0.16.0
azure-mgmt-storage 7.2.0
azure-mgmt-web 0.44.0
azure-nspkg 3.0.2
bcrypt 3.1.7
boto3 1.12.5
botocore 1.15.5
cachetools 4.0.0
certifi 2019.11.28
cffi 1.14.0
chardet 3.0.4
cheroot 8.3.0
CherryPy 18.5.0
cherrypy-cors 1.6
colorama 0.4.3
coloredlogs 14.0
configparser 4.0.2
crcmod 1.7
cryptography 2.8
docutils 0.15.2
google-api-core 1.16.0
google-api-python-client 1.7.11
google-auth 1.11.2
google-auth-httplib2 0.0.3
google-cloud-container 0.4.0
google-cloud-core 1.3.0
google-cloud-iam 0.2.1
google-cloud-kms 1.3.0
google-cloud-logging 1.14.0
google-cloud-monitoring 0.34.0
google-cloud-resource-manager 0.30.0
google-cloud-storage 1.26.0
google-resumable-media 0.5.0
googleapis-common-protos 1.51.0
grpc-google-iam-v1 0.12.3
grpcio 1.27.2
httpagentparser 1.9.0
httplib2 0.17.0
humanfriendly 7.1.1
idna 2.9
importlib-metadata 1.5.0
isodate 0.6.0
jaraco.classes 3.1.0
jaraco.collections 3.0.0
jaraco.functools 3.0.0
jaraco.text 3.2.0
jmespath 0.9.4
knack 0.6.3
more-itertools 8.2.0
msrest 0.6.11
msrestazure 0.6.2
netaddr 0.7.19
oauth2client 4.1.3
oauthlib 3.1.0
oci 2.10.5
oss2 2.9.1
paramiko 2.7.1
pip 19.0.3
policyuniverse 1.3.2.1
portalocker 1.5.2
portend 2.6
protobuf 3.11.3
pyasn1 0.4.8
pyasn1-modules 0.2.8
pycparser 2.19
pycryptodome 3.9.7
Pygments 2.5.2
PyJWT 1.7.1
PyNaCl 1.3.0
pyOpenSSL 19.1.0
python-dateutil 2.8.0
pytz 2019.3
PyYAML 5.3
requests 2.23.0
requests-oauthlib 1.3.0
rsa 4.0
s3transfer 0.3.3
setuptools 40.8.0
six 1.14.0
sqlitedict 1.6.0
tabulate 0.8.6
tempora 2.1.0
uritemplate 3.0.1
urllib3 1.25.8
wheel 0.30.0
zc.lockfile 2.0
zipp 3.0.0
You are using pip version 19.0.3, however version 20.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.Weird that this works and setup.py install doesn't.
If I follow this by a python setup.py install and run the installed scout, scout aws -h will fail as per OP. python ./scout.py aws -h continues to work, so there is a difference in the way importing/conflict management works depending on running installed vs running from a dir.
x4v13r64
commented
4 years ago If I follow this by a python setup.py install and run the installed scout, scout aws -h will fail as per OP. python ./scout.py aws -h continues to work, so there is a difference in the way importing/conflict management works depending on running installed vs running from a dir.
Thanks @berney, I guess we know where to look then.
berney
commented
4 years ago If I follow this by a python setup.py install and run the installed scout, scout aws -h will fail as per OP. python ./scout.py aws -h continues to work, so there is a difference in the way importing/conflict management works depending on running installed vs running from a dir.
This is the essential difference in docker as well. In my OP I was doing python setup.py install and then running scout, which hit the problem in OP. In the Dockerfile it is copying scout and running it from the directory, rather than installing it and running the installed version.
I can reproduce the same problem as in my OP if I start the docker image, install scout, and then try to run the installed version.
[user@work ScoutSuite]$ docker build -t berne/scoutsuite .
<snip>
[user@work ScoutSuite]$ docker run --rm -it --entrypoint bash berne/scoutsuite
root@2f88427148cb:/opt# python --version
Python 3.8.1
root@2f88427148cb:/opt# type scout
bash: type: scout: not found
root@2f88427148cb:/opt# python setup.py install running install
<snip>
Adding humanfriendly 4.18 to easy-install.pth file [1/1889]
Installing humanfriendly script to /usr/local/bin
Installed /usr/local/lib/python3.8/site-packages/humanfriendly-4.18-py3.8.egg
error: humanfriendly 4.18 is installed but humanfriendly>=7.1 is required by {'coloredlogs'}
root@2f88427148cb:/opt# type scout
scout is /usr/local/bin/scout
root@2f88427148cb:/opt# scout aws -h
Traceback (most recent call last):
<snip>
pkg_resources.ContextualVersionConflict: (humanfriendly 7.1.1 (/usr/local/lib/python3.8/site-packages), Requirement.parse('humanfriendly~=4.7'), {'az
ure-cli-core'})
<snip>It would be good is scout would try to load Azure dependencies dynamically so that if they are severely broken they don't interfere with AWS etc.
That shouldn't be the case unless, as @paurisa mentioned, you're reusing a "contaminated" venv.
I mean that instead of something like:
# Import everything we might need even if we don't need it
import azure_cliWhich blows up because of their old dependencies conflicting with your modern dependencies. To do something like this:
if args.azure:
# User wants to use azure, e.g `scout azure`
try:
import azure_cli
except SomeImportError as e:
print(f"Cannot import Azure CLI: {e}")So that use-cases unrelated to Azure (e.g. AWS) won't blow up on Azure dependencies being trash.
berney
commented
4 years ago Related Upstream azure-cli humanfriendly dependency issue https://github.com/Azure/azure-cli/issues/12283
Someone else has kindly asked Azure to update their humanfriendly dependency.
gonrial
commented
4 years ago I managed to do this quick fix to make it work until is resolved
# create a virtual env
python3 -m venv venv
# activate the virtual env
source venv/bin/activate
# install all dependencies
pip install scoutsuite
# force old dependencies
pip install humanfriendly==4.18 coloredlogs==10.0 setuptools==40.3.0
# success 🎉
scoutI managed to do this quick fix to make it work until is resolved
But from the above pip isn't failing, only python setup.py install does.
x4v13r64
commented
4 years ago Apparently pip works because it doesn't actual resolve dependencies (https://github.com/pypa/pip/issues/775#issuecomment-12748095). setup.py is failing because it's actually resolving them...
I'm not sure what we can do until they fix https://github.com/Azure/azure-cli/issues/12283.
x4v13r64
commented
4 years ago This was resolved by reverting coloredlogs to <=10.0, which relies on a compatible version of humandfriendly. Once azure-cli fix this we can update the coloredlogs requirement.
Closing, this has been fixed in master.
Describe the bug
Please provide:
Can't run
scoutdue to two dependencies have conflicting version requirements, which can't be simultaneously satisfied.Fully console output
To Reproduce
See above, I'm on master branch and included commit id, I'm using python3 venv ran setup, and I'm trying to run
scout,scout --help, andscout awsbut it fails due to dependency version requirement conflict.Additional context
Python can only have one version of a package installed, in a single environment, e.g a single venv, at a time. So you can't have version ~= 4.7 and >=7.1 of a single package.
Workaround Installing the minimum requirement for
azure-cli-core, rather than defaulting to the latest version that meets the restriction seems to work as then azure-cli-core's dependencies are different and not conflicting.It would be good if Azure didn't use just an outdated version of the library so that it wouldn't cause problems for the rest of the ecosystem that isn't in snooze-ville.
It would be good is scout would try to load Azure dependencies dynamically so that if they are severely broken they don't interfere with AWS etc.