search

cloud-custodian / cloud-custodian

Rules engine for cloud security, cost optimization, and governance, DSL in yaml for policies to query, filter, and take actions on resources
https://cloudcustodian.io
Apache License 2.0
5.37k stars 1.46k forks source link

azure - knack.util.CLIError: Please run 'az login' to setup account. #2561

Closed nimbusscale closed 6 years ago

nimbusscale commented 6 years ago

When getting the latest from master and building via tox the custodian command generates this error when attempting to validate or run a policy that interacts with AWS resources.

desktop:cloud-custodian$ git clone https://github.com/capitalone/cloud-custodian.git scratch/cloud-custodian
Cloning into 'scratch/cloud-custodian'...
remote: Counting objects: 47001, done.
remote: Compressing objects: 100% (86/86), done.
remote: Total 47001 (delta 54), reused 45 (delta 25), pack-reused 46890
Receiving objects: 100% (47001/47001), 67.87 MiB | 3.23 MiB/s, done.
Resolving deltas: 100% (34023/34023), done.
desktop:cloud-custodian$ cd scratch/cloud-custodian/
desktop:cloud-custodian$ tox -e py27
GLOB sdist-make: /home/jjk3/PycharmProjects/customer/cloud-custodian/scratch/cloud-custodian/setup.py
py27 create: /home/jjk3/PycharmProjects/customer/cloud-custodian/scratch/cloud-custodian/.tox/py27
py27 installdeps: -rrequirements-dev.txt, -rtools/c7n_mailer/requirements.txt, -rtools/c7n_azure/requirements.txt, -rtools/c7n_gcp/requirements.txt
py27 inst: /home/jjk3/PycharmProjects/customer/cloud-custodian/scratch/cloud-custodian/.tox/dist/c7n-0.8.28.2.zip
py27 installed: adal==0.5.1,alabaster==0.7.11,apipkg==1.4,applicationinsights==0.11.4,argcomplete==1.9.4,asn1crypto==0.24.0,atomicwrites==1.1.5,attrs==18.1.0,azure==3.0.0,azure-batch==4.1.3,azure-cli-core==2.0.38,azure-cli-nspkg==3.0.2,azure-common==1.1.12,azure-cosmosdb-nspkg==2.0.2,azure-cosmosdb-table==1.0.3,azure-datalake-store==0.0.22,azure-eventgrid==0.1.0,azure-graphrbac==0.40.0,azure-keyvault==0.3.7,azure-mgmt==2.0.0,azure-mgmt-advisor==1.0.1,azure-mgmt-applicationinsights==0.1.1,azure-mgmt-authorization==0.30.0,azure-mgmt-batch==5.0.1,azure-mgmt-batchai==0.2.0,azure-mgmt-billing==0.1.0,azure-mgmt-cdn==2.0.0,azure-mgmt-cognitiveservices==2.0.0,azure-mgmt-commerce==1.0.1,azure-mgmt-compute==3.0.1,azure-mgmt-consumption==2.0.0,azure-mgmt-containerinstance==0.3.1,azure-mgmt-containerregistry==1.0.1,azure-mgmt-containerservice==3.0.1,azure-mgmt-cosmosdb==0.3.1,azure-mgmt-datafactory==0.4.0,azure-mgmt-datalake-analytics==0.3.0,azure-mgmt-datalake-nspkg==2.0.0,azure-mgmt-datalake-store==0.3.0,azure-mgmt-devtestlabs==2.2.0,azure-mgmt-dns==1.2.0,azure-mgmt-eventgrid==0.4.0,azure-mgmt-eventhub==1.2.0,azure-mgmt-hanaonazure==0.1.1,azure-mgmt-iothub==0.4.0,azure-mgmt-iothubprovisioningservices==0.1.0,azure-mgmt-keyvault==0.40.0,azure-mgmt-loganalytics==0.1.0,azure-mgmt-logic==2.1.0,azure-mgmt-machinelearningcompute==0.4.1,azure-mgmt-managementpartner==0.1.0,azure-mgmt-marketplaceordering==0.1.0,azure-mgmt-media==0.2.0,azure-mgmt-monitor==0.4.0,azure-mgmt-msi==0.1.0,azure-mgmt-network==1.7.1,azure-mgmt-notificationhubs==1.0.0,azure-mgmt-nspkg==2.0.0,azure-mgmt-powerbiembedded==1.0.0,azure-mgmt-rdbms==0.1.0,azure-mgmt-recoveryservices==0.2.0,azure-mgmt-recoveryservicesbackup==0.1.1,azure-mgmt-redis==5.0.0,azure-mgmt-relay==0.1.0,azure-mgmt-reservations==0.1.0,azure-mgmt-resource==1.2.2,azure-mgmt-scheduler==1.1.3,azure-mgmt-search==1.0.0,azure-mgmt-servermanager==1.2.0,azure-mgmt-servicebus==0.4.0,azure-mgmt-servicefabric==0.1.0,azure-mgmt-sql==0.8.6,azure-mgmt-storage==1.5.0,azure-mgmt-subscription==0.1.0,azure-mgmt-trafficmanager==0.40.0,azure-mgmt-web==0.34.1,azure-nspkg==2.0.0,azure-servicebus==0.21.1,azure-servicefabric==6.1.2.9,azure-servicemanagement-legacy==0.20.6,azure-storage-blob==1.1.0,azure-storage-common==1.1.0,azure-storage-file==1.1.0,azure-storage-nspkg==3.0.0,azure-storage-queue==1.1.0,Babel==2.6.0,backports.functools-lru-cache==1.5,bcrypt==3.1.4,boto3==1.7.45,botocore==1.10.45,c7n==0.8.28.2,-e git+https://github.com/capitalone/cloud-custodian.git@a721e673b6404e2c675b5fa63fece44c2b5b02e3#egg=c7n_azure&subdirectory=tools/c7n_azure,-e git+https://github.com/capitalone/cloud-custodian.git@a721e673b6404e2c675b5fa63fece44c2b5b02e3#egg=c7n_gcp&subdirectory=tools/c7n_gcp,-e git+https://github.com/capitalone/cloud-custodian.git@a721e673b6404e2c675b5fa63fece44c2b5b02e3#egg=c7n_mailer&subdirectory=tools/c7n_mailer,-e git+https://github.com/capitalone/cloud-custodian.git@a721e673b6404e2c675b5fa63fece44c2b5b02e3#egg=c7n_sphinxext&subdirectory=tools/c7n_sphinxext,certifi==2018.4.16,cffi==1.11.5,chardet==3.0.4,click==6.7,colorama==0.3.9,configparser==3.5.0,contextlib2==0.5.5,coverage==4.5.1,cryptography==2.2.2,datadog==0.21.0,decorator==4.3.0,docutils==0.14,entrypoints==0.2.3,enum34==1.1.6,execnet==1.5.0,fakeredis==0.11.0,flake8==3.5.0,funcsigs==1.0.2,functools32==3.2.3.post2,futures==3.1.1,google-api-python-client==1.6.7,httplib2==0.11.3,humanfriendly==4.12.1,idna==2.7,imagesize==1.0.0,ipaddress==1.0.22,isodate==0.6.0,Jinja2==2.10,jmespath==0.9.3,jsonpatch==1.23,jsonpointer==2.0,jsonschema==2.6.0,keyring==13.1.0,knack==0.3.3,ldap3==2.5,markup==0.2,MarkupSafe==1.0,mccabe==0.6.1,mock==2.0.0,monotonic==1.5,more-itertools==4.2.0,msrest==0.5.1,msrestazure==0.4.33,nose==1.3.7,nose-timer==0.7.1,oauth2client==4.1.2,oauthlib==2.1.0,paramiko==2.4.1,pathlib2==2.3.2,pbr==4.0.4,pkginfo==1.4.2,placebo==0.8.1,pluggy==0.6.0,py==1.5.3,pyasn1==0.4.3,pyasn1-modules==0.2.1,pycodestyle==2.3.1,pycparser==2.18,pyflakes==1.6.0,Pygments==2.2.0,PyJWT==1.6.4,PyNaCl==1.2.1,pyOpenSSL==18.0.0,pytest==3.6.2,pytest-cov==2.5.1,pytest-forked==0.2,pytest-xdist==1.22.2,python-dateutil==2.7.3,python-http-client==3.1.0,pytz==2018.4,PyYAML==3.12,ratelimiter==1.2.0.post0,redis==2.10.6,requests==2.19.1,requests-oauthlib==1.0.0,requests-toolbelt==0.8.0,retrying==1.3.3,rsa==3.4.2,ruamel.ordereddict==0.4.13,ruamel.yaml==0.14.12,s3transfer==0.1.13,scandir==1.7,SecretStorage==2.3.1,sendgrid==5.4.0,simplejson==3.15.0,six==1.11.0,slackclient==1.2.1,snowballstemmer==1.2.1,Sphinx==1.6.7,sphinx-rtd-theme==0.2.4,sphinxcontrib-websupport==1.1.0,tabulate==0.8.2,termcolor==1.1.0,tox==3.0.0,tqdm==4.23.4,twine==1.11.0,typing==3.6.4,uritemplate==3.0.0,urllib3==1.23,vcrpy==1.11.0,vcrpy-unittest==0.1.6,vcversioner==2.16.0.0,virtualenv==16.0.0,websocket-client==0.48.0,wrapt==1.10.11
py27 runtests: PYTHONHASHSEED='1889879956'
py27 runtests: commands[0] | py.test --tb=native -n auto tests tools
============================================================================ test session starts =============================================================================
platform linux2 -- Python 2.7.14, pytest-3.6.2, py-1.5.3, pluggy-0.6.0
rootdir: /home/jjk3/PycharmProjects/customer/cloud-custodian/scratch/cloud-custodian, inifile: tox.ini
plugins: xdist-1.22.2, forked-0.2, cov-2.5.1
gw0 [1110] / gw1 [1110] / gw2 [1110] / gw3 [1110]
scheduling tests via LoadScheduling
...................................................................................................................................................................... [ 14%]
...................................................................................................................................................................... [ 29%]
...................................................................................................................................................................... [ 44%]
...................................................................................................................................................................... [ 59%]
...................................................................................................................................................................... [ 74%]
...................................................................................................................................................................... [ 89%]
..................................................................................................................                                                     [100%]
======================================================================== 1110 passed in 83.97 seconds ========================================================================
__________________________________________________________________________________ summary ___________________________________________________________________________________
  py27: commands succeeded
  congratulations :)
desktop:cloud-custodian$ source .tox/py27/bin/activate
(py27) desktop:cloud-custodian$ cd ../..
(py27) desktop:cloud-custodian$ custodian validate nothing-policy.yaml
Traceback (most recent call last):
  File "/home/jjk3/PycharmProjects/customer/cloud-custodian/scratch/cloud-custodian/.tox/py27/bin/custodian", line 11, in <module>
    sys.exit(main())
  File "/home/jjk3/PycharmProjects/customer/cloud-custodian/scratch/cloud-custodian/.tox/py27/local/lib/python2.7/site-packages/c7n/cli.py", line 357, in main
    command(config)
  File "/home/jjk3/PycharmProjects/customer/cloud-custodian/scratch/cloud-custodian/.tox/py27/local/lib/python2.7/site-packages/c7n/commands.py", line 165, in validate
    load_resources()
  File "/home/jjk3/PycharmProjects/customer/cloud-custodian/scratch/cloud-custodian/.tox/py27/local/lib/python2.7/site-packages/c7n/resources/__init__.py", line 95, in load_resources
    resources.load_plugins()
  File "/home/jjk3/PycharmProjects/customer/cloud-custodian/scratch/cloud-custodian/.tox/py27/local/lib/python2.7/site-packages/c7n/registry.py", line 109, in load_plugins
    f = ep.load()
  File "/home/jjk3/PycharmProjects/customer/cloud-custodian/scratch/cloud-custodian/.tox/py27/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2318, in load
    return self.resolve()
  File "/home/jjk3/PycharmProjects/customer/cloud-custodian/scratch/cloud-custodian/.tox/py27/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2324, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/home/jjk3/PycharmProjects/customer/cloud-custodian/scratch/cloud-custodian/tools/c7n_azure/c7n_azure/entry.py", line 30, in <module>
    import c7n_azure.resources.access_control
  File "/home/jjk3/PycharmProjects/customer/cloud-custodian/scratch/cloud-custodian/tools/c7n_azure/c7n_azure/resources/access_control.py", line 42, in <module>
    class RoleDefinition(QueryResourceManager):
  File "/home/jjk3/PycharmProjects/customer/cloud-custodian/scratch/cloud-custodian/tools/c7n_azure/c7n_azure/resources/access_control.py", line 44, in RoleDefinition
    class resource_type(object):
  File "/home/jjk3/PycharmProjects/customer/cloud-custodian/scratch/cloud-custodian/tools/c7n_azure/c7n_azure/resources/access_control.py", line 45, in resource_type
    s = Session()
  File "/home/jjk3/PycharmProjects/customer/cloud-custodian/scratch/cloud-custodian/tools/c7n_azure/c7n_azure/session.py", line 72, in __init__
    resource=AZURE_PUBLIC_CLOUD.endpoints.active_directory_resource_id)
  File "/home/jjk3/PycharmProjects/customer/cloud-custodian/scratch/cloud-custodian/.tox/py27/local/lib/python2.7/site-packages/azure/cli/core/_profile.py", line 449, in get_login_credentials
    account = self.get_subscription(subscription_id)
  File "/home/jjk3/PycharmProjects/customer/cloud-custodian/scratch/cloud-custodian/.tox/py27/local/lib/python2.7/site-packages/azure/cli/core/_profile.py", line 414, in get_subscription
    raise CLIError("Please run 'az login' to setup account.")
knack.util.CLIError: Please run 'az login' to setup account.

When install latest release things work as expected:

desktop:cloud-custodian$ virtualenv --python=python2 .venv/custodian
Running virtualenv with interpreter /usr/bin/python2
New python executable in /home/jjk3/PycharmProjects/customer/cloud-custodian/.venv/custodian/bin/python2
Also creating executable in /home/jjk3/PycharmProjects/customer/cloud-custodian/.venv/custodian/bin/python
Installing setuptools, pip, wheel...done.
desktop:cloud-custodian$ source .venv/custodian/bin/activate
(custodian) desktop:cloud-custodian$ pip install c7n
Collecting c7n
Collecting jsonpatch>=1.21 (from c7n)
  Using cached https://files.pythonhosted.org/packages/a0/e6/d50d526ae2218b765ddbdb2dda14d65e19f501ce07410b375bc43ad20b7a/jsonpatch-1.23-py2.py3-none-any.whl
Collecting boto3>=1.7.2 (from c7n)
  Using cached https://files.pythonhosted.org/packages/c8/aa/60603ce1b4959905bc1dd678fed95511e1d36e4448389bf274de89dc2c4b/boto3-1.7.45-py2.py3-none-any.whl
Collecting argcomplete (from c7n)
  Using cached https://files.pythonhosted.org/packages/31/88/ba8d8684a8a27749250c66ff7c2b408fdbc29b50da61200338ff9b2607bf/argcomplete-1.9.4-py2.py3-none-any.whl
Collecting jsonschema (from c7n)
  Using cached https://files.pythonhosted.org/packages/77/de/47e35a97b2b05c2fadbec67d44cfcdcd09b8086951b331d82de90d2912da/jsonschema-2.6.0-py2.py3-none-any.whl
Collecting botocore>=1.10.2 (from c7n)
  Using cached https://files.pythonhosted.org/packages/2a/c8/b180fb83fa46d2b56ea059177dc3c00647d622987daf5e7ffbc658555ede/botocore-1.10.45-py2.py3-none-any.whl
Collecting pyyaml (from c7n)
Collecting tabulate (from c7n)
Collecting jsonpointer>=1.9 (from jsonpatch>=1.21->c7n)
  Using cached https://files.pythonhosted.org/packages/18/b0/a80d29577c08eea401659254dfaed87f1af45272899e1812d7e01b679bc5/jsonpointer-2.0-py2.py3-none-any.whl
Collecting jmespath<1.0.0,>=0.7.1 (from boto3>=1.7.2->c7n)
  Using cached https://files.pythonhosted.org/packages/b7/31/05c8d001f7f87f0f07289a5fc0fc3832e9a57f2dbd4d3b0fee70e0d51365/jmespath-0.9.3-py2.py3-none-any.whl
Collecting s3transfer<0.2.0,>=0.1.10 (from boto3>=1.7.2->c7n)
  Using cached https://files.pythonhosted.org/packages/d7/14/2a0004d487464d120c9fb85313a75cd3d71a7506955be458eebfe19a6b1d/s3transfer-0.1.13-py2.py3-none-any.whl
Collecting functools32; python_version == "2.7" (from jsonschema->c7n)
Collecting docutils>=0.10 (from botocore>=1.10.2->c7n)
  Using cached https://files.pythonhosted.org/packages/50/09/c53398e0005b11f7ffb27b7aa720c617aba53be4fb4f4f3f06b9b5c60f28/docutils-0.14-py2-none-any.whl
Collecting python-dateutil<3.0.0,>=2.1; python_version >= "2.7" (from botocore>=1.10.2->c7n)
  Using cached https://files.pythonhosted.org/packages/cf/f5/af2b09c957ace60dcfac112b669c45c8c97e32f94aa8b56da4c6d1682825/python_dateutil-2.7.3-py2.py3-none-any.whl
Collecting futures<4.0.0,>=2.2.0; python_version == "2.6" or python_version == "2.7" (from s3transfer<0.2.0,>=0.1.10->boto3>=1.7.2->c7n)
  Using cached https://files.pythonhosted.org/packages/2d/99/b2c4e9d5a30f6471e410a146232b4118e697fa3ffc06d6a65efde84debd0/futures-3.2.0-py2-none-any.whl
Collecting six>=1.5 (from python-dateutil<3.0.0,>=2.1; python_version >= "2.7"->botocore>=1.10.2->c7n)
  Using cached https://files.pythonhosted.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl
Installing collected packages: jsonpointer, jsonpatch, jmespath, docutils, six, python-dateutil, botocore, futures, s3transfer, boto3, argcomplete, functools32, jsonschema, pyyaml, tabulate, c7n
Successfully installed argcomplete-1.9.4 boto3-1.7.45 botocore-1.10.45 c7n-0.8.28.2 docutils-0.14 functools32-3.2.3.post2 futures-3.2.0 jmespath-0.9.3 jsonpatch-1.23 jsonpointer-2.0 jsonschema-2.6.0 python-dateutil-2.7.3 pyyaml-3.12 s3transfer-0.1.13 six-1.11.0 tabulate-0.8.2
(custodian) desktop:cloud-custodian$ custodian validate nothing-policy.yaml
2018-06-25 11:23:57,378: custodian.commands:INFO Configuration valid: nothing-policy.yaml

Here is the nothing-policy.yaml:

policies:
  - name: nothing
    resource: s3
    comment: policy that does nothing to help with cleanup
    mode:
      role: arn:aws:iam::899826514230:role/cloudcustodian
      type: cloudtrail
      events:
        - CreateBucket
    filters:
      - "tag:BogusTag": present
    actions:
      - no-op

This same error occurs when attempting to custodian run with the policy as well. I tried with a few different policies, but same issue with all them and this is the simplest of the bunch.

thisisshi commented 6 years ago

Looks like an azure session is being instantiated when we're registering the resources here: https://github.com/capitalone/cloud-custodian/blob/master/tools/c7n_azure/c7n_azure/resources/access_control.py#L45 the session seems to be required to fill out the enum spec for the access control resource, and the verbose logging is actually related to this PR here: https://github.com/capitalone/cloud-custodian/pull/2534

thisisshi commented 6 years ago

@erwelch @stefangordon I'm not too familiar with azure session management, is it possible to refactor the access control resource to prevent session creation during the plugin registration and instantiate the session elsewhere?

stefangordon commented 6 years ago

I'm thinking of doing lazy init on the get client method. Seems better. I will do it today or tomorrow. (Or some other solution)