Azure AAD Hanging & Process Getting Killed

reach4bawer commented 4 years ago

Describe the bug When running scout suite for a subscription for Azure scout stops responding after showing the information about Fetching resources for the App Services service.

I updated to the latest version of scout suite. Running scout suite on two different systems with the same user credentials I get two different errors but in both instances the system gets stuck in both instances -

1st instance RedHat linux -

This was on the previous version -

[root@ip-x.x.x.x azure_test]# scout azure --user-account -u $u -p $PP --subscriptions xyz --debug 2020-04-02 15:58:25 ip-x.x.x.x.linux.internal scout[23057] INFO Launching Scout 2020-04-02 15:58:25 ip-x.x.x.x.linux.internal scout[23057] INFO Authenticating to cloud provider 2020-04-02 15:58:29 ip-x.x.x.x.linux.internal scout[23057] INFO Running against 1 subscription(s) 2020-04-02 15:58:29 ip-x.x.x.x.linux.internal scout[23057] INFO Gathering data from APIs 2020-04-02 15:58:29 ip-x.x.x.x.linux.internal scout[23057] INFO Fetching resources for the AAD service 2020-04-02 15:58:29 ip-x.x.x.x.linux.internal scout[23057] INFO Fetching resources for the ARM service 2020-04-02 15:58:29 ip-x.x.x.x.linux.internal scout[23057] INFO Fetching resources for the Security Center service 2020-04-02 15:58:29 ip-x.x.x.x.linux.internal scout[23057] INFO Fetching resources for the SQL Database service 2020-04-02 15:58:29 ip-x.x.x.x.linux.internal scout[23057] INFO Fetching resources for the Storage Accounts service 2020-04-02 15:58:29 ip-x.x.x.x.linux.internal scout[23057] INFO Fetching resources for the Key Vault service 2020-04-02 15:58:29 ip-x.x.x.x.linux.internal scout[23057] INFO Fetching resources for the Network service 2020-04-02 15:58:29 ip-x.x.x.x.linux.internal scout[23057] INFO Fetching resources for the Virtual Machines service 2020-04-02 15:58:29 ip-x.x.x.x.linux.internal scout[23057] INFO Fetching resources for the App Services service 2020-04-02 15:58:29 ip-x.x.x.x.linux.internal scout[23057] ERROR securitycenter.py L83: Failed to retrieve compliance results: 'SecurityCenter' object has no attribute 'compliance_results' Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/ScoutSuite/providers/azure/facade/securitycenter.py", line 83, in get_compliance_results lambda: list(client.compliance_results.list(scope=scope)) File "/usr/local/lib/python3.7/site-packages/ScoutSuite/providers/utils.py", line 24, in run_concurrently return await run_function_concurrently(function) File "/usr/lib64/python3.7/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, *self.kwargs) File "/usr/local/lib/python3.7/site-packages/ScoutSuite/providers/azure/facade/securitycenter.py", line 83, in lambda: list(client.compliance_results.list(scope=scope)) AttributeError: 'SecurityCenter' object has no attribute 'compliance_results' 2020-04-02 15:58:29 ip-x.x.x.x.linux.internal scout[23057] ERROR securitycenter.py L95: Failed to retrieve regulatory compliance standards: 'SecurityCenter' object has no attribute 'regulatory_compliance_standards' Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/ScoutSuite/providers/azure/facade/securitycenter.py", line 95, in get_regulatory_compliance_results lambda: list(client.regulatory_compliance_standards.list()) File "/usr/local/lib/python3.7/site-packages/ScoutSuite/providers/utils.py", line 24, in run_concurrently return await run_function_concurrently(function) File "/usr/lib64/python3.7/concurrent/futures/thread.py", line 57, in run result = self.fn(self.args, **self.kwargs) File "/usr/local/lib/python3.7/site-packages/ScoutSuite/providers/azure/facade/securitycenter.py", line 95, in lambda: list(client.regulatory_compliance_standards.list()) AttributeError: 'SecurityCenter' object has no attribute 'regulatory_compliance_standards'

After updating to the latest version -

Successfully installed azure-cli-core-2.3.1 azure-mgmt-compute-12.0.0 azure-mgmt-monitor-0.8.0 azure-mgmt-network-10.0.0 azure-mgmt-security-0.3.0 azure-mgmt-sql-0.18.0 azure-mgmt-storage-9.0.0 azure-mgmt-web-0.45.0 boto3-1.12.35 botocore-1.15.35 cryptography-2.9 google-auth-1.13.1 google-cloud-storage-1.27.0 httplib2-0.17.1 humanfriendly-8.1 importlib-metadata-1.6.0 knack-0.7.0rc1 oci-2.12.2 pkginfo-1.5.0.1 pytz-2019.3 pyyaml-5.3.1 rsa-4.0 scoutsuite-5.8.1 setuptools-46.1.3

[root@ip-x.x.x.x azure_test]# scout azure --user-account -u $u -p $PP --subscriptions xyz --debug 2020-04-02 16:32:17 ip-x.x.x.x.linux.internal scout[27034] INFO Launching Scout 2020-04-02 16:32:17 ip-x.x.x.x.linux.internal scout[27034] INFO Authenticating to cloud provider 2020-04-02 16:32:23 ip-x.x.x.x.linux.internal scout[27034] INFO Running against 1 subscription(s) 2020-04-02 16:32:23 ip-x.x.x.x.linux.internal scout[27034] INFO Gathering data from APIs 2020-04-02 16:32:23 ip-x.x.x.x.linux.internal scout[27034] INFO Fetching resources for the AAD service 2020-04-02 16:32:23 ip-x.x.x.x.linux.internal scout[27034] INFO Fetching resources for the ARM service 2020-04-02 16:32:23 ip-x.x.x.x.linux.internal scout[27034] INFO Fetching resources for the Security Center service 2020-04-02 16:32:23 ip-x.x.x.x.linux.internal scout[27034] INFO Fetching resources for the SQL Database service 2020-04-02 16:32:23 ip-x.x.x.x.linux.internal scout[27034] INFO Fetching resources for the Storage Accounts service 2020-04-02 16:32:23 ip-x.x.x.x.linux.internal scout[27034] INFO Fetching resources for the Key Vault service 2020-04-02 16:32:23 ip-x.x.x.x.linux.internal scout[27034] INFO Fetching resources for the Network service 2020-04-02 16:32:23 ip-x.x.x.x.linux.internal scout[27034] INFO Fetching resources for the Virtual Machines service 2020-04-02 16:32:23 ip-x.x.x.x.linux.internal scout[27034] INFO Fetching resources for the App Services service

2nd Instance RedHat linux - On my Macbook I get the following -

scout azure --user-account -u $u -p $PP --subscriptions xyz --debug 2020-04-02 16:46:42 MACBOOK scout[24261] INFO Launching Scout 2020-04-02 16:46:42 MACBOOK scout[24261] INFO Authenticating to cloud provider 2020-04-02 16:46:51 MACBOOK scout[24261] INFO Running against 1 subscription(s) 2020-04-02 16:46:51 MACBOOK scout[24261] INFO Gathering data from APIs 2020-04-02 16:46:51 MACBOOK scout[24261] INFO Fetching resources for the AAD service 2020-04-02 16:46:51 MACBOOK scout[24261] INFO Fetching resources for the ARM service 2020-04-02 16:46:51 MACBOOK scout[24261] INFO Fetching resources for the Security Center service 2020-04-02 16:46:51 MACBOOK scout[24261] INFO Fetching resources for the SQL Database service 2020-04-02 16:46:51 MACBOOK scout[24261] INFO Fetching resources for the Storage Accounts service 2020-04-02 16:46:51 MACBOOK scout[24261] INFO Fetching resources for the Key Vault service 2020-04-02 16:46:51 MACBOOK scout[24261] INFO Fetching resources for the Network service 2020-04-02 16:46:51 MACBOOK scout[24261] INFO Fetching resources for the Virtual Machines service 2020-04-02 16:46:52 MACBOOK asyncio[24261] ERROR Task exception was never retrieved future: <Task finished coro=<RoleAssignments.fetch_all() done, defined at /usr/local/lib/python3.7/site-packages/ScoutSuite/providers/azure/resources/arm/role_assignments.py:11> exception=AttributeError("'RoleAssignment' object has no attribute 'principal_type'")> Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/ScoutSuite/providers/azure/resources/arm/role_assignments.py", line 13, in fetch_all id, role_assignment = self._parse_role_assignment(raw_role_assignment) File "/usr/local/lib/python3.7/site-packages/ScoutSuite/providers/azure/resources/arm/role_assignments.py", line 24, in _parse_role_assignment role_assignment_dict['principal_type'] = raw_role_assignment.principal_type AttributeError: 'RoleAssignment' object has no attribute 'principal_type' 2020-04-02 16:46:55 MACBOOK scout[24261] ERROR storageaccounts.py L61: Failed to retrieve activity logs: 'AzureCredentials' object has no attribute 'signed_session' Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/ScoutSuite/providers/azure/facade/storageaccounts.py", line 61, in _get_and_set_activity_logs lambda: list(client.activity_logs.list(filter=logs_filter, select="eventTimestamp, operationName")) File "/usr/local/lib/python3.7/site-packages/ScoutSuite/providers/utils.py", line 24, in run_concurrently return await run_function_concurrently(function) File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, self.kwargs) File "/usr/local/lib/python3.7/site-packages/ScoutSuite/providers/azure/facade/storageaccounts.py", line 61, in lambda: list(client.activity_logs.list(filter=logs_filter, select="eventTimestamp, operationName")) File "/usr/local/lib/python3.7/site-packages/msrest/paging.py", line 143, in next self.advance_page() File "/usr/local/lib/python3.7/site-packages/msrest/paging.py", line 129, in advance_page self._response = self._get_next(self.next_link) File "/usr/local/lib/python3.7/site-packages/azure/mgmt/monitor/operations/activity_logs_operations.py", line 117, in internal_paging request, header_parameters, stream=False, operation_config) File "/usr/local/lib/python3.7/site-packages/msrest/service_client.py", line 336, in send pipeline_response = self.config.pipeline.run(request, kwargs) File "/usr/local/lib/python3.7/site-packages/msrest/pipeline/init.py", line 197, in run return first_node.send(pipeline_request, kwargs) # type: ignore File "/usr/local/lib/python3.7/site-packages/msrest/pipeline/init.py", line 150, in send response = self.next.send(request, *kwargs) File "/usr/local/lib/python3.7/site-packages/msrest/pipeline/requests.py", line 65, in send self._creds.signed_session(session) AttributeError: 'AzureCredentials' object has no attribute 'signed_session' 2020-04-02 16:46:55 MACBOOK scout[24261] ERROR storageaccounts.py L35: Failed to retrieve blob containers: 'ListContainerItems' object is not iterable Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/ScoutSuite/providers/azure/facade/storageaccounts.py", line 35, in get_blob_containers lambda: list(client.blob_containers.list(resource_group_name, storage_account_name)) File "/usr/local/lib/python3.7/site-packages/ScoutSuite/providers/utils.py", line 24, in run_concurrently return await run_function_concurrently(function) File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/concurrent/futures/thread.py", line 57, in run result = self.fn(self.args, **self.kwargs) File "/usr/local/lib/python3.7/site-packages/ScoutSuite/providers/azure/facade/storageaccounts.py", line 35, in lambda: list(client.blob_containers.list(resource_group_name, storage_account_name)) TypeError: 'ListContainerItems' object is not iterable

Please provide:

No JSON File was generated.
These scripts are run behind a proxy environment the proxy variables were set before running the command in CLI.
I have reader permissions on the subscription
The permissions given to me are limited I think in the documents Directory.read was needed but I cannot get that is there a way to run the report partially incase it's a permissions error.

Please let me know if I am doing something wrong or if I need to provide any other details.

x4v13r64 commented 4 years ago

Thanks I'll look into it.

I cannot get that is there a way to run the report partially incase it's a permissions error.

Yes, generally the tool will complain if it can't access something and just move on to the rest. If it fails and crashes we'll have to fix this instance.

x4v13r64 commented 4 years ago

1st instance RedHat linux -

Not sure what you are raising? Looks like updating solved the issue?

2nd Instance RedHat linux -

2020-04-02 16:46:55 MACBOOK scout[24261] ERROR storageaccounts.py L61: Failed to retrieve activity logs: 'AzureCredentials' object has no attribute 'signed_session'

This issue was resolved either in 5.6 or 5.7, what version are you running?

reach4bawer commented 4 years ago

@j4v Please let me know if I can help you with anything.

Yes, generally the tool will complain if it can't access something and just move on to the rest. If it fails and crashes we'll have to fix this instance.

This is what I thought as well.

reach4bawer commented 4 years ago

1st instance RedHat linux -

Not sure what you are raising? Looks like updating solved the issue?

2nd Instance RedHat linux -

2020-04-02 16:46:55 MACBOOK scout[24261] ERROR storageaccounts.py L61: Failed to retrieve activity logs: 'AzureCredentials' object has no attribute 'signed_session'

This issue was resolved either in 5.6 or 5.7, what version are you running?

I have tried both Scout Suite 5.7.0 and Scout Suite 5.8.1 but I get the same result. In both the cases I had to close the script, as the scripts kept showing the same thing

2020-04-02 16:32:23 ip-x.x.x.x.linux.internal scout[27034] INFO Fetching resources for the App Services service

same thing for MAC -

2020-04-03 16:20:17 MACBOOK scout[24261] INFO Fetching resources for the App Services service

I am not sure why it would get stuck at the same point for all the system.

x4v13r64 commented 4 years ago

What's most likely happening is that there's a specific service or resource that's causing issues with the Azure SDKs:

Does the subscription hold a large amount of resources?
It's going to be hard to debug remotly. As a first step could you please launch a scan for each service individually (with the --services flag), and identify which one is making it hand?

reach4bawer commented 4 years ago

I tried this with the --services flag for "aad", "appservice", "arm", "keyvault", "network", "securitycenter", "sqldatabase", "storageaccounts", "virtualmachines" services. I think it is the aad service that gets stuck which could be because we have a lot of active directory data, but given that there is only one rule aad-guest-users.json it does not make a lot of sense as to what might be taking that long. Does the code crawl through the active directory to fetch data?

Update -

I let scout run on one of my systems for 1.5 days and I got a file that was 2.05 GB which might explain why the HTML wasn't loading and why the other systems were also taking forever to complete the scan.

x4v13r64 commented 4 years ago

Thanks for testing this out:

I'll investigate what can be done for the AAD service. There's currently one rule, but we're going to be expanding the support significantly.
The second issue (report size) is a known limitation - https://github.com/nccgroup/ScoutSuite/issues/226. All you can do is limit how many services/subscriptions are scanned.

reach4bawer commented 4 years ago

Thank you @j4v also I when I run it for all the subscriptions I receive the following error -

Traceback (most recent call last): File "/usr/local/lib64/python3.7/site-packages/ScoutSuite/providers/azure/facade/securitycenter.py", line 95, in get_regulatory_compliance_results lambda: list(client.regulatory_compliance_standards.list()) File "/usr/local/lib64/python3.7/site-packages/ScoutSuite/providers/utils.py", line 24, in run_concurrently return await run_function_concurrently(function) File "/usr/lib64/python3.7/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "/usr/local/lib64/python3.7/site-packages/ScoutSuite/providers/azure/facade/securitycenter.py", line 95, in lambda: list(client.regulatory_compliance_standards.list()) File "/usr/local/lib/python3.7/site-packages/msrest/paging.py", line 143, in next self.advance_page() File "/usr/local/lib/python3.7/site-packages/msrest/paging.py", line 129, in advance_page self._response = self._get_next(self.next_link) File "/usr/local/lib/python3.7/site-packages/azure/mgmt/security/operations/_regulatory_compliance_standards_operations.py", line 99, in internal_paging raise exp msrestazure.azure_exceptions.CloudError: Azure Error: Subscription with no standard pricing bundle Message: Regulatory compliance is not supported for subscription 'xyz' as it has no standard pricing bundle

Is there a way to skip this particular test or elegantly handle this so that the report generation does not fail for all the subscriptions ?

x4v13r64 commented 4 years ago

elegantly handle this so that the report generation does not fail for all the subscriptions ?

When I see this error it has no impact on report generation. As is generally the case, Scout complains and just keeps running. Isn't that the case here?

reach4bawer commented 4 years ago

This error generates for 8 subscriptions and then the thing shuts down with a message Killed. On looking at the folder there were no reports stored in the directory.

I used all the same parameters, just replaced the the subscriptions with all subscriptions parameter.

x4v13r64 commented 4 years ago

No additional error? That's not what I'm seeing in our test environment.

If you can provide more info that would be great. If you could email the errors file to scoutsuite@nccgroup.com maybe there's something there to help us debug remotely.

reach4bawer commented 4 years ago

Are there any additional flags that I can use to generate more verbose logs?

x4v13r64 commented 4 years ago

Are there any additional flags that I can use to generate more verbose logs?

--debug

reach4bawer commented 4 years ago

@j4v I used the debug flag already was wondering if there was any other flag that would generate more verbose logs. Removed the logs This has the logs and the error.

x4v13r64 commented 4 years ago

That's odd, in my case it doesn't output Killed and just stop.

reach4bawer commented 4 years ago

Could this be because of the async version?

x4v13r64 commented 4 years ago

Doubt it, what version are you running?

reach4bawer commented 4 years ago

Output for pip freeze for Python 3.7.6

asyncio-throttle==0.1.1

adal==1.2.2
aliyun-python-sdk-actiontrail==2.0.1
aliyun-python-sdk-core==2.13.15
aliyun-python-sdk-core-v3==2.13.11
aliyun-python-sdk-ecs==4.19.2
aliyun-python-sdk-kms==2.10.1
aliyun-python-sdk-ocs==0.0.4
aliyun-python-sdk-ram==3.2.0
aliyun-python-sdk-rds==2.4.5
aliyun-python-sdk-sts==3.0.1
aliyun-python-sdk-vpc==3.0.9
amqp==2.5.2
antlr4-python3-runtime==4.8
applicationinsights==0.11.9
APScheduler==3.6.3
argcomplete==1.11.1
asgiref==3.2.3
asn1crypto==1.3.0
asyncio-throttle==0.1.1
awscli==1.18.15
azure==4.0.0
azure-applicationinsights==0.1.0
azure-batch==8.0.0
azure-cli==2.2.0
azure-cli-command-modules-nspkg==2.0.3
azure-cli-core==2.3.1
azure-cli-nspkg==3.0.4
azure-cli-telemetry==1.0.4
azure-common==1.1.25
azure-core==1.3.0
azure-cosmos==3.1.2
azure-cosmosdb-nspkg==2.0.2
azure-cosmosdb-table==1.0.6
azure-datalake-store==0.0.48
azure-eventgrid==1.3.0
azure-functions-devops-build==0.0.22
azure-graphrbac==0.61.1
azure-keyvault==1.1.0
azure-loganalytics==0.1.0
azure-mgmt==4.0.0
azure-mgmt-advisor==2.0.1
azure-mgmt-apimanagement==0.2.0
azure-mgmt-appconfiguration==0.4.0
azure-mgmt-applicationinsights==0.1.1
azure-mgmt-authorization==0.60.0
azure-mgmt-batch==7.0.0
azure-mgmt-batchai==2.0.0
azure-mgmt-billing==0.2.0
azure-mgmt-botservice==0.2.0
azure-mgmt-cdn==4.1.0rc1
azure-mgmt-cognitiveservices==5.0.0
azure-mgmt-commerce==1.0.1
azure-mgmt-compute==12.0.0
azure-mgmt-consumption==2.0.0
azure-mgmt-containerinstance==1.5.0
azure-mgmt-containerregistry==3.0.0rc11
azure-mgmt-containerservice==8.1.0
azure-mgmt-cosmosdb==0.12.0
azure-mgmt-datafactory==0.6.0
azure-mgmt-datalake-analytics==0.2.1
azure-mgmt-datalake-nspkg==3.0.1
azure-mgmt-datalake-store==0.5.0
azure-mgmt-datamigration==0.1.0
azure-mgmt-deploymentmanager==0.2.0
azure-mgmt-devspaces==0.1.0
azure-mgmt-devtestlabs==2.2.0
azure-mgmt-dns==2.1.0
azure-mgmt-eventgrid==2.2.0
azure-mgmt-eventhub==3.0.0
azure-mgmt-hanaonazure==0.1.1
azure-mgmt-hdinsight==1.3.0
azure-mgmt-imagebuilder==0.2.1
azure-mgmt-iotcentral==2.0.0
azure-mgmt-iothub==0.8.2
azure-mgmt-iothubprovisioningservices==0.2.0
azure-mgmt-keyvault==2.2.0
azure-mgmt-kusto==0.3.0
azure-mgmt-loganalytics==0.2.0
azure-mgmt-logic==3.0.0
azure-mgmt-machinelearningcompute==0.4.1
azure-mgmt-managedservices==1.0.0
azure-mgmt-managementgroups==0.1.0
azure-mgmt-managementpartner==0.1.1
azure-mgmt-maps==0.1.0
azure-mgmt-marketplaceordering==0.1.0
azure-mgmt-media==1.1.1
azure-mgmt-monitor==0.8.0
azure-mgmt-msi==0.2.0
azure-mgmt-netapp==0.7.0
azure-mgmt-network==10.0.0
azure-mgmt-notificationhubs==2.1.0
azure-mgmt-nspkg==3.0.2
azure-mgmt-policyinsights==0.4.0
azure-mgmt-powerbiembedded==2.0.0
azure-mgmt-privatedns==0.1.0
azure-mgmt-rdbms==2.0.0
azure-mgmt-recoveryservices==0.4.0
azure-mgmt-recoveryservicesbackup==0.6.0
azure-mgmt-redis==7.0.0rc1
azure-mgmt-relay==0.1.0
azure-mgmt-reservations==0.6.0
azure-mgmt-resource==8.0.1
azure-mgmt-scheduler==2.0.0
azure-mgmt-search==2.1.0
azure-mgmt-security==0.3.0
azure-mgmt-servicebus==0.6.0
azure-mgmt-servicefabric==0.4.0
azure-mgmt-signalr==0.3.0
azure-mgmt-sql==0.18.0
azure-mgmt-sqlvirtualmachine==0.5.0
azure-mgmt-storage==9.0.0
azure-mgmt-subscription==0.2.0
azure-mgmt-trafficmanager==0.51.0
azure-mgmt-web==0.45.0
azure-multiapi-storage==0.2.4
azure-nspkg==3.0.2
azure-servicebus==0.21.1
azure-servicefabric==6.3.0.0
azure-servicemanagement-legacy==0.20.6
azure-storage-blob==1.5.0
azure-storage-common==1.4.2
azure-storage-file==1.4.0
azure-storage-queue==1.4.0
bcrypt==3.1.7
beautifulsoup4==4.8.2
billiard==3.6.3.0
bonsai==1.2.0
boto==2.49.0
boto3==1.12.35
botocore==1.15.35
cachetools==4.0.0
celery==4.4.1
certifi==2019.11.28
cffi==1.14.0
chardet==3.0.4
cheroot==8.3.0
CherryPy==18.5.0
cherrypy-cors==1.6
Click==7.0
colorama==0.4.3
coloredlogs==10.0
colorhash==1.0.2
configparser==4.0.2
crcmod==1.7
cryptography==2.9
cx-Oracle==7.3.0
dash==1.9.1
dash-core-components==1.8.1
dash-html-components==1.0.2
dash-renderer==1.2.4
dash-table==4.6.1
dnspython==1.16.0
docutils==0.15.2
fabric==2.5.0
Flask==1.1.1
Flask-Compress==1.4.0
Flask-MonitoringDashboard==3.0.8
flask-oidc==1.4.0
Flask-Session==0.3.1
future==0.18.2
gevent==1.4.0
google-api-core==1.16.0
google-api-python-client==1.8.0
google-auth==1.13.1
google-auth-httplib2==0.0.3
google-cloud-container==0.4.0
google-cloud-core==1.3.0
google-cloud-iam==0.3.0
google-cloud-kms==1.3.0
google-cloud-logging==1.15.0
google-cloud-monitoring==0.34.0
google-cloud-resource-manager==0.30.1
google-cloud-storage==1.27.0
google-resumable-media==0.5.0
googleapis-common-protos==1.51.0
greenlet==0.4.15
grpc-google-iam-v1==0.12.3
grpcio==1.27.2
gunicorn==20.0.4
httpagentparser==1.9.0
httplib2==0.17.1
httplib2shim==0.0.3
humanfriendly==8.1
idna==2.9
importlib-metadata==1.6.0
invoke==1.4.1
isodate==0.6.0
itsdangerous==1.1.0
jaraco.classes==3.1.0
jaraco.collections==3.0.0
jaraco.functools==3.0.0
jaraco.text==3.2.0
javaproperties==0.5.1
Jinja2==2.11.1
jmespath==0.9.5
jsmin==2.2.2
jsondiff==1.2.0
knack==0.7.0rc1
kombu==4.6.8
ldap3==2.7
MarkupSafe==1.1.1
meld3==2.0.0
mock==2.0.0
more-itertools==8.2.0
msrest==0.6.11
msrestazure==0.6.3
mysql-connector-python==8.0.19
netaddr==0.7.19
ntlm-auth==1.4.0
numpy==1.18.1
oauth2client==4.1.3
oauthlib==3.1.0
oci==2.12.2
oss2==2.9.1
packaging==20.3
pandas==1.0.1
paramiko==2.7.1
pbr==5.4.4
pip-review==1.0
pkginfo==1.5.0.1
plotly==4.5.3
policyuniverse==1.3.2.1
portalocker==1.6.0
portend==2.6
protobuf==3.11.3
psutil==5.7.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycparser==2.20
pycryptodome==3.9.7
Pygments==2.6.1
PyJWT==1.7.1
PyMySQL==0.9.3
PyNaCl==1.3.0
pyodbc==4.0.30
pyOpenSSL==19.1.0
pyparsing==2.4.6
python-dateutil==2.8.0
pytz==2019.3
PyYAML==5.3.1
redis==3.4.1
requests==2.23.0
requests-ntlm==1.1.0
requests-oauthlib==1.3.0
retrying==1.3.3
rsa==4.0
s3transfer==0.3.3
ScoutSuite==5.8.1
scp==0.13.2
six==1.14.0
smart-open==1.9.0
soupsieve==2.0
SQLAlchemy==1.3.13
sqlitedict==1.6.0
sshtunnel==0.1.5
supervisor==4.1.0
tabulate==0.8.7
tempora==3.0.0
tzlocal==2.0.0
uritemplate==3.0.1
urllib3==1.25.8
vine==1.3.0
vsts==0.1.25
vsts-cd-manager==1.0.2
waitress==1.4.3
websocket-client==0.56.0
Werkzeug==0.16.0
xlrd==1.2.0
XlsxWriter==1.2.8
xmltodict==0.12.0
zc.lockfile==2.0
zipp==3.1.0

x4v13r64 commented 4 years ago

Yeah that looks all right.

reach4bawer commented 4 years ago

Is there something else that I can try to see if we can reach to the core of the problem?

x4v13r64 commented 4 years ago

Well there are 2 separate issues being raised here:

We've identified that for large directories, the AAD service runs for a very long time. I'm not certain there's an easy fix for this, but will look into it.
You're also getting a number of Message: Regulatory compliance is not supported for subscription 'xyz' as it has no standard pricing bundle for subscriptions. While this is normal, it appears that in your case it kills the process (which I can't reproduce).
- Can you try to run against each of the subscriptions, to see if there's a specific one which makes the process get killed?
- If you can also share the full --debug output to scoutsuite@nccgroup.com that would be great. We're potentially missing some info here.

aus commented 4 years ago

I think I am also experiencing the AAD hang. In my case, scout runs for almost exactly one hour then starts spitting out the Access Token missing errors below. Perhaps, the token has expired?

2020-04-15 16:01:09 macbook scout[93674] INFO Launching Scout
2020-04-15 16:01:09 macbook scout[93674] INFO Authenticating to cloud provider
/Users/user/Downloads/ScoutSuite-5.8.1/venv/lib/python3.7/site-packages/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made to host 'READACTED'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning,
2020-04-15 16:01:11 macbook scout[93674] INFO To authenticate to the Resource Manager API, use a web browser to access https://microsoft.com/devicelogin and enter the REDACTED code.
/Users/user/Downloads/ScoutSuite-5.8.1/venv/lib/python3.7/site-packages/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made to host 'READACTED'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning,
/Users/user/Downloads/ScoutSuite-5.8.1/venv/lib/python3.7/site-packages/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made to host 'READACTED'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning,
/Users/user/Downloads/ScoutSuite-5.8.1/venv/lib/python3.7/site-packages/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made to host 'READACTED'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning,
/Users/user/Downloads/ScoutSuite-5.8.1/venv/lib/python3.7/site-packages/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made to host 'READACTED'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning,
/Users/user/Downloads/ScoutSuite-5.8.1/venv/lib/python3.7/site-packages/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made to host 'READACTED'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning,
/Users/user/Downloads/ScoutSuite-5.8.1/venv/lib/python3.7/site-packages/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made to host 'READACTED'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning,
/Users/user/Downloads/ScoutSuite-5.8.1/venv/lib/python3.7/site-packages/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made to host 'READACTED'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning,
/Users/user/Downloads/ScoutSuite-5.8.1/venv/lib/python3.7/site-packages/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made to host 'READACTED'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning,
/Users/user/Downloads/ScoutSuite-5.8.1/venv/lib/python3.7/site-packages/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made to host 'READACTED'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning,
2020-04-15 16:01:53 macbook scout[93674] INFO To authenticate to the Azure Graph API, use a web browser to access https://microsoft.com/devicelogin and enter the REDACTED code.
/Users/user/Downloads/ScoutSuite-5.8.1/venv/lib/python3.7/site-packages/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made to host 'READACTED'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning,
/Users/user/Downloads/ScoutSuite-5.8.1/venv/lib/python3.7/site-packages/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made to host 'READACTED'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning,
/Users/user/Downloads/ScoutSuite-5.8.1/venv/lib/python3.7/site-packages/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made to host 'READACTED'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning,
2020-04-15 16:02:06 macbook scout[93674] INFO No subscription set, inferring ID
2020-04-15 16:02:07 macbook scout[93674] INFO Running against the "REDACTED" subscription
2020-04-15 16:02:07 macbook scout[93674] INFO Gathering data from APIs
2020-04-15 16:02:07 macbook scout[93674] INFO Fetching resources for the AAD service
2020-04-15 16:02:07 macbook scout[93674] INFO Fetching resources for the ARM service
2020-04-15 16:02:07 macbook scout[93674] INFO Fetching resources for the Security Center service
2020-04-15 16:02:07 macbook scout[93674] INFO Fetching resources for the SQL Database service
2020-04-15 16:02:07 macbook scout[93674] INFO Fetching resources for the Storage Accounts service
2020-04-15 16:02:07 macbook scout[93674] INFO Fetching resources for the Key Vault service
2020-04-15 16:02:07 macbook scout[93674] INFO Fetching resources for the Network service
2020-04-15 16:02:07 macbook scout[93674] INFO Fetching resources for the Virtual Machines service
2020-04-15 16:02:07 macbook scout[93674] INFO Fetching resources for the App Services service
2020-04-15 16:02:09 macbook scout[93674] ERROR securitycenter.py L95: Failed to retrieve regulatory compliance standards: Azure Error: Subscription with no standard pricing bundle
Message: Regulatory compliance is not supported for subscription 'REDACTED' as it has no standard pricing bundle
/Users/user/Downloads/ScoutSuite-5.8.1/venv/lib/python3.7/site-packages/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made to host 'REDACTED'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning,
2020-04-15 17:02:09 macbook scout[93674] ERROR aad.py L31: Failed to retrieve user's groups: Access Token missing or malformed.
2020-04-15 17:02:10 macbook scout[93674] ERROR aad.py L31: Failed to retrieve user's groups: Access Token missing or malformed.
2020-04-15 17:02:10 macbook scout[93674] ERROR aad.py L31: Failed to retrieve user's groups: Access Token missing or malformed.
2020-04-15 17:02:11 macbook scout[93674] ERROR aad.py L31: Failed to retrieve user's groups: Access Token missing or malformed.
2020-04-15 17:02:11 macbook scout[93674] ERROR aad.py L31: Failed to retrieve user's groups: Access Token missing or malformed.

The Access Token missing error is repeated for several hours until I kill off the process.

I should also note that I am going through a corporate HTTP proxy that does SSL intercepting (hence the SSL warnings), but I think I've worked out those issues. I'm also authenticating via --user-account-browser

If I run with --services appservice arm keyvault network securitycenter sqldatabase storageaccounts virtualmachines, I don't have any problems.

Let me know if you would like --debug output sent.

x4v13r64 commented 4 years ago

@aus is this also for a very large AAD tenant?

reach4bawer commented 4 years ago

@j4v Please find the attached logs. (apologies for delay) when I ran by skipping the aad I still got large files but I was able to run it for all the subscriptions.

Scout_run_logs.txt

aus commented 4 years ago

@j4v yes. Pretty large.

aus commented 4 years ago

FWIW, I was able to get past the Access Token missing error by using -c authentication instead of --user-account-browser. This allowed the AAD job to complete without the (presumably) token expiration. After several hours and probably ~5.6million requests to graph.windows.net, Scout moved onto to the AAD finalize where it chewed up a single core of my CPU for several hours. During the run, I attached a profiler and saw it was stuck in this loop for a while.

Eventually, it finished. But my JSON is over 1GB. 😄 So I'm currently re-running to format in sqlite.

I also ran into a Out of Memory problem when generating reports for --all-subscriptions. (I have 150+ subscriptions. That quickly chewed through 32GB of memory.) Using the Azure cli, I parsed out my subscriptions IDs and put together a quick bash script to run Scout for each individual subscription ID.

x4v13r64 commented 4 years ago

FWIW, I was able to get past the Access Token missing error by using -c authentication instead of --user-account-browser.

That's likely because the CLI takes care of token refreshes.

I attached a profiler and saw it was stuck in this loop for a while.

Cheers, I'll look at how/whether we can optimize this.

I also ran into a Out of Memory problem when generating reports for --all-subscriptions. (I have 150+ subscriptions.

While it's possible to run Scout against all your subscriptions, if you have a gazillion I wouldn't recommend it, as it makes the results hard to read. Do note that you can run Scout programatically, so you could easily run it against a bunch of subscriptions independently.

I'm currently re-running to format in sqlite

This feature is still in development / highly experimental, so you won't be able to view the report in this format.

x4v13r64 commented 4 years ago

Scout_run_logs.txt

Thanks @reach4bawer, I'll review the exceptions, see if there's anything we can fixed. Looking at it quickly I can see that it wasn't "killed", even though you got a bunch of the Regulatory compliance is not supported for subscription exceptions. I think these two are unrelated, and that you simply got the killed message because your host ran out of memory (@aus I'm guessing you got the same message when you ran out of memory?).

I'm inclined to think that there isn't really an Azure-specific issue here, but you're just running into https://github.com/nccgroup/ScoutSuite/issues/226 because 1) you have massive AAD tenants and 2) you're running Scout against a large number of subscriptions.

In this case, the available options are to skip the AAD service (--skip aad), and to run Scout against a limited set of subscriptions.

We're also looking in whether we can filter-down the users being pulled to only those which we'd like to flag, but the Azure SDKs don't make this trivial.

aus commented 4 years ago

This feature is still in development / highly experimental, so you won't be able to view the report in this format.

No problem! Thanks for the heads up. Is there a recommended way to view large reports right now? I think I read anything over 400MB isn't going to be rendered by the browser. Should I plan on manually querying the JSON / sqlite3 DB?

x4v13r64 commented 4 years ago

If you want to view the HTML report then I'd recommend running Scout against specific subscriptions.

x4v13r64 commented 4 years ago

To pull less users, we could only pull:

Guest users, which are currently the only ones directly used by a AAD rule
Users with IAM-assigned roles

This would allow significantly scoping down the users (as most of the users in large tenants are just synched accounts), but might conflict with future AAD rules.

x4v13r64 commented 4 years ago

@reach4bawer @aus the https://github.com/nccgroup/ScoutSuite/tree/issue/698 branch implements the fixes mentioned in the previous comment.

Could you please test? This should reduce the execution time, as well as the final report size, while including all the relevant information in the report.

reach4bawer commented 4 years ago

Will try and let you know.

x4v13r64 commented 4 years ago

@reach4bawer any updates?

@aus appreciate if you can also test this out.

aus commented 4 years ago

Apologies for the delay. #734 worked nicely. Performance was good, memory didn't balloon and stayed under 1GB. Execution took about 10 minutes. Report was successfully generated on 100K+ resources. Report size was around 140MB. Browser rendered fine.

Really nice work! Thanks again @j4v.

aus commented 4 years ago

One other thing... I wasn't able to test this with --all-subscriptions option. This run (100+ subscriptions) takes understandably longer. Memory is manageable (under 4GB), but after an hour of runtime, I get messages like:

ERROR appservice.py L46: Failed to retrieve web app auth settings: (AuthorizationFailed) The client 'REDACTED' with object id 'REDACTED' does not have authorization to perform action 'Microsoft.Web/sites/config/list/action' over scope '/subscriptions/REDACTED/resourceGroups/REDACTED/providers/Microsoft.Web/sites/REDACTED/config/authsettings' or the scope is invalid. If access was recently granted, please refresh your credentials.

In the past, I've been able to skirt around this by using --cli authentication which does refresh (out of band, I guess). Unfortunately, that's not an option here because it's not picking up all my subscriptions. See #738.

Let me know if you want a separate issue on the auth refresh.

munntjlx commented 4 years ago

I am having this issue too with a service principal. I get the Failed to retrieve web app auth settings:

redentials. 2020-05-13 18:52:04 ScoutSuite scout[116444] ERROR appservice.py L46: Failed to retrieve web app auth settings: (AuthorizationFailed) The client 'redacted' with object id 'redacted' does not have authorization to perform action 'Microsoft.Web/sites/config/list/action' over scope '/subscriptions/redacted/resourceGroups/redacted/providers/Microsoft.Web/sites/jredacted/config/authsettings' or the scope is invalid. If access was recently granted, please refresh your credentials.

x4v13r64 commented 4 years ago

734 worked nicely

@aus cheers for that. Could you please diff the JSON reports, just to confirm it isn't missing any information post-refactoring? I haven't observed anything being lost on my end, but would like to confirm before merging.

See #738.

Will look into it, thank you.

I am having this issue too with a service principal. I get the Failed to retrieve web app auth settings:

@munntjlx this seems like it's unrelated to the current issue? Also it just appears to be an authorization error, not an actual bug. In any even, please open a separate issue if you think it's an actual bug.

reach4bawer commented 4 years ago

@j4v Apologies for the delay.

Ran into few errors while running for all subscriptions and skipping aad.

2020-05-13 21:30:18 server asyncio[19172] ERROR Task exception was never retrieved future: <Task finished coro=<SecurityGroups.fetch_all() done, defined at C:\Users\xyz\Downloads\ScoutSuite-issue-698\ScoutSuite\providers\azure\resources\network\security_groups.py:12> exception=MemoryError()> Traceback (most recent call last): File "C:\Users\xyz\Downloads\ScoutSuite-issue-698\ScoutSuite\providers\azure\resources\network\security_groups.py", line 14, in fetch_all id, network_security_group = self._parse_network_security_group(raw_group) File "C:\Users\xyz\Downloads\ScoutSuite-issue-698\ScoutSuite\providers\azure\resources\network\security_groups.py", line 29, in _parse_network_security_group network_security_group_dict['security_rules'] = self._parse_security_rules(network_security_group) File "C:\Users\xyz\Downloads\ScoutSuite-issue-698\ScoutSuite\providers\azure\resources\network\security_groups.py", line 54, in _parse_security_rules security_rule_id, security_rule_dict = self._parse_security_rule(sr) File "C:\Users\xyz\Downloads\ScoutSuite-issue-698\ScoutSuite\providers\azure\resources\network\security_groups.py", line 88, in _parse_security_rule security_rule_dict['source_ports'] = self._parse_ports(source_port_ranges) File "C:\Users\xyz\Downloads\ScoutSuite-issue-698\ScoutSuite\providers\azure\resources\network\security_groups.py", line 111, in _parse_ports ports.add(p) MemoryError 2020-05-13 21:30:18 server scout[19172] ERROR virtualmachines.py L19: Failed to retrieve virtual machines: 2020-05-13 21:30:18 server asyncio[19172] ERROR Task exception was never retrieved future: <Task finished coro=<Disks.fetch_all() done, defined at C:\Users\xyz\Downloads\ScoutSuite-issue-698\ScoutSuite\providers\azure\resources\virtualmachines\disks.py:12> exception=MemoryError()> Traceback (most recent call last): File "C:\Users\xyz\Downloads\ScoutSuite-issue-698\ScoutSuite\providers\azure\resources\virtualmachines\disks.py", line 15, in fetch_all self[id] = disk MemoryError 2020-05-13 21:40:14 server scout[19172] ERROR storageaccounts.py L61: Failed to retrieve activity logs: Error occurred in request., ChunkedEncodingError: ('Connection broken: OSError("(10054, \'WSAECONNRESET\')")', OSError("(10054, 'WSAECONNRESET')"))

This was running for 200 subscriptions and I have 16 gigs of ram on the windows device.

The good news was that the script was able to handle the error much more smoothly and despite of the the permission errors it kept running. Will try and post the log data soon.

x4v13r64 commented 4 years ago

Thanks @reach4bawer, could you test with AAD? That's what I'm trying to fix after all.

reach4bawer commented 4 years ago

@j4v My system windows system crashes every time I try that. I can try to do it for a single subscription but not for all the subscription.

Is there a way to have this dump the information in some database or file instead of keeping everything in the memory ? I tried limiting the number of workers but even that didn't work too well.

x4v13r64 commented 4 years ago

@j4v My system windows system crashes every time I try that. I can try to do it for a single subscription but not for all the subscription.

Even when running the https://github.com/nccgroup/ScoutSuite/pull/734 branch? How many subscriptions do you have???

Is there a way to have this dump the information in some database or file instead of keeping everything in the memory ?

Not currently.

I tried limiting the number of workers but even that didn't work too well.

That will make the execution run slower but won't change how much data is stored in memory.

reach4bawer commented 4 years ago

Yes i used that branch to run the code. Downloaded it and ran the scout.py file to do the scans. Have around 220 subscriptions.

x4v13r64 commented 4 years ago

I'm closing this issue as I believe https://github.com/nccgroup/ScoutSuite/pull/734 resolves the underlying problem, which was that for very large tenants, information for all the users was being stored in the report, even though these users didn't have any roles assigned within the tenant or subscriptions.

There is still the underlying issue of handling very large cloud accounts, as tracked under https://github.com/nccgroup/ScoutSuite/issues/226. This issue seems to have been at play in this issue, but not being able to scan hundreds of subscriptions at once is an acceptable limitation at this time.

mortaelth commented 4 years ago

it should be reopened: It happened to when testing only a single subscription.

it generated 1.3GB js file (scoutsuite-report/scoutsuite-results/scountsuite_results_azure-tenant-xxx)

nearly 100% of the content of the .js file are listed ports in fields in source_ports or destination_ports. Whenever there is Azure NSG with ports defined as "Any", it will put all numbers between 1 and 65535 to the json. And because having "Any" is very common (the 3 default rules in every NSG which cannot be deleted / altered have these), even for small number of NSGs, it generates huge .js file

the json path example of such occurrence:

services.network.subscriptions.80a445bcxxx.security_groups.0312dddfd8be4025ec6958161xxx.security_rules./subscriptions/80a445bcxxx/resourceGroups/imp-prodeu/providers/Microsoft.Network/networkSecurityGroups/imp-prodeu-nsg/defaultSecurityRules/AllowAzureLoadBalancerInBound.destination_ports

x4v13r64 commented 4 years ago

nearly 100% of the content of the .js file are listed ports in fields in source_ports or destination_ports

Thanks for raising this. I remember seeing that code poorly written but didn't think it would have such an impact. Tracking under https://github.com/nccgroup/ScoutSuite/issues/793.

nccgroup / ScoutSuite

Azure AAD Hanging & Process Getting Killed #698

Update -

734 worked nicely