search

prowler-cloud / prowler

Prowler is an Open Source Security tool for AWS, Azure, GCP and Kubernetes to do security assessments, audits, incident response, compliance, continuous monitoring, hardening and forensics readiness. Includes CIS, NIST 800, NIST CSF, CISA, FedRAMP, PCI-DSS, GDPR, HIPAA, FFIEC, SOC2, GXP, Well-Architected Security, ENS and more
https://prowler.com
Apache License 2.0
10.84k stars 1.54k forks source link

[Bug]: Scan was aborted (Azure) #2088

Closed njmulsqb closed 1 year ago

njmulsqb commented 1 year ago

Steps to Reproduce

I used prowler azure both with --az-cli-login and --browser-auth but each time it fails

Expected behavior

It should run fine as I ran scoutsuite and it worked fine

Actual Result with Screenshots or Logs

Executing 20 checks, please wait...

Something went wrong in defender_ensure_defender_for_app_services_is_on, please use --log-level ERROR Something went wrong in defender_ensure_defender_for_arm_is_on, please use --log-level ERROR Something went wrong in defender_ensure_defender_for_azure_sql_databases_is_on, please use --log-level ERROR Something went wrong in defender_ensure_defender_for_containers_is_on, please use --log-level ERROR Something went wrong in defender_ensure_defender_for_cosmosdb_is_on, please use --log-level ERROR Something went wrong in defender_ensure_defender_for_databases_is_on, please use --log-level ERROR Something went wrong in defender_ensure_defender_for_dns_is_on, please use --log-level ERROR Something went wrong in defender_ensure_defender_for_keyvault_is_on, please use --log-level ERROR Something went wrong in defender_ensure_defender_for_os_relational_databases_is_on, please use --log-level ERROR Something went wrong in defender_ensure_defender_for_server_is_on, please use --log-level ERROR Something went wrong in defender_ensure_defender_for_sql_servers_is_on, please use --log-level ERROR Something went wrong in defender_ensure_defender_for_storage_is_on, please use --log-level ERROR 2023-03-14 19:39:22,136 [File: check.py:422] [Module: check] CRITICAL: Check 'iam_subscription_roles_owner_custom_not_created' was not found for the AZURE provider -> Scan was aborted! |▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉⚠︎ | (!) 12/20 [60%] in 0.2s

How did you install Prowler?

From pip package (pip install prowler)

Environment Resource

Python 10

OS used

Windows 11

Prowler version

3.2.4

Pip version

23.0.1

Context

No response

n4ch04 commented 1 year ago

Hi @njmulsqb, We have been facing some weird behaviours with Windows, it seems that deletes files or changes paths. Can you try to list checks prowler azure --list-checks to see if you can list the check? Also, can you run the tool with the same options appending the --log-level ERROR flag to get more info?

njmulsqb commented 1 year ago

Hi @n4ch04, yes I can list the checks here's the output 

PS C:\Users\saqicnaj> prowler azure --list-checks
                         _
 _ __  _ __ _____      _| | ___ _ __
| '_ \| '__/ _ \ \ /\ / / |/ _ \ '__|
| |_) | | | (_) \ V  V /| |  __/ |
| .__/|_|  \___/ \_/\_/ |_|\___|_|v3.2.4
|_| the handy cloud security tool

Date: 2023-03-14 19:51:28

[defender_ensure_defender_for_app_services_is_on] Ensure That Microsoft Defender for App Services Is Set To 'On'  - defender [high]
[defender_ensure_defender_for_arm_is_on] Ensure That Microsoft Defender for Azure Resource Manager Is Set To 'On'  - defender [high]
[defender_ensure_defender_for_azure_sql_databases_is_on] Ensure That Microsoft Defender for Azure SQL Databases Is Set To 'On'   - defender [high]
[defender_ensure_defender_for_containers_is_on] Ensure That Microsoft Defender for Containers Is Set To 'On'  - defender
 [high]
[defender_ensure_defender_for_cosmosdb_is_on] Ensure That Microsoft Defender for Cosmos DB Is Set To 'On'  - defender [high]
[defender_ensure_defender_for_databases_is_on] Ensure That Microsoft Defender for Databases Is Set To 'On'  - defender [high]
[defender_ensure_defender_for_dns_is_on] Ensure That Microsoft Defender for DNS Is Set To 'On'  - defender [high]
[defender_ensure_defender_for_keyvault_is_on] Ensure That Microsoft Defender for KeyVault Is Set To 'On'  - defender [high]
[defender_ensure_defender_for_os_relational_databases_is_on] Ensure That Microsoft Defender for Open-Source Relational Databases Is Set To 'On'  - defender [high]
[defender_ensure_defender_for_server_is_on] Ensure That Microsoft Defender for Servers Is Set to 'On' - defender [high]
[defender_ensure_defender_for_sql_servers_is_on] Ensure That Microsoft Defender for SQL Servers on Machines Is Set To 'On'  - defender [high]
[defender_ensure_defender_for_storage_is_on] Ensure That Microsoft Defender for Storage Is Set To 'On'  - defender [high]
[iam_subscription_roles_owner_custom_not_created] Ensure that no custom subscription owner roles are created - iam [high]
[storage_blob_public_access_level_is_disabled] Ensure that the 'Public access level' is set to 'Private (no anonymous access)' for all blob containers in your storage account - storage [medium]
[storage_default_network_access_rule_is_denied] Ensure Default Network Access Rule for Storage Accounts is Set to Deny -
 storage [medium]
[storage_ensure_azure_services_are_trusted_to_access_is_enabled] Ensure that 'Allow trusted Microsoft services to access this storage account' is enabled for storage accounts - storage [medium]
[storage_ensure_encryption_with_customer_managed_keys] Ensure that your Microsoft Azure Storage accounts are using Customer Managed Keys (CMKs) instead of Microsoft Managed Keys - storage [high]
[storage_ensure_minimum_tls_version_12] Ensure the 'Minimum TLS version' for storage accounts is set to 'Version 1.2' - storage [medium]
[storage_infrastructure_encryption_is_enabled] Ensure that 'Enable Infrastructure Encryption' for Each Storage Account in Azure Storage is Set to 'enabled'  - storage [low]
[storage_secure_transfer_required_is_enabled] Ensure that all data transferred between clients and your Azure Storage account is encrypted using the HTTPS protocol. - storage [medium]

There are 20 available checks.

and with --log-level ERROR it says to az login but it is still done, I did again but didnt work either, output:

2023-03-14 19:53:17,156 [File: azure_provider.py:104]   [Module: azure_provider]         ERROR: Provided identity does not have permissions to access AAD to retrieve tenant domain
2023-03-14 19:53:17,157 [File: azure_provider.py:107]   [Module: azure_provider]         ERROR: ClientAuthenticationError[99] -- DefaultAzureCredential failed to retrieve a token from the included credentials.
Attempted credentials:
        AzureCliCredential: Please run 'az login' to set up an account
2023-03-14 19:53:17,985 [File: azure_provider.py:129]   [Module: azure_provider]         ERROR: Provided identity does not have permissions to access AAD to retrieve user's metadata
2023-03-14 19:53:17,986 [File: azure_provider.py:132]   [Module: azure_provider]         ERROR: ClientAuthenticationError[125] -- DefaultAzureCredential failed to retrieve a token from the included credentials.
Attempted credentials:
        AzureCliCredential: Please run 'az login' to set up an account
Executing 20 checks, please wait...

2023-03-14 19:53:21,205 [File: defender_service.py:32]  [Module: defender_service]       ERROR: TypeError[26]: SecurityCenter.__init__() missing 1 required positional argument: 'asc_location'
2023-03-14 19:53:21,205 [File: defender_service.py:56]  [Module: defender_service]       ERROR: AttributeError[42]: 'NoneType' object has no attribute 'items'
Something went wrong in defender_ensure_defender_for_app_services_is_on, please use --log-level ERROR
2023-03-14 19:53:21,209 [File: check.py:329]    [Module: check]  ERROR: defender_ensure_defender_for_app_services_is_on -- AttributeError[8]: 'NoneType' object has no attribute 'items'
Something went wrong in defender_ensure_defender_for_arm_is_on, please use --log-level ERROR
2023-03-14 19:53:21,215 [File: check.py:329]    [Module: check]  ERROR: defender_ensure_defender_for_arm_is_on -- AttributeError[8]: 'NoneType' object has no attribute 'items'
Something went wrong in defender_ensure_defender_for_azure_sql_databases_is_on, please use --log-level ERROR
2023-03-14 19:53:21,218 [File: check.py:329]    [Module: check]  ERROR: defender_ensure_defender_for_azure_sql_databases_is_on -- AttributeError[8]: 'NoneType' object has no attribute 'items'
Something went wrong in defender_ensure_defender_for_containers_is_on, please use --log-level ERROR
2023-03-14 19:53:21,221 [File: check.py:329]    [Module: check]  ERROR: defender_ensure_defender_for_containers_is_on -- AttributeError[8]: 'NoneType' object has no attribute 'items'
Something went wrong in defender_ensure_defender_for_cosmosdb_is_on, please use --log-level ERROR
2023-03-14 19:53:21,224 [File: check.py:329]    [Module: check]  ERROR: defender_ensure_defender_for_cosmosdb_is_on -- AttributeError[8]: 'NoneType' object has no attribute 'items'
Something went wrong in defender_ensure_defender_for_databases_is_on, please use --log-level ERROR
2023-03-14 19:53:21,227 [File: check.py:329]    [Module: check]  ERROR: defender_ensure_defender_for_databases_is_on -- AttributeError[8]: 'NoneType' object has no attribute 'items'
Something went wrong in defender_ensure_defender_for_dns_is_on, please use --log-level ERROR
2023-03-14 19:53:21,232 [File: check.py:329]    [Module: check]  ERROR: defender_ensure_defender_for_dns_is_on -- AttributeError[8]: 'NoneType' object has no attribute 'items'
Something went wrong in defender_ensure_defender_for_keyvault_is_on, please use --log-level ERROR
2023-03-14 19:53:21,235 [File: check.py:329]    [Module: check]  ERROR: defender_ensure_defender_for_keyvault_is_on -- AttributeError[8]: 'NoneType' object has no attribute 'items'
Something went wrong in defender_ensure_defender_for_os_relational_databases_is_on, please use --log-level ERROR
2023-03-14 19:53:21,239 [File: check.py:329]    [Module: check]  ERROR: defender_ensure_defender_for_os_relational_databases_is_on -- AttributeError[8]: 'NoneType' object has no attribute 'items'
Something went wrong in defender_ensure_defender_for_server_is_on, please use --log-level ERROR
2023-03-14 19:53:21,243 [File: check.py:329]    [Module: check]  ERROR: defender_ensure_defender_for_server_is_on -- AttributeError[8]: 'NoneType' object has no attribute 'items'
Something went wrong in defender_ensure_defender_for_sql_servers_is_on, please use --log-level ERROR
2023-03-14 19:53:21,250 [File: check.py:329]    [Module: check]  ERROR: defender_ensure_defender_for_sql_servers_is_on -- AttributeError[8]: 'NoneType' object has no attribute 'items'
Something went wrong in defender_ensure_defender_for_storage_is_on, please use --log-level ERROR
2023-03-14 19:53:21,254 [File: check.py:329]    [Module: check]  ERROR: defender_ensure_defender_for_storage_is_on -- AttributeError[8]: 'NoneType' object has no attribute 'items'
2023-03-14 19:53:21,261 [File: check.py:422]    [Module: check]  CRITICAL: Check 'iam_subscription_roles_owner_custom_not_created' was not found for the AZURE provider
-> Scan was aborted! |▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉⚠︎
n4ch04 commented 1 year ago

Hi @njmulsqb ,

We are going to review this and let you know asap,

Thanks for using Prowler !!!

n4ch04 commented 1 year ago

Hi @njmulsqb,

After testing it seems that we have identified where the issue comes from. Can you check which version of the azure-mgmt-security library you have installed ? You can do it with pip show azure-mgmt-security

If the version is not 3.0.0 please update it to that version, and try again

njmulsqb commented 1 year ago

Hi @n4ch04

I have upgraded it but it didnt help, here's the terminal log from upgradation to running prowler

pip show azure-mgmt-security
Name: azure-mgmt-security
Version: 1.0.0
Summary: Microsoft Azure Security Center Management Client Library for Python
Home-page: https://github.com/Azure/azure-sdk-for-python
Author: Microsoft Corporation
Author-email: azpysdkhelp@microsoft.com
License: MIT License
Location: c:\users\saqicnaj\appdata\local\programs\python\python310\lib\site-packages
Requires: azure-common, azure-mgmt-core, msrest
Required-by: prowler, ScoutSuite
PS C:\Users\saqicnaj\OneDrive - Constellation HomeBuilder Systems\Cloud Tools> pip install azure-mgmt-security --upgrade

Requirement already satisfied: azure-mgmt-security in c:\users\saqicnaj\appdata\local\programs\python\python310\lib\site-packages (1.0.0)
Collecting azure-mgmt-security
  Using cached azure_mgmt_security-3.0.0-py3-none-any.whl (941 kB)
Requirement already satisfied: msrest>=0.7.1 in c:\users\saqicnaj\appdata\local\programs\python\python310\lib\site-packages (from azure-mgmt-security) (0.7.1)
Requirement already satisfied: azure-mgmt-core<2.0.0,>=1.3.2 in c:\users\saqicnaj\appdata\local\programs\python\python310\lib\site-packages (from azure-mgmt-security) (1.3.2)
Requirement already satisfied: azure-common~=1.1 in c:\users\saqicnaj\appdata\local\programs\python\python310\lib\site-packages (from azure-mgmt-security) (1.1.28)
Requirement already satisfied: azure-core<2.0.0,>=1.24.0 in c:\users\saqicnaj\appdata\local\programs\python\python310\lib\site-packages (from azure-mgmt-core<2.0.0,>=1.3.2->azure-mgmt-security) (1.26.3)
Requirement already satisfied: requests-oauthlib>=0.5.0 in c:\users\saqicnaj\appdata\local\programs\python\python310\lib\site-packages (from msrest>=0.7.1->azure-mgmt-security) (1.3.1)
Requirement already satisfied: certifi>=2017.4.17 in c:\users\saqicnaj\appdata\local\programs\python\python310\lib\site-packages (from msrest>=0.7.1->azure-mgmt-security) (2022.12.7)
Requirement already satisfied: isodate>=0.6.0 in c:\users\saqicnaj\appdata\local\programs\python\python310\lib\site-packages (from msrest>=0.7.1->azure-mgmt-security) (0.6.1)
Requirement already satisfied: requests~=2.16 in c:\users\saqicnaj\appdata\local\programs\python\python310\lib\site-packages (from msrest>=0.7.1->azure-mgmt-security) (2.28.2)
Requirement already satisfied: typing-extensions>=4.0.1 in c:\users\saqicnaj\appdata\local\programs\python\python310\lib\site-packages (from azure-core<2.0.0,>=1.24.0->azure-mgmt-core<2.0.0,>=1.3.2->azure-mgmt-security) (4.5.0)
Requirement already satisfied: six>=1.11.0 in c:\users\saqicnaj\appdata\local\programs\python\python310\lib\site-packages (from azure-core<2.0.0,>=1.24.0->azure-mgmt-core<2.0.0,>=1.3.2->azure-mgmt-security) (1.16.0)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in c:\users\saqicnaj\appdata\local\programs\python\python310\lib\site-packages (from requests~=2.16->msrest>=0.7.1->azure-mgmt-security) (1.26.15)
Requirement already satisfied: charset-normalizer<4,>=2 in c:\users\saqicnaj\appdata\local\programs\python\python310\lib\site-packages (from requests~=2.16->msrest>=0.7.1->azure-mgmt-security) (3.1.0)
Requirement already satisfied: idna<4,>=2.5 in c:\users\saqicnaj\appdata\local\programs\python\python310\lib\site-packages (from requests~=2.16->msrest>=0.7.1->azure-mgmt-security) (3.4)
Requirement already satisfied: oauthlib>=3.0.0 in c:\users\saqicnaj\appdata\local\programs\python\python310\lib\site-packages (from requests-oauthlib>=0.5.0->msrest>=0.7.1->azure-mgmt-security) (3.2.2)
Installing collected packages: azure-mgmt-security
  Attempting uninstall: azure-mgmt-security
    Found existing installation: azure-mgmt-security 1.0.0
    Uninstalling azure-mgmt-security-1.0.0:
      Successfully uninstalled azure-mgmt-security-1.0.0
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
scoutsuite 5.12.0 requires azure-mgmt-security==1.0.0, but you have azure-mgmt-security 3.0.0 which is incompatible.
prowler 3.2.4 requires azure-identity==1.12.0, but you have azure-identity 1.5.0 which is incompatible.
prowler 3.2.4 requires azure-mgmt-authorization==3.0.0, but you have azure-mgmt-authorization 1.0.0 which is incompatible.
prowler 3.2.4 requires azure-mgmt-storage==21.0.0, but you have azure-mgmt-storage 16.0.0 which is incompatible.
Successfully installed azure-mgmt-security-3.0.0
PS C:\Users\saqicnaj\OneDrive - Constellation HomeBuilder Systems\Cloud Tools>  prowler azure --az-cli-auth --subscription-ids 0c43b29a-25a1-41f0-b601-11d3181e8f1d
                         _
 _ __  _ __ _____      _| | ___ _ __
| '_ \| '__/ _ \ \ /\ / / |/ _ \ '__|
| |_) | | | (_) \ V  V /| |  __/ |
| .__/|_|  \___/ \_/\_/ |_|\___|_|v3.2.4
|_| the handy cloud security tool

Date: 2023-03-15 15:31:49

Executing 20 checks, please wait...

Something went wrong in defender_ensure_defender_for_app_services_is_on, please use --log-level ERROR
Something went wrong in defender_ensure_defender_for_arm_is_on, please use --log-level ERROR
Something went wrong in defender_ensure_defender_for_azure_sql_databases_is_on, please use --log-level ERROR
Something went wrong in defender_ensure_defender_for_containers_is_on, please use --log-level ERROR
Something went wrong in defender_ensure_defender_for_cosmosdb_is_on, please use --log-level ERROR
Something went wrong in defender_ensure_defender_for_databases_is_on, please use --log-level ERROR
Something went wrong in defender_ensure_defender_for_dns_is_on, please use --log-level ERROR
Something went wrong in defender_ensure_defender_for_keyvault_is_on, please use --log-level ERROR
Something went wrong in defender_ensure_defender_for_os_relational_databases_is_on, please use --log-level ERROR
Something went wrong in defender_ensure_defender_for_server_is_on, please use --log-level ERROR
Something went wrong in defender_ensure_defender_for_sql_servers_is_on, please use --log-level ERROR
Something went wrong in defender_ensure_defender_for_storage_is_on, please use --log-level ERROR
2023-03-15 15:32:18,403 [File: check.py:422]    [Module: check]  CRITICAL: Check 'iam_subscription_roles_owner_custom_not_created' was not found for the AZURE provider
-> Scan was aborted! |▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉⚠︎               | (!) 12/20 [60%] in 20.5s
PS C:\Users\saqicnaj\OneDrive - Constellation HomeBuilder Systems\Cloud Tools> pip show azure-mgmt-security
Name: azure-mgmt-security
Version: 3.0.0
Summary: Microsoft Azure Security Center Management Client Library for Python
Home-page: https://github.com/Azure/azure-sdk-for-python
Author: Microsoft Corporation
Author-email: azpysdkhelp@microsoft.com
License: MIT License
Location: c:\users\saqicnaj\appdata\local\programs\python\python310\lib\site-packages
Requires: azure-common, azure-mgmt-core, msrest
Required-by: prowler, ScoutSuite
PS C:\Users\saqicnaj\OneDrive - Constellation HomeBuilder Systems\Cloud Tools>
n4ch04 commented 1 year ago

@njmulsqb It seems that something is broken in your environment regarding dependences, can you create a new fresh virtualenv and install the tool into the virtualenv from scratch ?

I've face exactly the same issue you have downgrading the library from version 3.0.0 to version 1.0.0 (ScoutSuite uses the version 1.0.0) but using version 3.0.0 solves the issue.

Please refer to our official doc to get more details: https://docs.prowler.cloud/en/latest/#installation

n4ch04 commented 1 year ago

@njmulsqb If you have problems with creating a new virtualenv by yourself I recommend you to download the tool directly from the repo following the Github installation process: From the docs (https://docs.prowler.cloud/en/latest/#installation) Ensure you have git, python 3.9 and poetry installed (pip install poetry) Then 

git clone https://github.com/prowler-cloud/prowler
cd prowler
poetry shell
poetry install
python prowler.py -v
njmulsqb commented 1 year ago

Hi @n4ch04,

Unfortunately, it is getting unnecessarily annoying to set up prowler, I followed the git repo and poetry method but still it is failing

(prowler-py3.10) PS C:\Users\saqicnaj\OneDrive - Constellation HomeBuilder Systems\Cloud Tools\prowler> python3 .\prowler.py
Python was not found; run without arguments to install from the Microsoft Store, or disable this shortcut from Settings > Manage App Execution Aliases.
(prowler-py3.10) PS C:\Users\saqicnaj\OneDrive - Constellation HomeBuilder Systems\Cloud Tools\prowler> python .\prowler.py
Traceback (most recent call last):
  File "C:\Users\saqicnaj\OneDrive - Constellation HomeBuilder Systems\Cloud Tools\prowler\prowler.py", line 5, in <module>
    from prowler.__main__ import prowler
  File "C:\Users\saqicnaj\OneDrive - Constellation HomeBuilder Systems\Cloud Tools\prowler\prowler\__main__.py", line 7, in <module>
    from prowler.lib.check.check import (
  File "C:\Users\saqicnaj\OneDrive - Constellation HomeBuilder Systems\Cloud Tools\prowler\prowler\lib\check\check.py", line 7, in <module>
    from resource import RLIMIT_NOFILE, getrlimit
ModuleNotFoundError: No module named 'resource'
(prowler-py3.10) PS C:\Users\saqicnaj\OneDrive - Constellation HomeBuilder Systems\Cloud Tools\prowler>
n4ch04 commented 1 year ago

Hi @njmulsqb, that error comes from a ulimit testing that is not supported in Windows, it should be fixed now. If you perform a git pull into the poetry venv it should work. We really appreciate your feedback and we work every day in improving installation process and usage of the tool Your previous errors regarding dependencies comes from that you installed another tool that uses outdated versions of the libraries prior using Prowler, the installation with pip in a new fresh environment should work fine. Thanks for using Prowler !!

njmulsqb commented 1 year ago

Hi @n4ch04,

Took fresh clone and installed everything again but

Date: 2023-03-16 15:45:06

Traceback (most recent call last):
  File "C:\Users\saqicnaj\AppData\Local\pypoetry\Cache\virtualenvs\prowler-aEpmDHki-py3.10\Scripts\prowler", line 6, in <module>
    sys.exit(prowler())
  File "D:\Cloud Tools\prowler\prowler\__main__.py", line 159, in prowler
    findings = execute_checks(
  File "D:\Cloud Tools\prowler\prowler\lib\check\check.py", line 360, in execute_checks
    from resource import RLIMIT_NOFILE, getrlimit
ModuleNotFoundError: No module named 'resource'
MrCloudSec commented 1 year ago

Hi @njmulsqb, can you try it again with the new version 3.3.0 of Prowler? Thank you.

njmulsqb commented 1 year ago

Some improvement is here (this time the errors are different)

python prowler.py azure --az-cli-auth --subscription-ids 0c43b29a-25a1-41f0-b601-11d3181e8f1c --log-level ERROR
                         _
 _ __  _ __ _____      _| | ___ _ __
| '_ \| '__/ _ \ \ /\ / / |/ _ \ '__|
| |_) | | | (_) \ V  V /| |  __/ |
| .__/|_|  \___/ \_/\_/ |_|\___|_|v3.3.0
|_| the handy cloud security tool

Date: 2023-03-17 11:15:16

2023-03-17 11:15:21,047 [File: azure_provider.py:104]   [Module: azure_provider]         ERROR: Provided identity does not have permissions to access AAD to retrieve tenant domain
2023-03-17 11:15:21,047 [File: azure_provider.py:107]   [Module: azure_provider]         ERROR: ClientAuthenticationError[99] -- DefaultAzureCredential failed to retrieve a token from the included credentials.
Attempted credentials:
        AzureCliCredential: Please run 'az login' to set up an account
To mitigate this issue, please refer to the troubleshooting guidelines here at https://aka.ms/azsdk/python/identity/defaultazurecredential/troubleshoot.
2023-03-17 11:15:22,953 [File: azure_provider.py:129]   [Module: azure_provider]         ERROR: Provided identity does not have permissions to access AAD to retrieve user's metadata
2023-03-17 11:15:22,953 [File: azure_provider.py:132]   [Module: azure_provider]         ERROR: ClientAuthenticationError[125] -- DefaultAzureCredential failed to retrieve a token from the included credentials.
Attempted credentials:
        AzureCliCredential: Please run 'az login' to set up an account
To mitigate this issue, please refer to the troubleshooting guidelines here at https://aka.ms/azsdk/python/identity/defaultazurecredential/troubleshoot.
Executing 20 checks, please wait...

Something went wrong in defender_ensure_defender_for_app_services_is_on, please use --log-level ERROR
2023-03-17 11:16:01,705 [File: check.py:332]    [Module: check]  ERROR: defender_ensure_defender_for_app_services_is_on -- KeyError[13]: 'AppServices'
Something went wrong in defender_ensure_defender_for_arm_is_on, please use --log-level ERROR
2023-03-17 11:16:01,719 [File: check.py:332]    [Module: check]  ERROR: defender_ensure_defender_for_arm_is_on -- KeyError[12]: 'Arm'
Something went wrong in defender_ensure_defender_for_azure_sql_databases_is_on, please use --log-level ERROR
2023-03-17 11:16:01,750 [File: check.py:332]    [Module: check]  ERROR: defender_ensure_defender_for_azure_sql_databases_is_on -- KeyError[12]: 'SqlServers'
Something went wrong in defender_ensure_defender_for_containers_is_on, please use --log-level ERROR
2023-03-17 11:16:01,781 [File: check.py:332]    [Module: check]  ERROR: defender_ensure_defender_for_containers_is_on -- KeyError[12]: 'Containers'
Something went wrong in defender_ensure_defender_for_cosmosdb_is_on, please use --log-level ERROR
2023-03-17 11:16:01,812 [File: check.py:332]    [Module: check]  ERROR: defender_ensure_defender_for_cosmosdb_is_on -- KeyError[12]: 'CosmosDbs'
Something went wrong in defender_ensure_defender_for_databases_is_on, please use --log-level ERROR
2023-03-17 11:16:01,845 [File: check.py:332]    [Module: check]  ERROR: defender_ensure_defender_for_databases_is_on -- KeyError[12]: 'SqlServers'
Something went wrong in defender_ensure_defender_for_dns_is_on, please use --log-level ERROR
2023-03-17 11:16:01,873 [File: check.py:332]    [Module: check]  ERROR: defender_ensure_defender_for_dns_is_on -- KeyError[13]: 'Dns'
Something went wrong in defender_ensure_defender_for_keyvault_is_on, please use --log-level ERROR
2023-03-17 11:16:01,892 [File: check.py:332]    [Module: check]  ERROR: defender_ensure_defender_for_keyvault_is_on -- KeyError[13]: 'KeyVaults'
Something went wrong in defender_ensure_defender_for_os_relational_databases_is_on, please use --log-level ERROR
2023-03-17 11:16:01,909 [File: check.py:332]    [Module: check]  ERROR: defender_ensure_defender_for_os_relational_databases_is_on -- KeyError[13]: 'OpenSourceRelationalDatabases'
Something went wrong in defender_ensure_defender_for_server_is_on, please use --log-level ERROR
2023-03-17 11:16:01,940 [File: check.py:332]    [Module: check]  ERROR: defender_ensure_defender_for_server_is_on -- KeyError[13]: 'VirtualMachines'
Something went wrong in defender_ensure_defender_for_sql_servers_is_on, please use --log-level ERROR
2023-03-17 11:16:01,955 [File: check.py:332]    [Module: check]  ERROR: defender_ensure_defender_for_sql_servers_is_on -- KeyError[13]: 'SqlServerVirtualMachines'
Something went wrong in defender_ensure_defender_for_storage_is_on, please use --log-level ERROR
2023-03-17 11:16:01,987 [File: check.py:332]    [Module: check]  ERROR: defender_ensure_defender_for_storage_is_on -- KeyError[13]: 'StorageAccounts'
2023-03-17 11:16:28,826 [File: models.py:97]    [Module: models]         ERROR: ValidationError[57]: 1 validation error for Azure_Check_Output_CSV
tenant_domain
  none is not an allowed value (type=type_error.none.not_allowed)
2023-03-17 11:16:28,826 [File: outputs.py:174]  [Module: outputs]        ERROR: TypeError[137]: cannot unpack non-iterable NoneType object
2023-03-17 11:17:02,505 [File: models.py:97]    [Module: models]         ERROR: ValidationError[57]: 1 validation error for Azure_Check_Output_CSV
tenant_domain
  none is not an allowed value (type=type_error.none.not_allowed)
2023-03-17 11:17:02,505 [File: outputs.py:174]  [Module: outputs]        ERROR: TypeError[137]: cannot unpack non-iterable NoneType object
2023-03-17 11:17:02,650 [File: models.py:97]    [Module: models]         ERROR: ValidationError[57]: 1 validation error for Azure_Check_Output_CSV
tenant_domain
  none is not an allowed value (type=type_error.none.not_allowed)
2023-03-17 11:17:02,650 [File: outputs.py:174]  [Module: outputs]        ERROR: TypeError[137]: cannot unpack non-iterable NoneType object
2023-03-17 11:17:02,804 [File: models.py:97]    [Module: models]         ERROR: ValidationError[57]: 1 validation error for Azure_Check_Output_CSV
tenant_domain
  none is not an allowed value (type=type_error.none.not_allowed)
2023-03-17 11:17:02,804 [File: outputs.py:174]  [Module: outputs]        ERROR: TypeError[137]: cannot unpack non-iterable NoneType object
2023-03-17 11:17:02,930 [File: models.py:97]    [Module: models]         ERROR: ValidationError[57]: 1 validation error for Azure_Check_Output_CSV
tenant_domain
  none is not an allowed value (type=type_error.none.not_allowed)
2023-03-17 11:17:02,930 [File: outputs.py:174]  [Module: outputs]        ERROR: TypeError[137]: cannot unpack non-iterable NoneType object
2023-03-17 11:17:03,040 [File: models.py:97]    [Module: models]         ERROR: ValidationError[57]: 1 validation error for Azure_Check_Output_CSV
tenant_domain
  none is not an allowed value (type=type_error.none.not_allowed)
2023-03-17 11:17:03,040 [File: outputs.py:174]  [Module: outputs]        ERROR: TypeError[137]: cannot unpack non-iterable NoneType object
2023-03-17 11:17:03,182 [File: models.py:97]    [Module: models]         ERROR: ValidationError[57]: 1 validation error for Azure_Check_Output_CSV
tenant_domain
  none is not an allowed value (type=type_error.none.not_allowed)
2023-03-17 11:17:03,182 [File: outputs.py:174]  [Module: outputs]        ERROR: TypeError[137]: cannot unpack non-iterable NoneType object
2023-03-17 11:17:03,314 [File: models.py:97]    [Module: models]         ERROR: ValidationError[57]: 1 validation error for Azure_Check_Output_CSV
tenant_domain
  none is not an allowed value (type=type_error.none.not_allowed)
2023-03-17 11:17:03,314 [File: outputs.py:174]  [Module: outputs]        ERROR: TypeError[137]: cannot unpack non-iterable NoneType object
-> Scan completed! |▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉| 20/20 [100%] in 1:36.1

Overview Results:
╭─────────────────────┬─────────────────────╮
│ 54.47% (591) Failed │ 45.53% (494) Passed │
╰─────────────────────┴─────────────────────╯

Tenant ID/s f65d02be-9231-4769-9120-8d7f799652db 17876228-80af-4ab9-b217-d81e5e453231 Scan Results (severity columns are for fails only):

Subscriptions scanned: CSAR-Prod-Ext-Subscription CSAR-Prod-Int-Subscription CHLT-Prod-Ext-Subscription CSAR-Dev-Int-Subscription CHLT-Dev-Ext-Subscription CHLT-Dev-Int-Subscription CSAR-Dev-Ext-Subscription CPRD-Prod-Int-Subscription CHLT-Prod-Int-Subscription
╭────────────┬───────────┬────────────┬────────────┬────────┬──────────┬───────╮
│ Provider   │ Service   │ Status     │   Critical │   High │   Medium │   Low │
├────────────┼───────────┼────────────┼────────────┼────────┼──────────┼───────┤
│ azure      │ iam       │ FAIL (32)  │          0 │     32 │        0 │     0 │
├────────────┼───────────┼────────────┼────────────┼────────┼──────────┼───────┤
│ azure      │ storage   │ FAIL (559) │          0 │    140 │      279 │   140 │
╰────────────┴───────────┴────────────┴────────────┴────────┴──────────┴───────╯
* You only see here those services that contains resources.

Detailed results are in:
 - CSV: D:\Cloud Tools\prowler/output/prowler-output-f65d02be-9231-4769-9120-8d7f799652db-17876228-80af-4ab9-b217-d61e5e453231-20230317111516.csv
 - JSON: D:\Cloud Tools\prowler/output/prowler-output-f65d02be-9231-4769-9120-8d7f799652db-17876228-80af-4ab9-b217-d61e5e453231-20230317111516.json
n4ch04 commented 1 year ago

@njmulsqb thanks, we are progressing, it seems that your current Defender configuration generates an error, I'am going to review it and let you know asap

n4ch04 commented 1 year ago

Hi @njmulsqb we can't reproduce the issue you face, can you join our slack to follow up on this easily? https://join.slack.com/t/prowler-workspace/shared_invite/zt-1ruw3t8xv-CAyLRESly~AOgbHd1Z2g~A

I've added a workaround to solve that error here #2129 , but we would like to know a little bit about your azure subscription configuration to avoid this kind of errors

njmulsqb commented 1 year ago

@n4ch04 Apologies for late response, I took the latest pull and ran prowler on a specific subscription. It ran successfully but it didnt run only on mentioned subscription but on whole tenant. Why is it so?

python prowler.py azure --az-cli-auth --subscription-ids 0c43b29a-25a1-41f0-b601-11d3181e8f6h
                         _
 _ __  _ __ _____      _| | ___ _ __
| '_ \| '__/ _ \ \ /\ / / |/ _ \ '__|
| |_) | | | (_) \ V  V /| |  __/ |
| .__/|_|  \___/ \_/\_/ |_|\___|_|v3.3.0
|_| the handy cloud security tool

Date: 2023-03-28 16:28:27

Executing 20 checks, please wait...

-> Scan completed! |▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉| 20/20 [100%] in 1:00.0

Overview Results:
╭─────────────────────┬─────────────────────╮
│ 54.33% (646) Failed │ 45.42% (540) Passed │
╰─────────────────────┴─────────────────────╯

Tenant ID/s f65d02be-9231-4769-9120-8d7f799652du 17876228-80af-4ab9-b217-d81e5e453237 Scan Results (severity columns are for fails only):

Subscriptions scanned: CSAR-Prod-Ext-Subscription CSAR-Prod-Int-Subscription CHLT-Prod-Ext-Subscription CSAR-Dev-Int-Subscription CHLT-Dev-Ext-Subscription CHLT-Dev-Int-Subscription CSAR-Dev-Ext-Subscription CPRD-Prod-Int-Subscription CHLT-Prod-Int-Subscription
╭────────────┬───────────┬────────────┬────────────┬────────┬──────────┬───────╮
│ Provider   │ Service   │ Status     │   Critical │   High │   Medium │   Low │
├────────────┼───────────┼────────────┼────────────┼────────┼──────────┼───────┤
│ azure      │ defender  │ FAIL (52)  │          0 │     52 │        0 │     0 │
├────────────┼───────────┼────────────┼────────────┼────────┼──────────┼───────┤
│ azure      │ iam       │ FAIL (32)  │          0 │     32 │        0 │     0 │
├────────────┼───────────┼────────────┼────────────┼────────┼──────────┼───────┤
│ azure      │ storage   │ FAIL (562) │          0 │    141 │      280 │   141 │
╰────────────┴───────────┴────────────┴────────────┴────────┴──────────┴───────╯
* You only see here those services that contains resources.
n4ch04 commented 1 year ago

Humm, interesting, it seems that the subscription input option is not being correctly propagated. Good catch, I'm going to review it and reach you when I have something.

Thanks !!

njmulsqb commented 1 year ago

The -M flag also doesnt seem to work, html file is not being generated by default, if I specify with flag it still doesnt work. 

(prowler-py3.10) PS D:\Cloud Tools\prowler> python prowler.py azure --az-cli-auth --subscription-ids 0c43b29a-25a1-41f0-b601-11d318 -M html
                         _
 _ __  _ __ _____      _| | ___ _ __
| '_ \| '__/ _ \ \ /\ / / |/ _ \ '__|
| |_) | | | (_) \ V  V /| |  __/ |
| .__/|_|  \___/ \_/\_/ |_|\___|_|v3.3.0
|_| the handy cloud security tool

Date: 2023-03-28 18:00:08

Executing 20 checks, please wait...

-> Scan completed! |▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉| 20/20 [100%] in 1:05.2

Overview Results:
╭─────────────────────┬─────────────────────╮
│ 54.33% (646) Failed │ 45.42% (540) Passed │
╰─────────────────────┴─────────────────────╯

Tenant ID/s f65d02be-9231-4769-9120-8d7f79 17876228-80af-4ab9-b217-d81e5e451 Scan Results (severity columns are for fails only):

Subscriptions scanned: CSAR-Prod-Ext-Subscription CSAR-Prod-Int-Subscription CHLT-Prod-Ext-Subscription CSAR-Dev-Int-Subscription CHLT-Dev-Ext-Subscription CHLT-Dev-Int-Subscription CSAR-Dev-Ext-Subscription CPRD-Prod-Int-Subscription CHLT-Prod-Int-Subscription
╭────────────┬───────────┬────────────┬────────────┬────────┬──────────┬───────╮
│ Provider   │ Service   │ Status     │   Critical │   High │   Medium │   Low │
├────────────┼───────────┼────────────┼────────────┼────────┼──────────┼───────┤
│ azure      │ defender  │ FAIL (52)  │          0 │     52 │        0 │     0 │
├────────────┼───────────┼────────────┼────────────┼────────┼──────────┼───────┤
│ azure      │ iam       │ FAIL (32)  │          0 │     32 │        0 │     0 │
├────────────┼───────────┼────────────┼────────────┼────────┼──────────┼───────┤
│ azure      │ storage   │ FAIL (562) │          0 │    141 │      280 │   141 │
╰────────────┴───────────┴────────────┴────────────┴────────┴──────────┴───────╯
* You only see here those services that contains resources.

Detailed results are in:
njmulsqb commented 1 year ago

I see that very few work is done with Azure on this tool, most information is of AWS Let me know if I can contribute somehow to improve Azure support (apart from fixing this issue, haha)

n4ch04 commented 1 year ago

Hi @njmulsqb as I supposed the subscriptions flag was not being propagated. It is solved here #2147 ¿Could you test it please? Yep, the html output is not available yet in Azure. Regarding adding more features and checks in Azure provider we are currently working on it, we hope we can release sooner than later more checks and integrate all the current compatible AWS features. If you want to contribute with this of course your help is welcomed !! Feel free to visit our docs https://docs.prowler.cloud/en/latest/ and join our slack https://join.slack.com/t/prowler-workspace/shared_invite/zt-1ruw3t8xv-CAyLRESly~AOgbHd1Z2g~A to ask for help or propose new ideas, also, if you want you can open a PR with new features.

njmulsqb commented 1 year ago

Yes, it is working fine now

n4ch04 commented 1 year ago

Thank you, closing this then