search

microsoft / ARI

Azure Resource Inventory - It's a Powerful tool to create EXCEL inventory from Azure Resources with low effort
MIT License
1.12k stars 397 forks source link

New version is taking too long to building report... #215

Closed fepere closed 2 weeks ago

fepere commented 2 months ago

Hello @Claudio-Merola,

The previous version takes ~28 min to cover +/-390 subscriptions, now this new version is taking forever. more than 4 hours already.

also, I had tried the Automation account and get suspended.

I had enabled the debug but now able to find any issues.

Can you please take a look on that?

Thx

Claudio-Merola commented 2 months ago

Hi @fepere

There are some new steps to collect extra data in this version. Running the script with -Debug we can see what exactly the script is doing, with that, can you tell me the following:

The script is getting "stuck" at some specific part?

In case the script is running normally, but is just taking longer, at which part is actually taking longer than used to?

Thanks

fepere commented 2 months ago

Hi @Claudio-Merola

Yes, it's stuck here:

DEBUG: 2024-09-08_23_47_23 - Running Module: 'ServiceBUS'. Lines Count: 476 DEBUG: 2024-09-09_00_44_19 - Running Module: 'LoadBalancer'. Lines Count: 7859 DEBUG: 2024-09-09_08_43_44 - Running Module: 'NATGAteway'. Lines Count: 544 Azure Resource Inventory Reporting (21073) Resources [Processing Inventory ] Building Report [80% Complete. ]

since last friday

DEBUG: 2024-09-05_21_03_29 - Debbuging Mode: On. ErrorActionPreference was set to "Continue", every error will be presented. Checking for Powershell Module Updates.. DEBUG: 2024-09-05_21_03_30 - Starting Test-ARIPS function PowerShell Desktop Identified.

Claudio-Merola commented 2 months ago

@fepere

The script took 8 hours to send the Load Balancers to the Excel file after the processing was done. The script usually take longer times to process the resources, not to send them to the Excel File...

Apparently something is wrong here.

How much memory is the Powershell process consuming?

fepere commented 2 months ago

Hi @Claudio-Merola

yes, it's taking more then 20G as you can see the screenshot

image

Claudio-Merola commented 2 months ago

@fepere

This should also be addressed in the version 3.5.1, can you try it and see how it goes?

Thanks

fepere commented 2 months ago

Hi @Claudio-Merola I updated to 3.5.1 version and still running, the ARI detect

DEBUG: 2024-09-12_17_41_02 - Debbuging Mode: On. ErrorActionPreference was set to "Continue", every error will be presented.

DEBUG: 2024-09-12_17_41_57 - Number of Subscriptions Found: 411

DEBUG: 2024-09-12_19_08_42 - Excel file: C:\AzureResourceInventory\AzureResourceInventory_Report_2024-09-12_19_08.xlsx DEBUG: 2024-09-12_19_08_42 - Starting Default Jobs.

DEBUG: 2024-09-12_19_09_01 - Starting Resource Jobs. Large Size Environment Identified. Jobs will be run in batches to avoid CPU Overload. DEBUG: 2024-09-12_19_09_02 - Starting Processing Jobs in Large Mode. DEBUG: 2024-09-12_19_09_02 - Total Resources Being Processed: 22370 DEBUG: 2024-09-12_19_09_03 - Resources Being Processed in ResourceJob_1: 5000 DEBUG: 2024-09-12_19_09_17 - Resources Being Processed in ResourceJob_2: 5000 DEBUG: 2024-09-12_19_09_32 - Resources Being Processed in ResourceJob_3: 5000 DEBUG: 2024-09-12_19_11_34 - Resources Being Processed in ResourceJob_4: 5000 DEBUG: 2024-09-12_19_13_02 - Resources Being Processed in ResourceJob_5: 1853 DEBUG: 2024-09-12_19_13_05 - Waiting Batch of Jobs to Complete. DEBUG: 2024-09-12_19_13_08 - initial Jobs Running: 5 DEBUG: 2024-09-12_19_13_36 - initial Jobs Running: 5 DEBUG: 2024-09-12_19_13_51 - initial Jobs Running: 5 DEBUG: 2024-09-12_19_14_07 - initial Jobs Running: 5 DEBUG: 2024-09-12_19_14_22 - initial Jobs Running: 5 DEBUG: 2024-09-12_19_14_38 - initial Jobs Running: 5 DEBUG: 2024-09-12_19_14_53 - initial Jobs Running: 5 DEBUG: 2024-09-12_19_15_08 - initial Jobs Running: 5 DEBUG: 2024-09-12_19_15_23 - initial Jobs Running: 5 DEBUG: 2024-09-12_19_15_38 - initial Jobs Running: 5 DEBUG: 2024-09-12_19_15_53 - initial Jobs Running: 5 DEBUG: 2024-09-12_19_16_08 - initial Jobs Running: 5 DEBUG: 2024-09-12_19_16_23 - initial Jobs Running: 5 DEBUG: 2024-09-12_19_16_38 - initial Jobs Running: 5 DEBUG: 2024-09-12_19_16_53 - initial Jobs Running: 5 DEBUG: 2024-09-12_19_18_10 - initial Jobs Running: 5 DEBUG: 2024-09-12_19_18_25 - initial Jobs Running: 5 DEBUG: 2024-09-12_19_18_40 - initial Jobs Running: 4 DEBUG: 2024-09-12_19_18_56 - initial Jobs Running: 4 DEBUG: 2024-09-12_19_20_51 - initial Jobs Running: 4 DEBUG: 2024-09-12_19_21_07 - initial Jobs Running: 4 DEBUG: 2024-09-12_19_21_23 - initial Jobs Running: 4 DEBUG: 2024-09-12_19_21_38 - initial Jobs Running: 4 DEBUG: 2024-09-12_19_21_53 - initial Jobs Running: 4 DEBUG: 2024-09-12_19_22_08 - initial Jobs Running: 4 DEBUG: 2024-09-12_19_25_57 - initial Jobs Running: 4 DEBUG: 2024-09-12_19_26_15 - initial Jobs Running: 4 DEBUG: 2024-09-12_19_26_30 - initial Jobs Running: 4 DEBUG: 2024-09-12_19_27_33 - initial Jobs Running: 4 DEBUG: 2024-09-12_19_27_49 - initial Jobs Running: 4 DEBUG: 2024-09-12_19_28_05 - initial Jobs Running: 4 DEBUG: 2024-09-12_19_28_21 - initial Jobs Running: 4 DEBUG: 2024-09-12_19_28_36 - initial Jobs Running: 4 DEBUG: 2024-09-12_19_31_25 - initial Jobs Running: 4 DEBUG: 2024-09-12_19_33_25 - initial Jobs Running: 4 DEBUG: 2024-09-12_19_33_40 - initial Jobs Running: 4 DEBUG: 2024-09-12_19_33_56 - initial Jobs Running: 4 DEBUG: 2024-09-12_19_34_12 - initial Jobs Running: 4 DEBUG: 2024-09-12_19_40_22 - initial Jobs Running: 4 DEBUG: 2024-09-12_19_40_37 - initial Jobs Running: 4 DEBUG: 2024-09-12_19_40_52 - initial Jobs Running: 4 DEBUG: 2024-09-12_19_41_07 - initial Jobs Running: 4 DEBUG: 2024-09-12_19_41_22 - initial Jobs Running: 4 DEBUG: 2024-09-12_19_41_38 - initial Jobs Running: 4 DEBUG: 2024-09-12_19_41_53 - initial Jobs Running: 4 DEBUG: 2024-09-12_19_44_46 - initial Jobs Running: 4 DEBUG: 2024-09-12_19_52_37 - initial Jobs Running: 4 DEBUG: 2024-09-12_19_52_52 - initial Jobs Running: 2 DEBUG: 2024-09-12_19_55_42 - initial Jobs Running: 2 DEBUG: 2024-09-12_19_55_57 - initial Jobs Running: 2 DEBUG: 2024-09-12_20_03_02 - initial Jobs Running: 1 DEBUG: 2024-09-12_20_03_17 - initial Jobs Running: 1 DEBUG: 2024-09-12_20_13_29 - initial Jobs Running: 1 DEBUG: 2024-09-12_20_13_44 - initial Jobs Running: 1 DEBUG: 2024-09-12_20_13_59 - initial Jobs Running: 1 DEBUG: 2024-09-12_20_23_09 - initial Jobs Running: 1 DEBUG: 2024-09-12_20_23_24 - initial Jobs Running: 1 DEBUG: 2024-09-12_20_31_45 - initial Jobs Running: 1 DEBUG: 2024-09-12_20_32_00 - initial Jobs Running: 1 DEBUG: 2024-09-12_20_32_15 - initial Jobs Running: 1 DEBUG: 2024-09-12_20_35_44 - initial Jobs Running: 1 Azure Inventory [21% Complete. ] Processing Initial Resource Jobs [0% Complete. ]

Still running, now it's 8:45PM - GMT-3

Please take a look and let me know

Thanks Fernando Pereira

Rodzmig commented 2 months ago

Hi @Claudio-Merola, in this new version, and in the previous one, when generating the Report, the process remains stopped after the LoadBalanced Module, I left the process running for more than 24 hours, and it did not pass 78%.

DEBUG: 2024-09-12_14_17_21 - Running Module: 'AppGW'. Lines Count: 2 DEBUG: 2024-09-12_14_17_29 - Running Module: 'Connections'. Lines Count: 2 DEBUG: 2024-09-12_14_17_33 - Running Module: 'LoadBalancer'. Lines Count: 178524 Azure Resource Inventory Reporting (1088) Resources [Processing In

It works well for most subscriptions. Thanks for your effort.

Claudio-Merola commented 2 months ago

@fepere It is still running in what phase? Is that after the processing jobs? How much memory is the process consuming now?

@Rodzmig Is the Excel file populated with the Load Balancer's data? 178k lines is a lot of lines for the script to process, how much memory the powershell process is consuming?

fepere commented 2 months ago

@Claudio-Merola I started again the Invoke-ARI and per de debug logs the Jobs Completed now is taking forever for Cache Files as you can see:

DEBUG: 2024-09-13_10_30_33 - initial Jobs Running: 1 DEBUG: 2024-09-13_10_31_24 - Starting Jobs Collector. DEBUG: 2024-09-13_10_31_24 - Jobs Compleated. DEBUG: 2024-09-13_10_31_24 - Creating Folder for Cache Files. DEBUG: 2024-09-13_10_31_24 - Job ResourceJob_1 Returned: 41 Resource Types. DEBUG: 2024-09-13_10_31_25 - Creating Cache File: ResourceJob_1 DEBUG: 2024-09-13_10_31_49 - Job ResourceJob_2 Returned: 38 Resource Types. DEBUG: 2024-09-13_10_31_50 - Creating Cache File: ResourceJob_2 DEBUG: 2024-09-13_10_32_13 - Job ResourceJob_3 Returned: 38 Resource Types. DEBUG: 2024-09-13_10_32_15 - Creating Cache File: ResourceJob_3 DEBUG: 2024-09-13_10_32_45 - Job ResourceJob_4 Returned: 37 Resource Types. DEBUG: 2024-09-13_10_32_45 - Creating Cache File: ResourceJob_4 DEBUG: 2024-09-13_10_33_48 - Job ResourceJob_5 Returned: 2 Resource Types. DEBUG: 2024-09-13_10_33_49 - Creating Cache File: ResourceJob_5 DEBUG: 2024-09-13_10_33_49 - Starting Resource Reporting. DEBUG: 2024-09-13_10_33_50 - Starting Reporting Phase. DEBUG: 2024-09-13_10_33_50 - Running Offline, Gathering List Of Modules. DEBUG: 2024-09-13_10_33_51 - Modules Found: 83 DEBUG: 2024-09-13_10_33_51 - Looking for Cache Files. Azure Resource Inventory Reporting (22103) Resources [Processing Inventory ] Building Report [0% Complete.

The pwsh is consuming +/- 20GB as you can see the screenshot

image

Rodzmig commented 2 months ago

@Claudio-Merola, It consumes between 3.9 and 4.3 GB of RAM, but in the end I had to kill the process, and it was not reflected in the report.

Rodzmig commented 2 months ago

@Claudio-Merola, but Count: 178524, is the number of resources?, If I check it manually, it only has 2 load balancers. I tried to run it several times, but it always stayed at 78% and didn't go beyond that point.

ARI_process

Claudio-Merola commented 2 months ago

@fepere

Can you try the version 3.5.2?

Run it with the parameter "-Heavy" please

fepere commented 2 months ago

Hi @Claudio-Merola I updated to version 3.5.2 and using the -AppId -Secret -Heavy, but I'm seeing this error:

This is the command that I'm using: Invoke-ARI -TenantID $TenantId -AppId $ApplicationId -Secret $SecurePassword -IncludeTags -SkipDiagram -Heavy -Debug

here is the error:

Extracting Subscriptions from Tenant Microsoft.Azure.Commands.Profile.Models.Core.PSAzureContext xxxxxxxx-xxxxxxx Get-AzSubscription: Cannot convert 'System.Object[]' to the type 'System.String' required by parameter 'TenantId'. Specified method is not supported. DEBUG: 2024-09-18_09_15_20 - Checking report folder: C:\AzureResourceInventory\ Starting Resource Extraction.. DEBUG: 2024-09-18_09_15_20 - Starting Extractor function DEBUG: 2024-09-18_09_15_20 - Powershell Edition: Core DEBUG: 2024-09-18_09_15_20 - Powershell Version: 7.4.5 DEBUG: 2024-09-18_09_15_20 - Tags will be included DEBUG: 2024-09-18_09_15_20 - Number of Subscriptions Found: 0

See the number of subscription is 0 but this is not true.

Can you please check if you using SPN the system.object cause the problem to get the tenantid?

Claudio-Merola commented 2 months ago

@fepere

I changed the Login behavior of the module when using SPN. Can you try the version 3.5.3 and see if this is working as before?

fepere commented 2 months ago

Hi @Claudio-Merola I tried and the SPN is not working anymore can you please advise?

fepere commented 2 months ago

Hey @Claudio-Merola A versão 3.5.2 funciona o SPN mas não o ManagementGroup. A versão 3.5.3 não está funcionando o SPN mais e com isso não consigo testar o ManagementGroup. Testei o managementgroup com uma conta e esta funcionando, mas preciso que o SPN voltei a funcionar.

Claudio-Merola commented 1 month ago

Ola @fepere

Eu alterei o parametro do SPN para receber o secret como plaintext ao invés de securestring

fepere commented 1 month ago

Olá @Claudio-Merola Tem algo estranho, pq tentei como plaintext e SecureString e não funcionam. veja o exemplo:

Invoke-ARI: Cannot process argument transformation on parameter 'Secret'. Cannot convert the value of type "System.String" to type "System.Security.SecureString".

Depois tentei fazer o AsSecureString e tbem não funciona:

PS C:\Users\azureuser\Downloads> $SecurePassword = Read-Host -Prompt 'Enter a Password' -AsSecureString Enter a Password: **** PS C:\Users\azureuser\Downloads> Invoke-ARI -TenantID $TenantId -AppId $ApplicationId -Secret $SecurePassword -IncludeTags -SkipDiagram -Heavy -Debug DEBUG: 2024-09-23_11_54_53 - Debbuging Mode: On. ErrorActionPreference was set to "Continue", every error will be presented. Checking for Powershell Module Updates.. DEBUG: 2024-09-23_11_54_54 - Starting Test-ARIPS function PowerShell Desktop Identified. AzureCloud Extracting Subscriptions from Tenant xxxxxxxxxxxxxxxxxx Get-AzSubscription: Run Connect-AzAccount to login.

Ele não conecta com a SPN.

Obrigado Fernando

fepere commented 1 month ago

Olã @Claudio-Merola Vocë conseguiu verificar o meu ultimo comentário?

Um ponto importante, estou usando a versão anterior ao do módulo e não temos esse problema de performance e o relatório é concluído para 405 subscriptions + 200k resources em +- 30 minutos. Gostaria de te pedir para avaliar essa mudança pq o impacto está sendo grande e a nova versão deixei rodando por 2 dias e não termina, consumindo toda a memoria do recurso, mudei de 16gb para 32gb e mesmo assim não funciona.

Claudio-Merola commented 1 month ago

Ola @fepere

Eu acabei de atualizar a versão do modulo para 3.5.4. O SPN deve estar funcionando agora. Passe o SPN como plaintext.

Quanto a performance, é bem estranho este comportamento, não alteramos a forma como o script processa as informações.

Executando o script com o parametro -Debug, em qual parte o script fica "parado"?

github-actions[bot] commented 3 weeks ago

This Issue is stale and may be automatically close soon.