Azure / azure-powershell

Microsoft Azure PowerShell
Other
4.21k stars 3.82k forks source link

Az.Accounts slow import over networked I/O (Azure Functions) #25596

Open andystaples opened 2 months ago

andystaples commented 2 months ago

Description

In the Azure Functions environment, we have reports from customers of the Az.Accounts module taking 5 or more seconds to import. Our current theory is that this is primarily due to the PSModulePath being stored on network from the Functions VM and slow I/O leads to longer import times for this module (and likely other large modules with many files).

Note that the Issue Script & Debug output field, as well as the other fields in this bug report, were ran locally. To repro this issue most effectively, this command should be run in the Azure Functions environment, or simulated using networked storage with limited IOPS

Issue script & Debug output

Measure-Command { Import-Module Az.Accounts }

Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 2
Milliseconds      : 362
Ticks             : 23625645
TotalDays         : 2.73444965277778E-05
TotalHours        : 0.000656267916666667
TotalMinutes      : 0.039376075
TotalSeconds      : 2.3625645
TotalMilliseconds : 2362.5645

Environment data

Name                           Value
----                           -----
PSVersion                      7.4.3
PSEdition                      Core
GitCommitId                    7.4.3
OS                             Microsoft Windows 10.0.22631
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Module versions

ModuleType Version    PreRelease Name                                ExportedCommands
---------- -------    ---------- ----                                ----------------
Script     2.13.2                Az.Accounts                         {Add-AzEnvironment, Clear-AzConfig, Clear-AzConte…

Error output

DEBUG: Got version 0 of Az
DEBUG: Got version 0 of Az.Accounts
DEBUG: 10:38:56 AM - ResolveError begin processing with ParameterSet 'AnyErrorParameterSet'.
DEBUG: 10:38:56 AM - using account id 'andystaples@microsoft.com'...
DEBUG: 10:38:56 AM - [ConfigManager] Got nothing from [DisplayBreakingChangeWarning], Module = [], Cmdlet = []. Returning default value [True].

DEBUG: 10:38:56 AM - [ConfigManager] Got nothing from [DisplayRegionIdentified], Module = [], Cmdlet = []. Returning default value [True].
DEBUG: 10:38:56 AM - [ConfigManager] Got nothing from [CheckForUpgrade], Module = [], Cmdlet = []. Returning default value [True].
WARNING: You're using Az version 11.0.0. The latest version of Az is 11.2.0. Upgrade your Az modules using the following commands:
  Update-PSResource Az -WhatIf    -- Simulate updating your Az modules.
  Update-PSResource Az            -- Update your Az modules.
DEBUG: AzureQoSEvent:  Module: Az.Accounts:2.13.2; CommandName: Resolve-AzError; PSVersion: 7.4.3; IsSuccess: True; Duration: 00:00:01.3343702
DEBUG: 10:38:57 AM - [ConfigManager] Got nothing from [EnableDataCollection], Module = [], Cmdlet = []. Returning default value [True].
DEBUG: 10:38:57 AM - ResolveError end processing.
dolauli commented 1 month ago

Post more details

Az.Accounts is installed in the network storage shared through SMB. And the client is a Windows machine. So scenario is actually to import Az.Accounts from a SMB share in Windows.

After a local discussion, here are the main points we gathered:

Following is some further investigation followed by above investigation shared by me a few days ago.

In my Windows machine, I set up a Linux sub system Ubuntu 22.04. (So you can assume Linux and Windows run on the same hardware.)

In my NAS, I create a share with both NFS and SMB enabled.

Following are the test results.

image

Conclusion: The Windows SMB client implementation seems to be bottle neck here.

Suggestions: Try to tune Windows SMB client to improve the I/O performance for small files if possible Considering switch to Linux