dingmeng-xue
commented
2 years ago Hi Madbrenner, could you share more error message from other modules? such as Accounts, Keyvault or Monitor. Could you try to collect more debug trace?
Open madbrenner opened 2 years ago
dingmeng-xue
commented
2 years ago Hi Madbrenner, could you share more error message from other modules? such as Accounts, Keyvault or Monitor. Could you try to collect more debug trace?
madbrenner
commented
2 years ago Hi Madbrenner, could you share more error message from other modules? such as Accounts, Keyvault or Monitor. Could you try to collect more debug trace?
Hello @dingmeng-xue I can run this more and provide some samples of the output for the other modules, but they all follow the same pattern as the message above, so I am not sure how much that would really help.
I have used $DebugPrefence = "Continue" to generate the debug output above, is there another method I can use to generate more debug output? I have tried using the -Debug switch on the commands, but then they fail because they are not running interactively.
dingmeng-xue
commented
2 years ago @madbrenner , I understood your question. Current message is a little general and we cannot guess the root cause of it. We will also check our code. For the time being, we only can conjecture issue according to static code analysis. Your further information will be helpful.
Please also set environment variable Azure_PS_Intercept_Survey as false. I just suspect the issue is related to that feature.
madbrenner
commented
2 years ago @dingmeng-xue that seems to have resolved this issue for us, thank you for the quick follow up on this! Is there any chance you could share more information about what this survey is, when it was introduced, and/or why it causes this odd error?
Thank you!
dingmeng-xue
commented
2 years ago We maintain a dictionary to record when user use the module such as Az.Storage, Az.Keyvault and so on. If user has used module couple times in one month, Azure PowerShell cmdlet will show up one message including the link survey and look forward to feedback.
We don't know the root cause yet but your feedback gives us direction. We will check the code.
madbrenner
commented
2 years ago Thank you for the information @dingmeng-xue this resolved the issue for us, so you can feel free to close out this issue, unless you are still using it to track any work around this, thank you!
And my suggestion would be to add an extra guard around this to ensure the PowerShell session is interactive before attempting to display any survey.
dingmeng-xue
commented
2 years ago Hi @madbrenner , I will keep this issue open for a while because we want to check the root cause and whether other user reports the similar problem.
For survey prompt, we just show message on console and won't break any user interactive. We just worry about that user easily ignore survey message. Although we hope to hear user's feedback, usability is always the priority.
madbrenner
commented
2 years ago Hello @dingmeng-xue that makes sense - my recommendation would be to only prompt for the survey if the PowerShell session is interactive. In our case our scripts are running in a non-interactive session, which I suspect could be the reason it was breaking things for us. I am not certain of the best way to check for that, but if you update the logic guard from:
if ($env:Azure_PS_Intercept_Survey -eq "false") {
return
}to something like:
$sessionIsNonInteractive = <however you determine if a session is non-interactive>
if ($env:Azure_PS_Intercept_Survey -eq "false" -or $sessionIsNonInteractive) {
return
}we would never have seen this problem crop up.
madbrenner
commented
2 years ago FYI I also see a minor error in the 0.4.0 version of the survey script (viewed from the PowerShell Gallery at: https://www.powershellgallery.com/packages/Az.Tools.Predictor/0.4.0/Content/InterceptSurvey.ps1)
$mutexTiimeout = 1000 <---
...
$mutexTimeout = 500
VeryEarly
commented
2 years ago one possible cause could be that I used concurrentdictionary.Item[] to access data which might not be thread-safe. I'll try switch to thread-safe methods instead.
jberezanski-mdg
commented
2 years ago We have recently started seeing a similar issue on build servers in my company. The KeyNotFoundException gets thrown from a call to Disconnect-AzAccount (which is called automatically by the AzurePowerShell pipeline task).
After instrumenting the pipeline task with some logging, I got these error details, in particular the stack trace:
Disconnect-AzAccount -Scope Process -ErrorAction Stop
##[warning]The given key was not present in the dictionary.
error record:
PSMessageDetails :
Exception : System.Collections.Generic.KeyNotFoundException: The given key was not present in the dictionary.
at System.Collections.Concurrent.ConcurrentDictionary`2.get_Item(TKey key)
at Microsoft.Azure.PowerShell.Common.Share.Survey.SurveyHelper.ShouldModulePrompt(SurveyHelper helper, String moduleName, Int32 majorVersion)
at Microsoft.Azure.PowerShell.Common.Share.Survey.SurveyHelper.ShouldFlush(String moduleName, Int32 majorVersion, Func`4 condition, Action`3 updateModule)
at Microsoft.Azure.PowerShell.Common.Share.Survey.SurveyHelper.ShouldPropmtSurvey(String moduleName, Version moduleVersion)
at Microsoft.WindowsAzure.Commands.Utilities.Common.AzurePSCmdlet.EndProcessing()
at System.Management.Automation.CommandProcessorBase.Complete()
TargetObject :
CategoryInfo : NotSpecified: (:) [Disconnect-AzAccount], KeyNotFoundException
FullyQualifiedErrorId : System.Collections.Generic.KeyNotFoundException,Microsoft.Azure.Commands.Profile.DisconnectAzureRmAccountCommand
ErrorDetails :
InvocationInfo : System.Management.Automation.InvocationInfo
ScriptStackTrace : at Disconnect-UsingAzModule, D:\B\1\_tasks\AzurePowerShell_72a1931b-effb-4d2e-8fd8-f8472a07cb62\5.202.0\ps_modules\VstsAzureHelpers_\Utility.ps1: line 568
at Disconnect-AzureAndClearContext, D:\B\1\_tasks\AzurePowerShell_72a1931b-effb-4d2e-8fd8-f8472a07cb62\5.202.0\ps_modules\VstsAzureHelpers_\Utility.ps1: line 530
at <ScriptBlock>, D:\B\1\_tasks\AzurePowerShell_72a1931b-effb-4d2e-8fd8-f8472a07cb62\5.202.0\azurepowershell.ps1: line 173
at <ScriptBlock>, <No file>: line 1
at <ScriptBlock>, <No file>: line 22
at <ScriptBlock>, <No file>: line 18
at <ScriptBlock>, <No file>: line 1It leads to this code in SurveyHelper. Upon first glance, I'm not really sure why this exception happens - the code checks first if the module name is present in the dictionary and adds it if neded. But obviously it does happen somehow.
Our servers have 4 Azure Pipelines agents installed on each machine and usually run at least two pipelines concurrently; many of the pipelines use the AzurePowerShell task. The problem does not appear often - I'd say once in 3-4 days. (But when it does appear, it breaks our build, because it causes a warning and we have a "no build warnings" policy.)
For the record, our environment is currently using:
I've disabled the survey using the environment variable as a workaround and will observe what happens over the following weeks.
Description
We use PowerShell and the Az module to deploy our services to Azure, running off a docker image via Azure Container Instances, and since updating our image around July 14th we have been seeing this error mentioned in the title intermittently during our deployments. We have been able use an older image which works fine that was build on June 7th, but any newer images we try to build have this problem, with the latest versions of PowerShell and Az. Our deployments will be proceeding fine, then a command (such as Set-AzWebapp) will fail with the message regarding the module the command is in, in this case:
The given key 'Az.Websites' was not present in the dictionary.We have seen this error mention multiple modules:Error messages appear as below:
I have verified this does not seem to be an issue with ACI specifically, as I have also built and run the image locally and see the same error come up.
Steps to reproduce
Build a docker image based on
jenkins/inbound-agent:4.9-1, and install PowerShell 7.1.3, and Az 6.3.0. Use the docker image to run a long, automated deployment with multiple threads and eventually the error message above will come up.Environment data
Module versions
Debug output
Error output