Azure / azure-powershell

Microsoft Azure PowerShell
Other
4.21k stars 3.82k forks source link

Not able to connect to Azure with Az.Accounts version 2.12.3 (get_SerializationSettings) from within Visual studio code / Azure Automation runbook on Hybrid Worker #21960

Open mrvdsah opened 1 year ago

mrvdsah commented 1 year ago

Description

With the latest version of Az.Accounts version 2.12.3 we face an issue when running the Connect-AzAccount command in PowerShell 5.1.3 terminal in Visual Studio code. The previous version(s) 2.12.2 did not throw an error.

The error occurs with fully installed AZ module latest version 10.0.0.0 or only having the Az.Accounts module installed.

Issue script & Debug output

Connect-AzAccount : Method 'get_SerializationSettings' in type 'Microsoft.Azure.Management.Internal.Resources.ResourceManagementClient' from assembly 'Microsoft.Azure.PowerShell.Clients.Reso
urceManager, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' does not have an implementation.
At line:1 char:1
+ Connect-AzAccount
+ ~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Connect-AzAccount], TypeLoadException
    + FullyQualifiedErrorId : System.TypeLoadException,Microsoft.Azure.Commands.Profile.ConnectAzureRmAccountCommand

Environment data

Name                           Value
----                           -----
PSVersion                      5.1.17763.3770
PSEdition                      Desktop
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
BuildVersion                   10.0.17763.3770
CLRVersion                     4.0.30319.42000
WSManStackVersion              3.0
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1

Module versions

ModuleType Version    Name                                ExportedCommands
---------- -------    ----                                ----------------
Script     2.12.3     Az.Accounts                         {Add-AzEnvironment, Clear-AzConfig, Clear-AzContext, Clear-AzDefault...}

Error output

Resolve-AzError : Method 'get_SerializationSettings' in type 'Microsoft.Azure.Management.Internal.Resources.ResourceManagementClient' from assembly 'Microsoft.Azure.PowerShell.Clients.ResourceManager, Version=1.0.0.0, 
Culture=neutral, PublicKeyToken=31bf3856ad364e35' does not have an implementation.
At line:1 char:1
+ Resolve-AzError
+ ~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Resolve-AzError], TypeLoadException
    + FullyQualifiedErrorId : System.TypeLoadException,Microsoft.Azure.Commands.Profile.Errors.ResolveError
cpiment commented 9 months ago

Hi @darrens280

Still an issue today via Azure Automation runbook using Hybrid Worker VM (Runtime version 5.1) with latest PowerShell Az module (installed fresh using PSGallery method as per MS documentation):

Connect-AzAccount cmdlet breaks with: _Method 'getSerializationSettings' in type 'Microsoft.Azure.Management.Internal.Resources.ResourceManagementClient' from assembly 'Microsoft.Azure.PowerShell.Clients.ResourceManager, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31bxxxxxxx35' does not have an implementation.

I have the same issue and have solved it uninstalling Azure Powershell and removing all "Az" directories from C:\Program Files\WindowsPowerShell\Modules and re-installing them using the MSI from this release (9.6.0). In my case I do not use "Az.Storage" commands, but I use other modules and although it throws some deprecation warnings, it works while we get an official solution in this bug.

MBerdasco commented 8 months ago

I have the same issue and have solved it uninstalling Azure Powershell and removing all "Az" directories from C:\Program Files\WindowsPowerShell\Modules and re-installing them using the MSI from this release (9.6.0). In my case I do not use "Az.Storage" commands, but I use other modules and although it throws some deprecation warnings, it works while we get an official solution in this bug.

Worked for me (even without removing the Az directories)

cpiment commented 7 months ago

Hello @isra-fel, this bug is open since 2023/05/31 and we do not have a official solution. In our organization we have been able to run some RunBooks using Hybrid Workers and an outdated version of the Azure Powershell modules, but others don't work.

We need to know when this issue will be addressed and solved so we have a working version of the Azure Powershell modules that can work using Hybrid Workers of Azure Automation.

Thanks in advance for your help

o-l-a-v commented 7 months ago

@cpiment

Doesn't it work with PowerShell 7.2 which is GA for Hybrid Workers as well?

cpiment commented 7 months ago

Hi @o-l-a-v

I have tested it and I can run Powershell 7.2 runbooks with the latest version of Azure Powershell.

Thanks for the suggestion.

ITJoeSchmo commented 7 months ago

Hey everyone, as this has been a whack-a-mole issue that seems to be "resolved" and re-appear in the updates of other sub-modules, I set out to find a stable workaround to this issue for Non-Core versions of PowerShell (aka 6.0 and below). We would love to just jump over to PWSH, but we use a lot of dot-sourcing which isn't supported in AzureAutomation on 7.1/7.2/etc per MSFT documentation.

I learned about implementing Binding Redirects in app.configs for .NET applications -- this has proven to be a viable solution allowing us to use the latest versions of sub-modules without problem while we work towards transitioning to PowerShell Core on AzureAutomation as well as another 3rd party automation platform.

<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:dt="uuid:C2F41010-65B3-11d1-A29F-00AA00C14882" xmlns="http://www.w3.org/TR/REC-html40">

 

Implemented Solution: Binding Redirects

To circumvent the module versioning conflicts, we've successfully implemented Binding Redirects in the application configuration files. This approach forces the system to reference the newer module versions.
After I implemented this solution, I actually found it discussed in Microsoft documentation here also: Troubleshoot agent-based Hybrid Runbook Worker issues in Azure Automation | Microsoft Learn

 

Here's how to apply it:

Disclaimer: This workaround is effective for applications built on .NET

WE HAVE FOUND THAT WITH THIS WORK AROUND, SOMETIMES EXECUTING CONNECT-MGGRAPH BEFORE CONNECT-AZACCOUNT STILL RESULTS IN AN ERROR, SO BE SURE TO EXECUTE CONNECT-AZACCOUNT FIRST IF YOU LEVERAGE BOTH IN THE SAME SESSION

 

If the configuration file doesn't exist, create one with the following content:

```                                       ```

If the configuration file exists, insert the assemblyBinding section within the <runtime> tags.

> <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1"> > <dependentAssembly> > <assemblyIdentity name="Newtonsoft.Json" publicKeyToken="30ad4fe6b2a6aeed" culture="neutral" /> > <bindingRedirect oldVersion="0.0.0.0-13.0.0.0" newVersion="13.0.0.0" /> > </dependentAssembly> > </assemblyBinding> To identify dependency conflicts: 1. Re-produce the error in a PowerShell session 2. Execute the below snippet to output all assemblies loaded into the session. You will expect to see 2 assemblies with different versions to confirm it is an assembly conflict: `[System.AppDomain]::CurrentDomain.GetAssemblies() | Where−Object Location | Sort−Object −Property FullName | Select−Object −Property FullName, Location`
e-jr commented 7 months ago

Solution: Enable-AzureRmAlias -Scope LocalMachine

The error was gone for me when I enabled AzureRmAlias which is officially the compatibility mode for migrating from AzureRM to AzurePowershell https://learn.microsoft.com/en-us/powershell/azure/migrate-from-azurerm-to-az?view=azps-11.3.0#option-2-use-compatibility-mode-with-enable-azurermalias. I restarted my application (in my case a windows service) and it worked.

Not sure if if this means that the newest AzurePowershell version has a dependency to AzureRM and needs this alias or why I need this enabled.

jpawlowski commented 6 months ago

I can also confirm this issue is hindering the change from Az 8.0.0 to Az 11.2.0 in Azure Automation accounts.

Due to the introduction of runtime environments and the new UI, reverting to Az 8.0.0 is cumbersome and even more complicated when deploying automation accounts using the Azure REST API.

It even gets worse somehow because even though we are using pure new Az commands, there is now this warning where nobody knows what to do:

Both Az and AzureRM modules were detected on this machine. Az and AzureRM modules cannot be imported in the same session or used in the same script or runbook. If you are running PowerShell in an environment you control you can use the 'Uninstall-AzureRm' cmdlet to remove all AzureRm modules from your machine. If you are running in Azure Automation, take care that none of your runbooks import both Az and AzureRM modules. More information can be found here: https://aka.ms/azps-migration-guide

All we do is an explicit Import-Module Az.Accounts -MinimumVersion 2.8 -MaximumVersion 2.9999.

This somehow all seems related and worries me a lot everything would stop working tomorrow.

bandlor commented 3 months ago

I'm sad to say that I'm having this issue right now on Azure VM with:

PSVersion 5.1.19041.4412 PSEdition Desktop PSCompatibleVersions {1.0, 2.0, 3.0, 4.0...} BuildVersion 10.0.19041.4412 CLRVersion 4.0.30319.42000 WSManStackVersion 3.0 PSRemotingProtocolVersion 2.3 SerializationVersion 1.1.0.1`

Exact same error (as posted originally) coming up in VS Code, a different error appearing in PowerShell directly:

Get-AzStorageAccount : Your Azure credentials have not been set up or have expired, please run Connect-AzAccount to set up your Azure credentials.

I called Connect-AzAccount just before this, but it can only work with the -DeviceCode, and not through the browser.

My module versions are:

ModuleType Version Name Script 3.0.0 Az.Accounts Script 7.0.0 Az.Storage

I am testing it locally on Hybrid-worker VM, but it must eventually work from within a hybrid Runbook.

Any advise is welcome.

jpawlowski commented 2 months ago

The issue you're encountering appears to stem from conflicts between the Microsoft.Graph and Az modules when they're jointly utilized in a Windows PowerShell 5.1 setting.

Interestingly, the Graph PowerShell SDK team has devised a temporary solution within their modules, although the root of the problem is expected to be addressed in the Az module.

This provisional solution will only function under certain conditions. Specifically, before you engage any Microsoft.Graph module or execute any Microsoft Graph PowerShell command, you must initially load the Az.Accounts module and establish a connection to Azure using the Connect-AzAccount command. Once these steps are completed, you can proceed to use the Graph modules as required. The Graph modules are designed to recognize the concurrent use of the Az module, and their built-in solution will aid in circumventing the issue at hand.

bandlor commented 2 months ago

Thank you for your reply, @jpawlowski. There are no Microsoft.Graph modules installed on this machine. In fact, I found two machines very close in setup, where the problem was manifesting on just one of them. Eventually, I found out that problem is cause by PowerShellEditorServices module installed by VS Code PowerShell extension under C:\Users\[user]\.vscode\extensions\ms-vscode.powershell-2024.2.2\modules Removing this module will no longer prevent Az.Accounts from connecting with Connect-AzAccount -Identity Only if the module is loaded, the command throws this error:

Connect-AzAccount : Method 'get_SerializationSettings' in type 'Microsoft.Azure.Management.Internal.Resources.ResourceManagementClient' from assembly 'Microsoft.Azure.PowerShell.Clients.ResourceManager, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' does not have an implementation. At line:1 char:1 Connect-AzAccount -Identity CategoryInfo : NotSpecified: (:) [Connect-AzAccount], TypeLoadException FullyQualifiedErrorId : System.TypeLoadException,Microsoft.Azure.Commands.Profile.ConnectAzureRmAccountCommand

This appears to be a problem with VS Code/PowerShell extension, and I will try to report it there.

nilang-shah commented 2 months ago

The issue you're encountering appears to stem from conflicts between the Microsoft.Graph and Az modules when they're jointly utilized in a Windows PowerShell 5.1 setting.

Interestingly, the Graph PowerShell SDK team has devised a temporary solution within their modules, although the root of the problem is expected to be addressed in the Az module.

This provisional solution will only function under certain conditions. Specifically, before you engage any Microsoft.Graph module or execute any Microsoft Graph PowerShell command, you must initially load the Az.Accounts module and establish a connection to Azure using the Connect-AzAccount command. Once these steps are completed, you can proceed to use the Graph modules as required. The Graph modules are designed to recognize the concurrent use of the Az module, and their built-in solution will aid in circumventing the issue at hand.

Thank you for your comment! moving the Connect-AzAccount command above other 'Microsoft.Graph' module connection helped me solve the issue.

michaelmsonne commented 2 months ago

Have still this issue, also when trying to connect to Microsoft Graph in an Automation Account in Azure... :(

So far no fix - only a "workaround" to do this before calling "Connect-MgGraph -Identity -NoWelcome":

$AzureContext = (Connect-AzAccount -Identity).context $AzureContext = Set-AzContext -SubscriptionName $AzureContext.Subscription -DefaultProfile $AzureContext

jpawlowski commented 2 months ago

I have now started to migrate projects to use PowerShell modules instead of child runbooks, which was the only reason left to stay with PowerShell 5.1 runbooks.

This allows us to switch to PowerShell 7.2 runbook types, which will hopefully be a much more stable solution in the long term. I don't think that child runbooks will ever be supported in PowerShell 7.2 runbooks, so it is not worth waiting for these to suddenly appear.

Looking at the introduction of runtime environments in Azure Automation a few months ago, I am fairly sure that the manual module upload is the replacement function for child runbooks. Indeed, this will prevent some hacks that were necessary before, e.g. to avoid that child runbooks can only be run from another runbook, but not by the runbook operators.

I know it sounds very painful to migrate (and it is!). But I think this is how it is going to be in the long term.

o-l-a-v commented 2 months ago

@jpawlowski Some off topic advice / experience sharing from using Automation Account with PowerShell 7.2 for some time:

I'm considering checking out Function Apps as a replacement. I've been a long time Automation Account user, but these reliability problems and my poor Azure support experience kind of forces me to look for something else. At least for workloads were realiability is a concern, like with log ingestion.

jpawlowski commented 2 months ago

@o-l-a-v, thank you for sharing these insides. Sounds like I should reevaluate to keep the PS 5.1 workarounds and compatibility in the code. I was hoping to make the code easier by removing 5.1 backwards compatibility, especially for performance and memory optimization that's really hard to achieve sometimes in PS 5.1. I still want to try it but might now be prepared to go back to PS 5.1, but at least be prepared to switch to PS 7.2 by cleaning up the child runbook dependency.

Azure Functions are not really a replacement for Azure Automation, it really depends on the specific use case. Sometimes you might even need both in combination. Azure Functions are meant to perform quick actions like within a few seconds and can give direct output back to the caller for that reason. However, it doesn't work for long-running tasks, like for user provisioning or something that requires further processing. Azure Automation has its domain and issues shall hopefully be fixed. The support experience you describe may be different when you have access to primer support, or a direct contact to the Microsoft support organization and product groups (my experience).

ITJoeSchmo commented 2 months ago

I provided a work-around for these persistent NewtonSoft.Json conflicts we have seen rear their head time and time again between the HybridWorker modules, Az. Modules, Graph. Modules, and now VSCode in my post above back in February. It became a big hassle to try to keep specific versions with missing functionalities across the board.

My team has been leveraging this workaround I came up with in a production environment for 2 different automation platforms (AzureAutomation HybridWorkers for 1, a 3rd Party Platform for the other) which both leverage PowerShell (under different executables) to work-around the NewtonSoft.Json assembly conflict.

This assembly redirection is even mentioned by Microsoft documentation here as a potential solution which I found after I had implemented it:

  1. https://learn.microsoft.com/en-us/azure/automation/troubleshoot/hybrid-runbook-worker#unable-to-update-az-modules-while-using-the-hybrid-worker
  2. https://learn.microsoft.com/en-us/powershell/scripting/dev-cross-plat/resolving-dependency-conflicts?view=powershell-7.4#static-binding-redirect-with-appconfig-to-force-using-the-same-dependency-version
    • The only "concern" in the 2nd article for this type of solution is ".NET Core doesn't support app.config, so this solution only applies to powershell.exe (5.1)" but PowerShell Core shouldn't have this issue because it can handle assembly conflicts itself, so I don't really think it is relevant to mention?

The only quirk is some modules have to be imported in a specific order to be happy if more than 1 is used in your script:

  1. Az.*
  2. Graph.*
  3. ExchangeOnline

I have only had to re-apply the bindingRedirect once when the app was updated and it was overwritten. I am surprised the work-around has not gotten much traction since I have shared...

Today I tested this with the VSCode conflict by adding the redirect to the powershell.exe.config file located @ "$PSHOME\powershell.exe.config" and it does also resolve this issue. I decided to write up a script to allow others to easily patch their own app.config file and insert the same redirection without having to do so manually. Find that script here

The script will back up the current configuration file to $ENV:temp, take ownership and grant your user FullControl, then insert the XML into the config, then rollback the ownership/permissions. After that you should be NewtonSoft.Json-assembly-conflict-free in new PowerShell sessions.

Here is what to expect to see when running the PS1: image

Afterwards I started a new PowerShell Extension session in VSCode, Ran Import-Module Az.Accounts, and connected without issue as you can see here: image

DISCLAIMER: This script is provided "as is" without warranty of any kind, express or implied. Use this script at your own risk. While this solution has been leveraged in a PRODUCTION capacity by my team for >5 months without issue, the author and contributors are not responsible for any damage or issues potentially caused by the use of this script. Always test scripts in a non-production environment before deploying them into a production setting.

jpawlowski commented 2 months ago

I provided a work-around for these persistent NewtonSoft.Json conflicts we have seen rear their head time and time again between the HybridWorker modules, Az. Modules, Graph. Modules, and now VSCode in my post above back in February.

This workaround only helps if you don't want to do fancy serverless things, which I think is what most people want to do nowadays (if there is no on-prem dependency).

I've not been using hybrid workers intensively but wonder why people would still stick with Windows PowerShell on it. I'd be curious to learn more should you be willing to share some aspects.

For local issues on the workstation, it is just becoming too cumbersome and complicated to fix good old PowerShell 5.1. Microsoft has clearly moved on and doesn't care about PS 5.1 anymore; it is just that they can't remove it as a default on Windows (yet). Sadly, most people are not aware of the complex situation (I've been explaining this to customers quite often within the last few months).

VSCode is designed to be used with PowerShell Core. I personally would recommend everybody to leave Windows PowerShell Desktop version behind. I know that for development of code that should still work on PowerShell 5.1, this comes with some challenges to handle the backwards compatibility. I've been able to handle this okay to be honest, and PSScriptAnalyzer can be taught to help with some warnings for backwards compatibility.

In Microsoft Land (maybe not limited to that), I do think they embraced developing inside a (Linux) devcontainer as a standard which means that you will need to use PowerShell Core. I personally also wouldn't want to go back to only local development either (or switch to a Windows PC, since I'm using a Mac for decades already).

Taking all of this into consideration, you may not expect much from anyone in Microsoft to fix stuff in any area that is still using PowerShell 5.1. Any investment is in PowerShell Core and even though nobody tells you that officially, you may assume that they often simply would like to send a reply like "move to the newest version". They will not do this as an official statement anytime soon for many reasons.

That being said, I have decided to stop making own investments (timewise and moneywise) into PowerShell 5.1, too. I know this sounds like an unpopular decision to make, but you must make that sooner or later.

... and all Microsoft SaaS related modules have now been updated in a way so they can be used in PowerShell Core and are multi-platform compatible. This was different like 2 years ago. But the situation has changed for the better in that regard and the journey of PowerShell 5.1 is about to come to a dead end.

ITJoeSchmo commented 2 months ago

I provided a work-around for these persistent NewtonSoft.Json conflicts we have seen rear their head time and time again between the HybridWorker modules, Az. Modules, Graph. Modules, and now VSCode in my post above back in February.

This workaround only helps if you don't want to do fancy serverless things, which I think is what most people want to do nowadays (if there is no on-prem dependency).

I've not been using hybrid workers intensively but wonder why people would still stick with Windows PowerShell on it. I'd be curious to learn more should you be willing to share some aspects.

For local issues on the workstation, it is just becoming too cumbersome and complicated to fix good old PowerShell 5.1. Microsoft has clearly moved on and doesn't care about PS 5.1 anymore; it is just that they can't remove it as a default on Windows (yet). Sadly, most people are not aware of the complex situation (I've been explaining this to customers quite often within the last few months).

VSCode is designed to be used with PowerShell Core. I personally would recommend everybody to leave Windows PowerShell Desktop version behind. I know that for development of code that should still work on PowerShell 5.1, this comes with some challenges to handle the backwards compatibility. I've been able to handle this okay to be honest, and PSScriptAnalyzer can be taught to help with some warnings for backwards compatibility.

In Microsoft Land (maybe not limited to that), I do think they embraced developing inside a (Linux) devcontainer as a standard which means that you will need to use PowerShell Core. I personally also wouldn't want to go back to only local development either (or switch to a Windows PC, since I'm using a Mac for decades already).

Taking all of this into consideration, you may not expect much from anyone in Microsoft to fix stuff in any area that is still using PowerShell 5.1. Any investment is in PowerShell Core and even though nobody tells you that officially, you may assume that they often simply would like to send a reply like "move to the newest version". They will not do this as an official statement anytime soon for many reasons.

That being said, I have decided to stop making own investments (timewise and moneywise) into PowerShell 5.1, too. I know this sounds like an unpopular decision to make, but you must make that sooner or later.

... and all Microsoft SaaS related modules have now been updated in a way so they can be used in PowerShell Core and are multi-platform compatible. This was different like 2 years ago. But the situation has changed for the better in that regard and the journey of PowerShell 5.1 is about to come to a dead end.

I totally agree with everything you've said here. The serverless aspect makes sense as this wouldn't be applicable.

I've been slowly adjusting all our automation to be compatible with PowerShell Core. Right now a lot of our runbooks dot-source other run books. I'm working on turning some of those children runbooks into functions in a module, and making other children runbooks use basic variables over complex ones so we can leverage start-azautomationrunbook. Anything new I've written I make sure is written in a way that can be used with PowerShell Core.

One problem is I don't think our 3rd party automation platform has the option of using Core, but with 5.1 EOS coming up in a few years I'm sure that will change as well.

Our goal is to use Core for sure. Unfortunately we don't have the resources to make the migration our #1 priority but it is something we are iterating towards.

MrFly72 commented 2 months ago

So there is the upcomming breaking change:

2024-07-17T05:19:22.978 [Warning] WARNING: Upcoming breaking changes in the cmdlet 'Get-AzAccessToken' :The Token property of the output type will be changed from String to SecureString. Add the [-AsSecureString] switch to avoid the impact of this upcoming breaking change.- The change is expected to take effect in Az version : '13.0.0'- The change is expected to take effect in Az.Accounts version : '4.0.0'Note : Go to https://aka.ms/azps-changewarnings for steps to suppress this breaking change warning, and other information on breaking changes in Azure PowerShell.

And this case is still valid for me in VSCode, which still trows the error on Connect-AZAccount when using 3.0.2 of AZ Accounts.

Clawz2k commented 3 weeks ago

Was getting the same error for couple of months and finally got time to have a closer look. Downgrade the packages to the below list and haven't had an failure for the past 3 days. Usually I get at least 1 failed per day. Az.Accounts 2.12.1 Az.Resources 6.5.3 Az.Storage 5.4.1 AzTable 2.1.0

MrFly72 commented 3 weeks ago

This is not a good solution anymore, as AZ.Accounts will have a breaking change soon. Its really a pain that nobody takes care of this for such a long time ....

michaelmsonne commented 3 weeks ago

Can't agree more - but I'm trying to push here with my contacts at Microsoft and the team behind - stay tuned!

MrFly72 commented 2 weeks ago

Any news on this? Somebody again upgraded the AZ.Accounts, as he needed a new module on a server and now again VSCode is broken in PS5.1 (not in 7.x). But i need to test backward compatibility as people in my teams usually dont use PS7. This is a nightmare Microsoft!!!