The Get-AzSynapseSparkConfiguration command has stopped working.

[Karni-G]

Description

It has stopped working since yesterday.

Issue script & Debug output

unable to paste anything

Environment data

Name                           Value                                                                                                                   
----                           -----                                                                                                                   
PSVersion                      5.1.22621.2506                                                                                                          
PSEdition                      Desktop                                                                                                                 
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}                                                                                                 
BuildVersion                   10.0.22621.2506                                                                                                         
CLRVersion                     4.0.30319.42000                                                                                                         
WSManStackVersion              3.0                                                                                                                     
PSRemotingProtocolVersion      2.3                                                                                                                     
SerializationVersion           1.1.0.1

Module versions

ModuleType Version    Name                                ExportedCommands                                                                             
---------- -------    ----                                ----------------                                                                             
Script     2.15.0     Az.Accounts                         {Add-AzEnvironment, Clear-AzConfig, Clear-AzContext, Clear-AzDefault...}                     
Script     3.2.0      Az.OperationalInsights              {Disable-AzOperationalInsightsIISLogCollection, Disable-AzOperationalInsightsLinuxCustomLo...
Script     6.14.0     Az.Resources                        {Export-AzResourceGroup, Export-AzTemplateSpec, Get-AzDenyAssignment, Get-AzDeployment...}   
Script     3.0.5      Az.Synapse                          {Add-AzSynapseDataFlowDebugSessionPackage, Add-AzSynapseTriggerSubscription, Clear-AzSynap...

Error output

DEBUG: 6:59:44 PM - ResolveError begin processing with ParameterSet 'AnyErrorParameterSet'.
DEBUG: 6:59:44 PM - using account id 'Id'...
DEBUG: 6:59:44 PM - [ConfigManager] Got nothing from [DisplayBreakingChangeWarning], Module = [], Cmdlet = []. Returning default value [True].

   HistoryId: 47

Message        : The requested operation requires an element of type 'String', but the target element has type 'Number'.
StackTrace     :    at System.Text.Json.JsonDocument.GetString(Int32 index, JsonTokenType expectedType)
                    at Azure.Analytics.Synapse.Artifacts.Models.SparkConfiguration.DeserializeSparkConfiguration(JsonElement element)
                    at Azure.Analytics.Synapse.Artifacts.Models.SparkConfigurationResource.DeserializeSparkConfigurationResource(JsonElement element)
                    at Azure.Core.PageableHelpers.PageableImplementation`1.<GetEnumerator>d__17.MoveNext()
                    at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext()
                    at System.Management.Automation.MshCommandRuntime._WriteObjectsSkipAllowCheck(Object sendToPipeline)
                    at System.Security.SecurityContext.Run(SecurityContext securityContext, ContextCallback callback, Object state)
                    at System.Management.Automation.MshCommandRuntime.WriteObject(Object sendToPipeline, Boolean enumerateCollection)
                    at System.Management.Automation.Cmdlet.WriteObject(Object sendToPipeline, Boolean enumerateCollection)
                    at Microsoft.Azure.Commands.Synapse.GetAzureSynapseSparkConfiguration.ExecuteCmdlet()
                    at Microsoft.WindowsAzure.Commands.Utilities.Common.AzurePSCmdlet.ProcessRecord()
Exception      : System.InvalidOperationException
InvocationInfo : {Get-AzSynapseSparkConfiguration}
Line           : $config = Get-AzSynapseSparkConfiguration -WorkspaceName $SynapseWorkspaces
Position       : At line:3 char:11
                 + $config = Get-AzSynapseSparkConfiguration -WorkspaceName $SynapseWork ...
                 +           

DEBUG: 6:59:44 PM - [ConfigManager] Got nothing from [DisplayRegionIdentified], Module = [], Cmdlet = []. Returning default value [True].
DEBUG: 6:59:44 PM - [ConfigManager] Got nothing from [CheckForUpgrade], Module = [], Cmdlet = []. Returning default value [True].
DEBUG: AzureQoSEvent:  Module: Az.Accounts:2.15.0; CommandName: Resolve-AzError; PSVersion: 5.1.22621.2506; IsSuccess: True; Duration: 00:00:00.2229976
DEBUG: 6:59:44 PM - [ConfigManager] Got nothing from [EnableDataCollection], Module = [], Cmdlet = []. Returning default value [True].

[Karni-G]

I was unable to create the issue with debug report attached. Please ping me directly to provide more details. Thanks!

[AleksanderOlbert]

Can somebody pick this up? we see a similar error with the same command. The only difference is that we see:

Get-AzSynapseSparkConfiguration : The requested operation requires an element of type 'String', but the target element has type 'True'. At line:1 char:1 Get-AzSynapseSparkConfiguration -WorkspaceName ~~~~~~~~~~~~~~~~~ CategoryInfo : CloseError: (:) [Get-AzSynapseSparkConfiguration], InvalidOperationException FullyQualifiedErrorId : Microsoft.Azure.Commands.Synapse.GetAzureSynapseSparkConfiguration

[isra-fel]

Could be a breaking change of the backend API. Let me loop in Synapse team.

[microsoft-github-policy-service[bot]]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @wonner, @v-yanjungao.

[kevinzz6]

@Karni-G, could you please share your spark config definition by uploading a file on attachment? Thanks.

[Karni-G]

@Karni-G, could you please share your spark config definition by uploading a file on attachment? Thanks.

I'm not sure if sharing it as an attachment here would be a good idea since it is from customer environment. It is a standard config file, using default config Synapse param names. It was working earlier and has abruptly stopped. The problem/bug is with the powershell commands and associated API.

Having said above, if you are a FTE member of Synapse team (unable to find your ID), we can connect via call on Teams for this.