search

Azure / azure-powershell

Microsoft Azure PowerShell
Other
4.26k stars 3.87k forks source link

Set-AzSynapseLinkedService creates empty linked service #26079

Open nbarichev opened 2 months ago

nbarichev commented 2 months ago

Description

The Set-AzSynapseLinkedService command in PowerShell creates linked service of type AzureSqlDW with missing fields. The JSON file is copied from the Synapse workspace, tested, and successfully running (when created manually in the synapse workspace). The same behaviour applies to linked service of type AzureSqlMI.

Also there is issue for these types for az cli, but unlike PowerShell module az cli fails with error and does not create linked service at all.

Definition file that is provided to PowerShell command

{
    "name": "test_ls",
    "properties": {
        "annotations": [],
        "type": "AzureSqlDW",
        "typeProperties": {
            "server": "SERVER.sql.azuresynapse.net",
            "database": "DATABASE",
            "encrypt": "mandatory",
            "trustServerCertificate": false,
            "authenticationType": "SystemAssignedManagedIdentity"
        },
        "connectVia": {
            "referenceName": "ManagedNetworkIR",
            "type": "IntegrationRuntimeReference"
        }
    }
}

json content displayed in synapse after command successfully executed without errors

{
    "name": "test_ls",
    "type": "Microsoft.Synapse/workspaces/linkedservices",
    "properties": {
        "type": "AzureSqlDW",
        "connectVia": {
            "referenceName": "ManagedNetworkIR",
            "type": "IntegrationRuntimeReference"
        },
        "annotations": [],
        "typeProperties": {
            "connectionString": null
        }
    }
}

Issue script & Debug output

#> Set-AzSynapseLinkedService -WorkspaceName 'WORKSPACE_NAME' -Name 'TEST_LS'  -DefinitionFile ./FILE.json

DEBUG: False MSAL 4.61.3.0 MSAL.CoreCLR .NET 8.0.4 MacOS [2024-09-11 21:38:41Z - ce86d4b8-2a5a-46e3-9355-5a943b685210] === Token Acquisition (SilentRequest) started:
         Scopes: https://dev.azuresynapse.net/.default
        Authority Host: login.microsoftonline.com
DEBUG: False MSAL 4.61.3.0 MSAL.CoreCLR .NET 8.0.4 MacOS [2024-09-11 21:38:41Z - ce86d4b8-2a5a-46e3-9355-5a943b685210] [Region discovery] Not using a regional authority. 
DEBUG: False MSAL 4.61.3.0 MSAL.CoreCLR .NET 8.0.4 MacOS [2024-09-11 21:38:41Z - ce86d4b8-2a5a-46e3-9355-5a943b685210] Access token is not expired. Returning the found cache entry. [Current time (09/11/2024 21:38:41) - Expiration Time (09/11/2024 22:30:00 +00:00) - Extended Expiration Time (09/11/2024 22:30:00 +00:00)]
DEBUG: False MSAL 4.61.3.0 MSAL.CoreCLR .NET 8.0.4 MacOS [2024-09-11 21:38:41Z - ce86d4b8-2a5a-46e3-9355-5a943b685210] Returning access token found in cache. RefreshOn exists ? False
DEBUG: False MSAL 4.61.3.0 MSAL.CoreCLR .NET 8.0.4 MacOS [2024-09-11 21:38:41Z - ce86d4b8-2a5a-46e3-9355-5a943b685210] [Region discovery] Not using a regional authority. 
DEBUG: False MSAL 4.61.3.0 MSAL.CoreCLR .NET 8.0.4 MacOS [2024-09-11 21:38:41Z - ce86d4b8-2a5a-46e3-9355-5a943b685210] 
        === Token Acquisition finished successfully:
DEBUG: False MSAL 4.61.3.0 MSAL.CoreCLR .NET 8.0.4 MacOS [2024-09-11 21:38:41Z - ce86d4b8-2a5a-46e3-9355-5a943b685210]  AT expiration time: 9/11/2024 10:30:00 PM +00:00, scopes: https://dev.azuresynapse.net/.default https://dev.azuresynapse.net/user_impersonation. source: Cache
DEBUG: SharedTokenCacheCredential.GetToken succeeded. Scopes: [ https://dev.azuresynapse.net/.default ] ParentRequestId:  ExpiresOn: 2024-09-11T22:30:00.0000000+00:00
DEBUG: 12:38:41 AM - [ConfigManager] Got nothing from [DisableInstanceDiscovery], Module = [], Cmdlet = []. Returning default value [False].
DEBUG: 12:38:41 AM - [ConfigManager] Got nothing from [EnableLoginByWam], Module = [], Cmdlet = []. Returning default value [True].
DEBUG: 12:38:41 AM - [SilentAuthenticator] Calling SharedTokenCacheCredential.GetTokenAsync - TenantId:'TENANT_ID', Scopes:'https://dev.azuresynapse.net/.default', AuthorityHost:'https://login.microsoftonline.com/', UserId:'user@contoso.com'
DEBUG: SharedTokenCacheCredential.GetToken invoked. Scopes: [ https://dev.azuresynapse.net/.default ] ParentRequestId: 
DEBUG: False MSAL 4.61.3.0 MSAL.CoreCLR .NET 8.0.4 MacOS [2024-09-11 21:38:41Z - e4f1dffe-2515-4db1-aebd-26215da02657] IsLegacyAdalCacheEnabled: yes
DEBUG: False MSAL 4.61.3.0 MSAL.CoreCLR .NET 8.0.4 MacOS [2024-09-11 21:38:41Z - e4f1dffe-2515-4db1-aebd-26215da02657] [Region discovery] Not using a regional authority. 
DEBUG: False MSAL 4.61.3.0 MSAL.CoreCLR .NET 8.0.4 MacOS [2024-09-11 21:38:41Z - e4f1dffe-2515-4db1-aebd-26215da02657] [Region discovery] Not using a regional authority. 
DEBUG: False MSAL 4.61.3.0 MSAL.CoreCLR .NET 8.0.4 MacOS [2024-09-11 21:38:41Z - e4f1dffe-2515-4db1-aebd-26215da02657] IsLegacyAdalCacheEnabled: yes
DEBUG: False MSAL 4.61.3.0 MSAL.CoreCLR .NET 8.0.4 MacOS [2024-09-11 21:38:41Z - e4f1dffe-2515-4db1-aebd-26215da02657] IsLegacyAdalCacheEnabled: yes
DEBUG: False MSAL 4.61.3.0 MSAL.CoreCLR .NET 8.0.4 MacOS [2024-09-11 21:38:41Z] Found 1 cache accounts and 0 broker accounts
DEBUG: False MSAL 4.61.3.0 MSAL.CoreCLR .NET 8.0.4 MacOS [2024-09-11 21:38:41Z] Returning 1 accounts
DEBUG: False MSAL 4.61.3.0 MSAL.CoreCLR .NET 8.0.4 MacOS [2024-09-11 21:38:41Z - 1a236007-9cd7-443a-b813-19bbb0ae21bd] MSAL MSAL.CoreCLR with assembly version '4.61.3.0'. CorrelationId(1a236007-9cd7-443a-b813-19bbb0ae21bd)
DEBUG: False MSAL 4.61.3.0 MSAL.CoreCLR .NET 8.0.4 MacOS [2024-09-11 21:38:41Z - 1a236007-9cd7-443a-b813-19bbb0ae21bd] === AcquireTokenSilent Parameters ===
DEBUG: False MSAL 4.61.3.0 MSAL.CoreCLR .NET 8.0.4 MacOS [2024-09-11 21:38:41Z - 1a236007-9cd7-443a-b813-19bbb0ae21bd] LoginHint provided: False
DEBUG: False MSAL 4.61.3.0 MSAL.CoreCLR .NET 8.0.4 MacOS [2024-09-11 21:38:41Z - 1a236007-9cd7-443a-b813-19bbb0ae21bd] Account provided: True
DEBUG: False MSAL 4.61.3.0 MSAL.CoreCLR .NET 8.0.4 MacOS [2024-09-11 21:38:41Z - 1a236007-9cd7-443a-b813-19bbb0ae21bd] ForceRefresh: False
DEBUG: False MSAL 4.61.3.0 MSAL.CoreCLR .NET 8.0.4 MacOS [2024-09-11 21:38:41Z - 1a236007-9cd7-443a-b813-19bbb0ae21bd] 
=== Request Data ===
Authority Provided? - True
Scopes - https://dev.azuresynapse.net/.default
Extra Query Params Keys (space separated) - 
ApiId - AcquireTokenSilent
IsConfidentialClient - False
SendX5C - False
LoginHint ? False
IsBrokerConfigured - False
HomeAccountId - False
CorrelationId - 1a236007-9cd7-443a-b813-19bbb0ae21bd
UserAssertion set: False
LongRunningOboCacheKey set: False
Region configured: 

DEBUG: False MSAL 4.61.3.0 MSAL.CoreCLR .NET 8.0.4 MacOS [2024-09-11 21:38:41Z - 1a236007-9cd7-443a-b813-19bbb0ae21bd] === Token Acquisition (SilentRequest) started:
         Scopes: https://dev.azuresynapse.net/.default
        Authority Host: login.microsoftonline.com
DEBUG: False MSAL 4.61.3.0 MSAL.CoreCLR .NET 8.0.4 MacOS [2024-09-11 21:38:41Z - 1a236007-9cd7-443a-b813-19bbb0ae21bd] [Region discovery] Not using a regional authority. 
DEBUG: False MSAL 4.61.3.0 MSAL.CoreCLR .NET 8.0.4 MacOS [2024-09-11 21:38:41Z - 1a236007-9cd7-443a-b813-19bbb0ae21bd] Access token is not expired. Returning the found cache entry. [Current time (09/11/2024 21:38:41) - Expiration Time (09/11/2024 22:30:00 +00:00) - Extended Expiration Time (09/11/2024 22:30:00 +00:00)]
DEBUG: False MSAL 4.61.3.0 MSAL.CoreCLR .NET 8.0.4 MacOS [2024-09-11 21:38:41Z - 1a236007-9cd7-443a-b813-19bbb0ae21bd] Returning access token found in cache. RefreshOn exists ? False
DEBUG: False MSAL 4.61.3.0 MSAL.CoreCLR .NET 8.0.4 MacOS [2024-09-11 21:38:41Z - 1a236007-9cd7-443a-b813-19bbb0ae21bd] [Region discovery] Not using a regional authority. 
DEBUG: False MSAL 4.61.3.0 MSAL.CoreCLR .NET 8.0.4 MacOS [2024-09-11 21:38:41Z - 1a236007-9cd7-443a-b813-19bbb0ae21bd] 
        === Token Acquisition finished successfully:
DEBUG: False MSAL 4.61.3.0 MSAL.CoreCLR .NET 8.0.4 MacOS [2024-09-11 21:38:41Z - 1a236007-9cd7-443a-b813-19bbb0ae21bd]  AT expiration time: 9/11/2024 10:30:00 PM +00:00, scopes: https://dev.azuresynapse.net/.default https://dev.azuresynapse.net/user_impersonation. source: Cache
DEBUG: SharedTokenCacheCredential.GetToken succeeded. Scopes: [ https://dev.azuresynapse.net/.default ] ParentRequestId:  ExpiresOn: 2024-09-11T22:30:00.0000000+00:00
DEBUG: Request [4f491a5b-5d16-46d8-a1b6-b1b879cdeaa9] PUT https://SYNAPSE_NAME.dev.azuresynapse.net/linkedservices/test_ls?api-version=2020-12-01
Accept:application/json
Content-Type:application/json
x-ms-client-request-id:4f491a5b-5d16-46d8-a1b6-b1b879cdeaa9
x-ms-return-client-request-id:true
User-Agent:azsdk-net-Analytics.Synapse.Artifacts/1.0.0-preview.20 (.NET 8.0.4; Darwin 23.4.0 Darwin Kernel Version 23.4.0: Fri Mar 15 00:10:42 PDT 2024; root:xnu-10063.101.17~1/RELEASE_ARM64_T6000)
Authorization:REDACTED
client assembly: Azure.Analytics.Synapse.Artifacts
DEBUG: Response [4f491a5b-5d16-46d8-a1b6-b1b879cdeaa9] 202 Accepted (01.0s)
Location:REDACTED
Retry-After:10
Server:Microsoft-HTTPAPI/2.0
Strict-Transport-Security:REDACTED
Access-Control-Allow-Headers:REDACTED
Access-Control-Expose-Headers:REDACTED
x-ms-request-id:d288019a-ee6a-4115-9ab4-b3245fef2f49
x-ms-client-request-id:4f491a5b-5d16-46d8-a1b6-b1b879cdeaa9
Date:Wed, 11 Sep 2024 21:38:41 GMT
Content-Length:427
Content-Type:application/json; charset=utf-8

DEBUG: Request [123afc87-a838-4803-b070-8eecb9eee1d8] GET https://SYNAPSE_NAME.dev.azuresynapse.net/operationResults/72460ffd-39c7-46fb-bb0e-d14067bef6e1?api-version=2020-12-01
x-ms-client-request-id:123afc87-a838-4803-b070-8eecb9eee1d8
x-ms-return-client-request-id:true
User-Agent:azsdk-net-Analytics.Synapse.Artifacts/1.0.0-preview.20 (.NET 8.0.4; Darwin 23.4.0 Darwin Kernel Version 23.4.0: Fri Mar 15 00:10:42 PDT 2024; root:xnu-10063.101.17~1/RELEASE_ARM64_T6000)
Authorization:REDACTED
client assembly: Azure.Analytics.Synapse.Artifacts
DEBUG: Response [123afc87-a838-4803-b070-8eecb9eee1d8] 202 Accepted (00.2s)
Location:REDACTED
Retry-After:10
Server:Microsoft-HTTPAPI/2.0
Strict-Transport-Security:REDACTED
Access-Control-Allow-Headers:REDACTED
Access-Control-Expose-Headers:REDACTED
x-ms-request-id:e8418eea-c278-4d0c-a5e8-b8a38ac16554
x-ms-client-request-id:123afc87-a838-4803-b070-8eecb9eee1d8
Date:Wed, 11 Sep 2024 21:38:41 GMT
Content-Length:23
Content-Type:application/json; charset=utf-8

DEBUG: Request [fc09e9e5-8d0d-4069-9736-8d6424926e0b] GET https://SYNAPSE_NAME.dev.azuresynapse.net/operationResults/72460ffd-39c7-46fb-bb0e-d14067bef6e1?api-version=2020-12-01
x-ms-client-request-id:fc09e9e5-8d0d-4069-9736-8d6424926e0b
x-ms-return-client-request-id:true
User-Agent:azsdk-net-Analytics.Synapse.Artifacts/1.0.0-preview.20 (.NET 8.0.4; Darwin 23.4.0 Darwin Kernel Version 23.4.0: Fri Mar 15 00:10:42 PDT 2024; root:xnu-10063.101.17~1/RELEASE_ARM64_T6000)
Authorization:REDACTED
client assembly: Azure.Analytics.Synapse.Artifacts
DEBUG: Response [fc09e9e5-8d0d-4069-9736-8d6424926e0b] 200 OK (00.5s)
Server:Kestrel,Microsoft-HTTPAPI/2.0
Strict-Transport-Security:REDACTED
x-ms-correlation-request-id:REDACTED
x-ms-request-id:fc20fa87-bb6e-45be-ba40-bb7145d55cdc
x-ms-client-request-id:fc09e9e5-8d0d-4069-9736-8d6424926e0b
Date:Wed, 11 Sep 2024 21:38:52 GMT
Content-Length:547
Content-Type:application/json; charset=utf-8

DEBUG: 12:38:52 AM - [ConfigManager] Got nothing from [DisplaySecretsWarning], Module = [], Cmdlet = []. Returning default value [True].

WorkspaceName : SYNAPSE_NAME
Properties    : Azure.Analytics.Synapse.Artifacts.Models.AzureSqlDWLinkedService
Id            : /subscriptions/SUBSCRIPTION_ID/resourceGroups/RESOURCE_GROUP/providers/Microsoft.Synapse/workspaces/SYNAPSE_NAME/linkedservices/test_
                ls
Name          : test_ls
Type          : Microsoft.Synapse/workspaces/linkedservices
Etag          : 4600c0bc-0000-0200-0000-66e205610000

DEBUG: 12:38:52 AM - [ConfigManager] Got nothing from [DisplayRegionIdentified], Module = [], Cmdlet = []. Returning default value [True].
DEBUG: 12:38:52 AM - [ConfigManager] Got nothing from [CheckForUpgrade], Module = [], Cmdlet = []. Returning default value [True].
DEBUG: AzureQoSEvent:  Module: Az.Synapse:3.0.10; CommandName: Set-AzSynapseLinkedService; PSVersion: 7.4.2; IsSuccess: True; Duration: 00:00:12.2426281; SanitizeDuration: 00:00:00.0001897
DEBUG: 12:38:52 AM - [ConfigManager] Got nothing from [EnableDataCollection], Module = [], Cmdlet = []. Returning default value [True].
DEBUG: 12:38:52 AM - SetAzureSynapseLinkedService end processing.

### Environment data

```PowerShell
Name                           Value
----                           -----
PSVersion                      7.4.2
PSEdition                      Core
GitCommitId                    7.4.2
OS                             Darwin 23.4.0 Darwin Kernel Version 23.4.0: Fri Mar 15 00:10:42 PDT 2024; root:xnu-10063.101.17~1/RELEASE_ARM64_T6000
Platform                       Unix
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

### Module versions

```PowerShell
ModuleType Version    PreRelease Name                                ExportedCommands
---------- -------    ---------- ----                                ----------------
Script     3.0.4                 Az.Accounts                         {Add-AzEnvironment, Clear-AzConfig, Clear-AzContext, Clear-AzDefault…}
Script     3.0.10                Az.Synapse                          {Add-AzSynapseDataFlowDebugSessionPackage, Add-AzSynapseTriggerSubscription, Clear-AzSynapseSqlPoolVul…


### Error output

_No response_
microsoft-github-policy-service[bot] commented 2 months ago

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @wonner, @v-yanjungao.

wanyang7 commented 2 months ago

please use this as a workaround

#Create a linked service
$token=Get-AzAccessToken -ResourceUrl https://dev.azuresynapse.net/

$authHeader=@{
'Content-Type'='application/octet-stream' 
'Authorization'='Bearer '+$token.Token
}
$workspaceName ="<your-worksapce-name>"
$linkedServiceName ="<your-linkedservice-name>"
$linkedServiceFile = "<your-linkedservice-definition-File>"  
$json = Get-Content  $linkedServiceFile 
#Set linkedService
Invoke-WebRequest -Method PUT -Uri "https://${workspaceName}.dev.azuresynapse.net/pipelines/${linkedServiceName}?api-version=2019-06-01-preview" -Headers $authHeader -Body $json