search

Azure / azure-powershell

Microsoft Azure PowerShell
Other
4.26k stars 3.86k forks source link

New-AzDataFactoryV2Pipeline creates corrupt pipeline if it contains a lookup to SQL Server dataset with Table "None" #9431

Closed Limp-monkey closed 5 years ago

Limp-monkey commented 5 years ago

Description

The issue starts with the fact that it’s possible to create a dataset in Azure Data Factory, which is a SQL server MI dataset, which doesn’t point to any table. image This is possible to do in the portal, but not with PowerShell scripts, and I believe it’s a bug as indicated here: https://github.com/Azure/azure-powershell/issues/9261

However, when creating a pipeline in a new ADF using a pipeline with reference to such a dataset, it will allow to create the pipeline, but with an invalid JSON . Even if the dataset is corrected in the target dataset. So if we have ADF_1 • Dataset_A with sql server managed instance dataset without any table • Pipeline_B using an activity lookup to dataset_A

When running New-AzDataFactoryV2Dataset on dataset_A to copy ADF_2 based on JSON from ADF_1, it will fail as indicated in the bug above. However, if I create it manually in ADF_2 as it is in ADF_1 and then run New-AzDataFactoryV2Pipeline to move Pipeline_B to ADF_2 it will succeed, but the pipeline_B JSON in ADF_2 will be corrupt. The reason why it will be corrupt is that in the lookup will change type SqlServerSource to type CopySource. If I manually change that value to SqlServerSource, it will validate. image image

Steps to reproduce

New-AzDataFactoryV2Pipeline -Name $pipeline -DataFactoryName $env:PIPELINE_TARGET_DATA_FACTORY_NAME -ResourceGroupName $env:PIPELINE_TARGET_RESOURCE_GROUP_NAME -DefinitionFile "$path_json\pipeline\$pipeline.json" -Force

Environment data

Name                           Value                                                                                                                                                                                                                             
----                           -----                                                                                                                                                                                                                             
PSVersion                      5.1.17134.590                                                                                                                                                                                                                     
PSEdition                      Desktop                                                                                                                                                                                                                           
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}                                                                                                                                                                                                           
BuildVersion                   10.0.17134.590                                                                                                                                                                                                                    
CLRVersion                     4.0.30319.42000                                                                                                                                                                                                                   
WSManStackVersion              3.0                                                                                                                                                                                                                               
PSRemotingProtocolVersion      2.3                                                                                                                                                                                                                               
SerializationVersion           1.1.0.1

Module versions

    Directory: C:\Users\jkolodziejcz\Documents\WindowsPowerShell\Modules

ModuleType Version    Name                                ExportedCommands                                                                                                                                                                                       
---------- -------    ----                                ----------------                                                                                                                                                                                       
Script     1.4.0      Az.Accounts                         {Disable-AzDataCollection, Disable-AzContextAutosave, Enable-AzDataCollection, Enable-AzContextAutosave...}                                                                                            
Script     1.0.1      Az.Aks                              {Get-AzAks, New-AzAks, Remove-AzAks, Import-AzAksCredential...}                                                                                                                                        
Script     1.0.2      Az.AnalysisServices                 {Resume-AzAnalysisServicesServer, Suspend-AzAnalysisServicesServer, Get-AzAnalysisServicesServer, Remove-AzAnalysisServicesServer...}                                                                  
Script     1.0.0      Az.ApiManagement                    {Add-AzApiManagementRegion, Get-AzApiManagementSsoToken, New-AzApiManagementCustomHostnameConfiguration, New-AzApiManagementSystemCertificate...}                                                      
Script     1.0.0      Az.ApplicationInsights              {Get-AzApplicationInsights, New-AzApplicationInsights, Remove-AzApplicationInsights, Set-AzApplicationInsightsPricingPlan...}                                                                          
Script     1.2.0      Az.Automation                       {Get-AzAutomationHybridWorkerGroup, Remove-AzAutomationHybridWorkerGroup, Get-AzAutomationJobOutputRecord, Import-AzAutomationDscNodeConfiguration...}                                                 
Script     1.0.0      Az.Batch                            {Remove-AzBatchAccount, Get-AzBatchAccount, Get-AzBatchAccountKeys, New-AzBatchAccount...}                                                                                                             
Script     1.0.0      Az.Billing                          {Get-AzBillingInvoice, Get-AzBillingPeriod, Get-AzEnrollmentAccount, Get-AzConsumptionBudget...}                                                                                                       
Script     1.1.0      Az.Cdn                              {Get-AzCdnProfile, Get-AzCdnProfileSsoUrl, New-AzCdnProfile, Remove-AzCdnProfile...}                                                                                                                   
Script     1.0.1      Az.CognitiveServices                {Get-AzCognitiveServicesAccount, Get-AzCognitiveServicesAccountKey, Get-AzCognitiveServicesAccountSkus, Get-AzCognitiveServicesAccountType...}                                                         
Script     1.6.0      Az.Compute                          {Remove-AzAvailabilitySet, Get-AzAvailabilitySet, New-AzAvailabilitySet, Update-AzAvailabilitySet...}                                                                                                  
Script     1.0.0      Az.ContainerInstance                {New-AzContainerGroup, Get-AzContainerGroup, Remove-AzContainerGroup, Get-AzContainerInstanceLog}                                                                                                      
Script     1.0.1      Az.ContainerRegistry                {New-AzContainerRegistry, Get-AzContainerRegistry, Update-AzContainerRegistry, Remove-AzContainerRegistry...}                                                                                          
Script     1.0.2      Az.DataFactory                      {Set-AzDataFactoryV2, Update-AzDataFactoryV2, Get-AzDataFactoryV2, Remove-AzDataFactoryV2...}                                                                                                          
Script     1.0.0      Az.DataLakeAnalytics                {Get-AzDataLakeAnalyticsDataSource, New-AzDataLakeAnalyticsCatalogCredential, Remove-AzDataLakeAnalyticsCatalogCredential, Set-AzDataLakeAnalyticsCatalogCredential...}                                
Script     1.1.0      Az.DataLakeStore                    {Get-AzDataLakeStoreTrustedIdProvider, Remove-AzDataLakeStoreTrustedIdProvider, Remove-AzDataLakeStoreFirewallRule, Set-AzDataLakeStoreTrustedIdProvider...}                                           
Script     1.0.0      Az.DevTestLabs                      {Get-AzDtlAllowedVMSizesPolicy, Get-AzDtlAutoShutdownPolicy, Get-AzDtlAutoStartPolicy, Get-AzDtlVMsPerLabPolicy...}                                                                                    
Script     1.0.0      Az.Dns                              {Get-AzDnsRecordSet, New-AzDnsRecordConfig, Remove-AzDnsRecordSet, Set-AzDnsRecordSet...}                                                                                                              
Script     1.1.0      Az.EventGrid                        {New-AzEventGridTopic, Get-AzEventGridTopic, Set-AzEventGridTopic, New-AzEventGridTopicKey...}                                                                                                         
Script     1.0.1      Az.EventHub                         {New-AzEventHubNamespace, Get-AzEventHubNamespace, Set-AzEventHubNamespace, Remove-AzEventHubNamespace...}                                                                                             
Script     1.0.0      Az.HDInsight                        {Get-AzHDInsightJob, New-AzHDInsightSqoopJobDefinition, Wait-AzHDInsightJob, New-AzHDInsightStreamingMapReduceJobDefinition...}                                                                        
Script     1.0.2      Az.IotHub                           {Add-AzIotHubKey, Get-AzIotHubEventHubConsumerGroup, Get-AzIotHubConnectionString, Get-AzIotHubJob...}                                                                                                 
Script     1.1.0      Az.KeyVault                         {Add-AzKeyVaultCertificate, Update-AzKeyVaultCertificate, Stop-AzKeyVaultCertificateOperation, Get-AzKeyVaultCertificateOperation...}                                                                  
Script     1.2.1      Az.LogicApp                         {Get-AzIntegrationAccountAgreement, Get-AzIntegrationAccountAssembly, Get-AzIntegrationAccountBatchConfiguration, Get-AzIntegrationAccountCallbackUrl...}                                              
Script     1.0.0      Az.MachineLearning                  {Move-AzMlCommitmentAssociation, Get-AzMlCommitmentAssociation, Get-AzMlCommitmentPlanUsageHistory, Remove-AzMlCommitmentPlan...}                                                                      
Script     1.0.0      Az.MarketplaceOrdering              {Get-AzMarketplaceTerms, Set-AzMarketplaceTerms}                                                                                                                                                       
Script     1.0.0      Az.Media                            {Sync-AzMediaServiceStorageKeys, Set-AzMediaServiceKey, Get-AzMediaServiceKeys, Get-AzMediaServiceNameAvailability...}                                                                                 
Script     1.0.1      Az.Monitor                          {Get-AzMetricDefinition, Get-AzMetric, Remove-AzLogProfile, Get-AzLogProfile...}                                                                                                                       
Script     1.6.0      Az.Network                          {Add-AzApplicationGatewayAuthenticationCertificate, Get-AzApplicationGatewayAuthenticationCertificate, New-AzApplicationGatewayAuthenticationCertificate, Remove-AzApplicationGatewayAuthenticationC...
Script     1.0.0      Az.NotificationHubs                 {Get-AzNotificationHub, Get-AzNotificationHubAuthorizationRules, Get-AzNotificationHubListKeys, Get-AzNotificationHubPNSCredentials...}                                                                
Script     1.1.0      Az.OperationalInsights              {New-AzOperationalInsightsAzureActivityLogDataSource, New-AzOperationalInsightsCustomLogDataSource, Disable-AzOperationalInsightsLinuxCustomLogCollection, Disable-AzOperationalInsightsIISLogCollec...
Script     1.0.0      Az.PolicyInsights                   {Get-AzPolicyEvent, Get-AzPolicyState, Get-AzPolicyStateSummary, Get-AzPolicyRemediation...}                                                                                                           
Script     1.0.0      Az.PowerBIEmbedded                  {Remove-AzPowerBIWorkspaceCollection, Get-AzPowerBIWorkspaceCollection, Get-AzPowerBIWorkspaceCollectionAccessKeys, Get-AzPowerBIWorkspace...}                                                         
Script     1.2.0      Az.RecoveryServices                 {Get-AzRecoveryServicesBackupProperty, Get-AzRecoveryServicesVault, Get-AzRecoveryServicesVaultSettingsFile, New-AzRecoveryServicesVault...}                                                           
Script     1.0.0      Az.RedisCache                       {Remove-AzRedisCachePatchSchedule, New-AzRedisCacheScheduleEntry, Get-AzRedisCachePatchSchedule, New-AzRedisCachePatchSchedule...}                                                                     
Script     1.0.0      Az.Relay                            {New-AzRelayNamespace, Get-AzRelayNamespace, Set-AzRelayNamespace, Remove-AzRelayNamespace...}                                                                                                         
Script     1.2.1      Az.Resources                        {Get-AzProviderOperation, Remove-AzRoleAssignment, Get-AzRoleAssignment, New-AzRoleAssignment...}                                                                                                      
Script     1.0.0      Az.ServiceBus                       {New-AzServiceBusNamespace, Get-AzServiceBusNamespace, Set-AzServiceBusNamespace, Remove-AzServiceBusNamespace...}                                                                                     
Script     1.0.1      Az.ServiceFabric                    {Add-AzServiceFabricApplicationCertificate, Add-AzServiceFabricClientCertificate, Add-AzServiceFabricClusterCertificate, Add-AzServiceFabricNode...}                                                   
Script     1.0.2      Az.SignalR                          {New-AzSignalR, Get-AzSignalR, Get-AzSignalRKey, New-AzSignalRKey...}                                                                                                                                  
Script     1.7.0      Az.Sql                              {Get-AzSqlDatabaseTransparentDataEncryption, Get-AzSqlDatabaseTransparentDataEncryptionActivity, Set-AzSqlDatabaseTransparentDataEncryption, Get-AzSqlDatabaseUpgradeHint...}                          
Script     1.1.0      Az.Storage                          {Get-AzStorageAccount, Get-AzStorageAccountKey, New-AzStorageAccount, New-AzStorageAccountKey...}                                                                                                      
Script     1.0.0      Az.StreamAnalytics                  {Get-AzStreamAnalyticsFunction, Get-AzStreamAnalyticsDefaultFunctionDefinition, New-AzStreamAnalyticsFunction, Remove-AzStreamAnalyticsFunction...}                                                    
Script     1.0.1      Az.TrafficManager                   {Add-AzTrafficManagerCustomHeaderToEndpoint, Remove-AzTrafficManagerCustomHeaderFromEndpoint, Add-AzTrafficManagerCustomHeaderToProfile, Remove-AzTrafficManagerCustomHeaderFromProfile...}            
Script     1.1.2      Az.Websites                         {Get-AzAppServicePlan, Set-AzAppServicePlan, New-AzAppServicePlan, Remove-AzAppServicePlan...}                                                                                                         

    Directory: C:\Program Files\WindowsPowerShell\Modules

ModuleType Version    Name                                ExportedCommands                                                                                                                                                                                       
---------- -------    ----                                ----------------                                                                                                                                                                                       
Script     5.1.2      Azure                               {Get-AzureAutomationCertificate, Get-AzureAutomationConnection, New-AzureAutomationConnection, Remove-AzureAutomationConnection...}                                                                    
Manifest   0.5.0      Azure.AnalysisServices              {Add-AzureAnalysisServicesAccount, Restart-AzureAnalysisServicesInstance, Export-AzureAnalysisServicesInstanceLog, Sync-AzureAnalysisServicesInstance}                                                 
Script     4.2.1      Azure.Storage                       {Get-AzureStorageTable, New-AzureStorageTableSASToken, New-AzureStorageTableStoredAccessPolicy, New-AzureStorageTable...}                                                                              
Script     5.7.0      AzureRM                                                                                                                                                                                                                                    
Script     0.6.6      AzureRM.AnalysisServices            {Resume-AzureRmAnalysisServicesServer, Suspend-AzureRmAnalysisServicesServer, Get-AzureRmAnalysisServicesServer, Remove-AzureRmAnalysisServicesServer...}                                              
Script     5.1.2      AzureRM.ApiManagement               {Add-AzureRmApiManagementRegion, Get-AzureRmApiManagementSsoToken, New-AzureRmApiManagementHostnameConfiguration, New-AzureRmApiManagementRegion...}                                                   
Script     0.1.3      AzureRM.ApplicationInsights         {Get-AzureRmApplicationInsights, New-AzureRmApplicationInsights, Remove-AzureRmApplicationInsights, Set-AzureRmApplicationInsightsPricingPlan...}                                                      
Script     4.3.2      AzureRM.Automation                  {Get-AzureRMAutomationHybridWorkerGroup, Get-AzureRmAutomationJobOutputRecord, Import-AzureRmAutomationDscNodeConfiguration, Export-AzureRmAutomationDscConfiguration...}                              
Script     4.0.4      AzureRM.Backup                      {Backup-AzureRmBackupItem, Enable-AzureRmBackupContainerReregistration, Get-AzureRmBackupContainer, Register-AzureRmBackupContainer...}                                                                
Script     4.0.6      AzureRM.Batch                       {Remove-AzureRmBatchAccount, Get-AzureRmBatchAccount, Get-AzureRmBatchAccountKeys, New-AzureRmBatchAccount...}                                                                                         
Script     0.14.1     AzureRM.Billing                     {Get-AzureRmBillingInvoice, Get-AzureRmBillingPeriod, Get-AzureRmEnrollmentAccount}                                                                                                                    
Script     4.2.2      AzureRM.Cdn                         {Get-AzureRmCdnProfile, Get-AzureRmCdnProfileSsoUrl, New-AzureRmCdnProfile, Remove-AzureRmCdnProfile...}                                                                                               
Script     0.9.4      AzureRM.CognitiveServices           {Get-AzureRmCognitiveServicesAccount, Get-AzureRmCognitiveServicesAccountKey, Get-AzureRmCognitiveServicesAccountSkus, Get-AzureRmCognitiveServicesAccountUsage...}                                    
Script     4.6.0      AzureRM.Compute                     {Remove-AzureRmAvailabilitySet, Get-AzureRmAvailabilitySet, New-AzureRmAvailabilitySet, Update-AzureRmAvailabilitySet...}                                                                              
Script     0.3.1      AzureRM.Consumption                 Get-AzureRmConsumptionUsageDetail                                                                                                                                                                      
Script     0.2.5      AzureRM.ContainerInstance           {New-AzureRmContainerGroup, Get-AzureRmContainerGroup, Remove-AzureRmContainerGroup, Get-AzureRmContainerInstanceLog}                                                                                  
Script     1.0.4      AzureRM.ContainerRegistry           {New-AzureRmContainerRegistry, Get-AzureRmContainerRegistry, Update-AzureRmContainerRegistry, Remove-AzureRmContainerRegistry...}                                                                      
Script     4.2.2      AzureRM.DataFactories               {Remove-AzureRmDataFactory, Get-AzureRmDataFactoryRun, Get-AzureRmDataFactorySlice, Save-AzureRmDataFactoryLog...}                                                                                     
Script     0.5.3      AzureRM.DataFactoryV2               {Set-AzureRmDataFactoryV2, Update-AzureRmDataFactoryV2, Get-AzureRmDataFactoryV2, Remove-AzureRmDataFactoryV2...}                                                                                      
Script     4.2.3      AzureRM.DataLakeAnalytics           {Get-AzureRmDataLakeAnalyticsDataSource, New-AzureRmDataLakeAnalyticsCatalogCredential, Remove-AzureRmDataLakeAnalyticsCatalogCredential, Remove-AzureRmDataLakeAnalyticsCatalogSecret...}             
Script     5.2.0      AzureRM.DataLakeStore               {Get-AzureRmDataLakeStoreTrustedIdProvider, Remove-AzureRmDataLakeStoreTrustedIdProvider, Remove-AzureRmDataLakeStoreFirewallRule, Set-AzureRmDataLakeStoreTrustedIdProvider...}                       
Script     4.0.4      AzureRM.DevTestLabs                 {Get-AzureRmDtlAllowedVMSizesPolicy, Get-AzureRmDtlAutoShutdownPolicy, Get-AzureRmDtlAutoStartPolicy, Get-AzureRmDtlVMsPerLabPolicy...}                                                                
Script     4.1.2      AzureRM.Dns                         {Get-AzureRmDnsRecordSet, New-AzureRmDnsRecordConfig, Remove-AzureRmDnsRecordSet, Set-AzureRmDnsRecordSet...}                                                                                          
Script     0.3.2      AzureRM.EventGrid                   {New-AzureRmEventGridTopic, Get-AzureRmEventGridTopic, Set-AzureRmEventGridTopic, New-AzureRmEventGridTopicKey...}                                                                                     
Script     0.6.3      AzureRM.EventHub                    {New-AzureRmEventHubNamespace, Get-AzureRmEventHubNamespace, Set-AzureRmEventHubNamespace, Remove-AzureRmEventHubNamespace...}                                                                         
Script     4.1.2      AzureRM.HDInsight                   {Get-AzureRmHDInsightJob, New-AzureRmHDInsightSqoopJobDefinition, Wait-AzureRmHDInsightJob, New-AzureRmHDInsightStreamingMapReduceJobDefinition...}                                                    
Script     4.0.4      AzureRM.Insights                    {Get-AzureRmMetricDefinition, Get-AzureRmMetric, Remove-AzureRmLogProfile, Get-AzureRmLogProfile...}                                                                                                   
Script     3.1.2      AzureRM.IotHub                      {Add-AzureRmIotHubKey, Get-AzureRmIotHubEventHubConsumerGroup, Get-AzureRmIotHubConnectionString, Get-AzureRmIotHubJob...}                                                                             
Script     4.3.0      AzureRM.KeyVault                    {Add-AzureKeyVaultCertificate, Set-AzureKeyVaultCertificateAttribute, Stop-AzureKeyVaultCertificateOperation, Get-AzureKeyVaultCertificateOperation...}                                                
Script     4.0.3      AzureRM.LogicApp                    {Get-AzureRmIntegrationAccountAgreement, Get-AzureRmIntegrationAccountCallbackUrl, Get-AzureRmIntegrationAccountCertificate, Get-AzureRmIntegrationAccount...}                                         
Script     0.17.2     AzureRM.MachineLearning             {Move-AzureRmMlCommitmentAssociation, Get-AzureRmMlCommitmentAssociation, Get-AzureRmMlCommitmentPlanUsageHistory, Remove-AzureRmMlCommitmentPlan...}                                                  
Script     0.4.2      AzureRM.MachineLearningCompute      {Get-AzureRmMlOpCluster, Get-AzureRmMlOpClusterKey, Test-AzureRmMlOpClusterSystemServicesUpdateAvailability, Update-AzureRmMlOpClusterSystemService...}                                                
Script     0.2.1      AzureRM.MarketplaceOrdering         {Get-AzureRmMarketplaceTerms, Set-AzureRmMarketplaceTerms}                                                                                                                                             
Script     0.9.2      AzureRM.Media                       {Sync-AzureRmMediaServiceStorageKeys, Set-AzureRmMediaServiceKey, Get-AzureRmMediaServiceKeys, Get-AzureRmMediaServiceNameAvailability...}                                                             
Script     5.4.2      AzureRM.Network                     {Add-AzureRmApplicationGatewayAuthenticationCertificate, Get-AzureRmApplicationGatewayAuthenticationCertificate, New-AzureRmApplicationGatewayAuthenticationCertificate, Remove-AzureRmApplicationGa...
Script     4.1.1      AzureRM.NotificationHubs            {Get-AzureRmNotificationHub, Get-AzureRmNotificationHubAuthorizationRules, Get-AzureRmNotificationHubListKeys, Get-AzureRmNotificationHubPNSCredentials...}                                            
Script     4.3.2      AzureRM.OperationalInsights         {New-AzureRmOperationalInsightsAzureActivityLogDataSource, New-AzureRmOperationalInsightsCustomLogDataSource, Disable-AzureRmOperationalInsightsLinuxCustomLogCollection, Disable-AzureRmOperational...
Script     4.1.4      AzureRM.PowerBIEmbedded             {Remove-AzureRmPowerBIWorkspaceCollection, Get-AzureRmPowerBIWorkspaceCollection, Get-AzureRmPowerBIWorkspaceCollectionAccessKeys, Get-AzureRmPowerBIWorkspace...}                                     
Script     4.6.0      AzureRM.Profile                     {Disable-AzureRmDataCollection, Disable-AzureRmContextAutosave, Enable-AzureRmDataCollection, Enable-AzureRmContextAutosave...}                                                                        
Script     4.1.2      AzureRM.RecoveryServices            {Get-AzureRmRecoveryServicesBackupProperty, Get-AzureRmRecoveryServicesVault, Get-AzureRmRecoveryServicesVaultSettingsFile, New-AzureRmRecoveryServicesVault...}                                       
Script     4.1.2      AzureRM.RecoveryServices.Backup     {Backup-AzureRmRecoveryServicesBackupItem, Get-AzureRmRecoveryServicesBackupManagementServer, Get-AzureRmRecoveryServicesBackupContainer, Unregister-AzureRmRecoveryServicesBackupContainer...}        
Script     0.2.4      AzureRM.RecoveryServices.SiteRec... {Edit-AzureRmRecoveryServicesAsrRecoveryPlan, Get-AzureRmRecoveryServicesAsrAlertSetting, Get-AzureRmRecoveryServicesAsrEvent, Get-AzureRmRecoveryServicesAsrFabric...}                                
Script     4.1.2      AzureRM.RedisCache                  {Remove-AzureRmRedisCachePatchSchedule, New-AzureRmRedisCacheScheduleEntry, Get-AzureRmRedisCachePatchSchedule, New-AzureRmRedisCachePatchSchedule...}                                                 
Script     0.3.3      AzureRM.Relay                       {New-AzureRmRelayNamespace, Get-AzureRmRelayNamespace, Set-AzureRmRelayNamespace, Remove-AzureRmRelayNamespace...}                                                                                     
Script     5.5.2      AzureRM.Resources                   {Get-AzureRmProviderOperation, Remove-AzureRmRoleAssignment, Get-AzureRmRoleAssignment, New-AzureRmRoleAssignment...}                                                                                  
Script     0.16.3     AzureRM.Scheduler                   {Disable-AzureRmSchedulerJobCollection, Enable-AzureRmSchedulerJobCollection, Get-AzureRmSchedulerJobCollection, Get-AzureRmSchedulerJob...}                                                           
Script     4.1.2      AzureRM.ServerManagement            {Invoke-AzureRmServerManagementPowerShellCommand, Get-AzureRmServerManagementSession, New-AzureRmServerManagementSession, Remove-AzureRmServerManagementSession...}                                    
Script     0.6.4      AzureRM.ServiceBus                  {New-AzureRmServiceBusNamespace, Get-AzureRmServiceBusNamespace, Set-AzureRmServiceBusNamespace, Remove-AzureRmServiceBusNamespace...}                                                                 
Script     0.3.4      AzureRM.ServiceFabric               {Add-AzureRmServiceFabricApplicationCertificate, Add-AzureRmServiceFabricClientCertificate, Add-AzureRmServiceFabricClusterCertificate, Add-AzureRmServiceFabricNode...}                               
Script     5.0.6      AzureRM.SiteRecovery                {Get-AzureRmSiteRecoveryFabric, New-AzureRmSiteRecoveryFabric, Remove-AzureRmSiteRecoveryFabric, Stop-AzureRmSiteRecoveryJob...}                                                                       
Script     4.4.0      AzureRM.Sql                         {Get-AzureRmSqlDatabaseTransparentDataEncryption, Get-AzureRmSqlDatabaseTransparentDataEncryptionActivity, Set-AzureRmSqlDatabaseTransparentDataEncryption, Get-AzureRmSqlDatabaseUpgradeHint...}      
Script     4.2.3      AzureRM.Storage                     {Get-AzureRmStorageAccount, Get-AzureRmStorageAccountKey, New-AzureRmStorageAccount, New-AzureRmStorageAccountKey...}                                                                                  
Script     4.0.4      AzureRM.StreamAnalytics             {Get-AzureRmStreamAnalyticsFunction, Get-AzureRmStreamAnalyticsDefaultFunctionDefinition, New-AzureRmStreamAnalyticsFunction, Remove-AzureRmStreamAnalyticsFunction...}                                
Script     4.0.1      AzureRM.Tags                        {Remove-AzureRmTag, Get-AzureRmTag, New-AzureRmTag}                                                                                                                                                    
Script     4.0.3      AzureRM.TrafficManager              {Disable-AzureRmTrafficManagerEndpoint, Enable-AzureRmTrafficManagerEndpoint, Set-AzureRmTrafficManagerEndpoint, Get-AzureRmTrafficManagerEndpoint...}                                                 
Script     4.0.2      AzureRM.UsageAggregates             Get-UsageAggregates                                                                                                                                                                                    
Script     4.2.2      AzureRM.Websites                    {Get-AzureRmAppServicePlan, Set-AzureRmAppServicePlan, New-AzureRmAppServicePlan, Remove-AzureRmAppServicePlan...}                                                                                     
Script     1.0.1      Microsoft.PowerShell.Operation.V... {Get-OperationValidation, Invoke-OperationValidation}                                                                                                                                                  
Binary     1.0.0.1    PackageManagement                   {Find-Package, Get-Package, Get-PackageProvider, Get-PackageSource...}                                                                                                                                 
Script     3.4.0      Pester                              {Describe, Context, It, Should...}                                                                                                                                                                     
Script     1.0.0.1    PowerShellGet                       {Install-Module, Find-Module, Save-Module, Update-Module...}                                                                                                                                           
Script     1.2        PSReadline                          {Get-PSReadlineKeyHandler, Set-PSReadlineKeyHandler, Remove-PSReadlineKeyHandler, Get-PSReadlineOption...}                                                                                             

    Directory: C:\WINDOWS\system32\WindowsPowerShell\v1.0\Modules

ModuleType Version    Name                                ExportedCommands                                                                                                                                                                                       
---------- -------    ----                                ----------------                                                                                                                                                                                       
Manifest   1.0.0.0    AppBackgroundTask                   {Disable-AppBackgroundTaskDiagnosticLog, Enable-AppBackgroundTaskDiagnosticLog, Set-AppBackgroundTaskResourcePolicy, Unregister-AppBackgroundTask...}                                                  
Manifest   2.0.0.0    AppLocker                           {Get-AppLockerFileInformation, Get-AppLockerPolicy, New-AppLockerPolicy, Set-AppLockerPolicy...}                                                                                                       
Manifest   1.0.0.0    AppvClient                          {Add-AppvClientConnectionGroup, Add-AppvClientPackage, Add-AppvPublishingServer, Disable-Appv...}                                                                                                      
Manifest   2.0.0.0    Appx                                {Add-AppxPackage, Get-AppxPackage, Get-AppxPackageManifest, Remove-AppxPackage...}                                                                                                                     
Script     1.0.0.0    AssignedAccess                      {Clear-AssignedAccess, Get-AssignedAccess, Set-AssignedAccess}                                                                                                                                         
Manifest   1.0.0.0    BitLocker                           {Unlock-BitLocker, Suspend-BitLocker, Resume-BitLocker, Remove-BitLockerKeyProtector...}                                                                                                               
Manifest   2.0.0.0    BitsTransfer                        {Add-BitsFile, Complete-BitsTransfer, Get-BitsTransfer, Remove-BitsTransfer...}                                                                                                                        
Manifest   1.0.0.0    BranchCache                         {Add-BCDataCacheExtension, Clear-BCCache, Disable-BC, Disable-BCDowngrading...}                                                                                                                        
Manifest   1.0.0.0    CimCmdlets                          {Get-CimAssociatedInstance, Get-CimClass, Get-CimInstance, Get-CimSession...}                                                                                                                          
Manifest   1.0        ConfigCI                            {Get-SystemDriver, New-CIPolicyRule, New-CIPolicy, Get-CIPolicy...}                                                                                                                                    
Manifest   1.0        Defender                            {Get-MpPreference, Set-MpPreference, Add-MpPreference, Remove-MpPreference...}                                                                                                                         
Manifest   1.0.0.0    DeliveryOptimization                {Get-DeliveryOptimizationStatus, Get-DeliveryOptimizationPerfSnap, Get-DeliveryOptimizationLog, Get-DODownloadMode...}                                                                                 
Manifest   1.0.0.0    DirectAccessClientComponents        {Disable-DAManualEntryPointSelection, Enable-DAManualEntryPointSelection, Get-DAClientExperienceConfiguration, Get-DAEntryPointTableItem...}                                                           
Script     3.0        Dism                                {Add-AppxProvisionedPackage, Add-WindowsDriver, Add-WindowsCapability, Add-WindowsImage...}                                                                                                            
Manifest   1.0.0.0    DnsClient                           {Resolve-DnsName, Clear-DnsClientCache, Get-DnsClient, Get-DnsClientCache...}                                                                                                                          
Manifest   1.0.0.0    EventTracingManagement              {Start-EtwTraceSession, New-EtwTraceSession, Get-EtwTraceSession, Update-EtwTraceSession...}                                                                                                           
Manifest   2.0.0.0    International                       {Get-WinDefaultInputMethodOverride, Set-WinDefaultInputMethodOverride, Get-WinHomeLocation, Set-WinHomeLocation...}                                                                                    
Manifest   1.0.0.0    iSCSI                               {Get-IscsiTargetPortal, New-IscsiTargetPortal, Remove-IscsiTargetPortal, Update-IscsiTargetPortal...}                                                                                                  
Script     1.0.0.0    ISE                                 {New-IseSnippet, Import-IseSnippet, Get-IseSnippet}                                                                                                                                                    
Manifest   1.0.0.0    Kds                                 {Add-KdsRootKey, Get-KdsRootKey, Test-KdsRootKey, Set-KdsConfiguration...}                                                                                                                             
Manifest   1.0.1.0    Microsoft.PowerShell.Archive        {Compress-Archive, Expand-Archive}                                                                                                                                                                     
Manifest   3.0.0.0    Microsoft.PowerShell.Diagnostics    {Get-WinEvent, Get-Counter, Import-Counter, Export-Counter...}                                                                                                                                         
Manifest   3.0.0.0    Microsoft.PowerShell.Host           {Start-Transcript, Stop-Transcript}                                                                                                                                                                    
Manifest   1.0.0.0    Microsoft.PowerShell.LocalAccounts  {Add-LocalGroupMember, Disable-LocalUser, Enable-LocalUser, Get-LocalGroup...}                                                                                                                         
Manifest   3.1.0.0    Microsoft.PowerShell.Management     {Add-Content, Clear-Content, Clear-ItemProperty, Join-Path...}                                                                                                                                         
Script     1.0        Microsoft.PowerShell.ODataUtils     Export-ODataEndpointProxy                                                                                                                                                                              
Manifest   3.0.0.0    Microsoft.PowerShell.Security       {Get-Acl, Set-Acl, Get-PfxCertificate, Get-Credential...}                                                                                                                                              
Manifest   3.1.0.0    Microsoft.PowerShell.Utility        {Format-List, Format-Custom, Format-Table, Format-Wide...}                                                                                                                                             
Manifest   3.0.0.0    Microsoft.WSMan.Management          {Disable-WSManCredSSP, Enable-WSManCredSSP, Get-WSManCredSSP, Set-WSManQuickConfig...}                                                                                                                 
Manifest   1.0        MMAgent                             {Disable-MMAgent, Enable-MMAgent, Set-MMAgent, Get-MMAgent...}                                                                                                                                         
Manifest   1.0.0.0    MsDtc                               {New-DtcDiagnosticTransaction, Complete-DtcDiagnosticTransaction, Join-DtcDiagnosticResourceManager, Receive-DtcDiagnosticTransaction...}                                                              
Manifest   2.0.0.0    NetAdapter                          {Disable-NetAdapter, Disable-NetAdapterBinding, Disable-NetAdapterChecksumOffload, Disable-NetAdapterEncapsulatedPacketTaskOffload...}                                                                 
Manifest   1.0.0.0    NetConnection                       {Get-NetConnectionProfile, Set-NetConnectionProfile}                                                                                                                                                   
Manifest   1.0.0.0    NetEventPacketCapture               {New-NetEventSession, Remove-NetEventSession, Get-NetEventSession, Set-NetEventSession...}                                                                                                             
Manifest   2.0.0.0    NetLbfo                             {Add-NetLbfoTeamMember, Add-NetLbfoTeamNic, Get-NetLbfoTeam, Get-NetLbfoTeamMember...}                                                                                                                 
Manifest   1.0.0.0    NetNat                              {Get-NetNat, Get-NetNatExternalAddress, Get-NetNatStaticMapping, Get-NetNatSession...}                                                                                                                 
Manifest   2.0.0.0    NetQos                              {Get-NetQosPolicy, Set-NetQosPolicy, Remove-NetQosPolicy, New-NetQosPolicy}                                                                                                                            
Manifest   2.0.0.0    NetSecurity                         {Get-DAPolicyChange, New-NetIPsecAuthProposal, New-NetIPsecMainModeCryptoProposal, New-NetIPsecQuickModeCryptoProposal...}                                                                             
Manifest   1.0.0.0    NetSwitchTeam                       {New-NetSwitchTeam, Remove-NetSwitchTeam, Get-NetSwitchTeam, Rename-NetSwitchTeam...}                                                                                                                  
Manifest   1.0.0.0    NetTCPIP                            {Get-NetIPAddress, Get-NetIPInterface, Get-NetIPv4Protocol, Get-NetIPv6Protocol...}                                                                                                                    
Manifest   1.0.0.0    NetworkConnectivityStatus           {Get-DAConnectionStatus, Get-NCSIPolicyConfiguration, Reset-NCSIPolicyConfiguration, Set-NCSIPolicyConfiguration}                                                                                      
Manifest   1.0.0.0    NetworkSwitchManager                {Disable-NetworkSwitchEthernetPort, Enable-NetworkSwitchEthernetPort, Get-NetworkSwitchEthernetPort, Remove-NetworkSwitchEthernetPortIPAddress...}                                                     
Manifest   1.0.0.0    NetworkTransition                   {Add-NetIPHttpsCertBinding, Disable-NetDnsTransitionConfiguration, Disable-NetIPHttpsProfile, Disable-NetNatTransitionConfiguration...}                                                                
Manifest   1.0.0.0    PcsvDevice                          {Get-PcsvDevice, Start-PcsvDevice, Stop-PcsvDevice, Restart-PcsvDevice...}                                                                                                                             
Binary     1.0.0.0    PersistentMemory                    {Get-PmemDisk, Get-PmemPhysicalDevice, Get-PmemUnusedRegion, New-PmemDisk...}                                                                                                                          
Manifest   1.0.0.0    PKI                                 {Add-CertificateEnrollmentPolicyServer, Export-Certificate, Export-PfxCertificate, Get-CertificateAutoEnrollmentPolicy...}                                                                             
Manifest   1.0.0.0    PnpDevice                           {Get-PnpDevice, Get-PnpDeviceProperty, Enable-PnpDevice, Disable-PnpDevice}                                                                                                                            
Manifest   1.1        PrintManagement                     {Add-Printer, Add-PrinterDriver, Add-PrinterPort, Get-PrintConfiguration...}                                                                                                                           
Binary     1.0.11     ProcessMitigations                  {Get-ProcessMitigation, Set-ProcessMitigation, ConvertTo-ProcessMitigationPolicy}                                                                                                                      
Script     3.0        Provisioning                        {Install-ProvisioningPackage, Export-ProvisioningPackage, Install-TrustedProvisioningCertificate, Export-Trace...}                                                                                     
Manifest   1.1        PSDesiredStateConfiguration         {Set-DscLocalConfigurationManager, Start-DscConfiguration, Test-DscConfiguration, Publish-DscConfiguration...}                                                                                         
Script     1.0.0.0    PSDiagnostics                       {Disable-PSTrace, Disable-PSWSManCombinedTrace, Disable-WSManTrace, Enable-PSTrace...}                                                                                                                 
Binary     1.1.0.0    PSScheduledJob                      {New-JobTrigger, Add-JobTrigger, Remove-JobTrigger, Get-JobTrigger...}                                                                                                                                 
Manifest   2.0.0.0    PSWorkflow                          {New-PSWorkflowExecutionOption, New-PSWorkflowSession, nwsn}                                                                                                                                           
Manifest   1.0.0.0    PSWorkflowUtility                   Invoke-AsWorkflow                                                                                                                                                                                      
Manifest   1.0.0.0    ScheduledTasks                      {Get-ScheduledTask, Set-ScheduledTask, Register-ScheduledTask, Unregister-ScheduledTask...}                                                                                                            
Manifest   2.0.0.0    SecureBoot                          {Confirm-SecureBootUEFI, Set-SecureBootUEFI, Get-SecureBootUEFI, Format-SecureBootUEFI...}                                                                                                             
Manifest   2.0.0.0    SmbShare                            {Get-SmbShare, Remove-SmbShare, Set-SmbShare, Block-SmbShareAccess...}                                                                                                                                 
Manifest   2.0.0.0    SmbWitness                          {Get-SmbWitnessClient, Move-SmbWitnessClient, gsmbw, msmbw...}                                                                                                                                         
Manifest   1.0.0.0    StartLayout                         {Export-StartLayout, Import-StartLayout, Export-StartLayoutEdgeAssets, Get-StartApps}                                                                                                                  
Manifest   2.0.0.0    Storage                             {Add-InitiatorIdToMaskingSet, Add-PartitionAccessPath, Add-PhysicalDisk, Add-StorageFaultDomain...}                                                                                                    
Manifest   2.0.0.0    TLS                                 {New-TlsSessionTicketKey, Enable-TlsSessionTicketKey, Disable-TlsSessionTicketKey, Export-TlsSessionTicketKey...}                                                                                      
Manifest   1.0.0.0    TroubleshootingPack                 {Get-TroubleshootingPack, Invoke-TroubleshootingPack}                                                                                                                                                  
Manifest   2.0.0.0    TrustedPlatformModule               {Get-Tpm, Initialize-Tpm, Clear-Tpm, Unblock-Tpm...}                                                                                                                                                   
Binary     2.1.639.0  UEV                                 {Clear-UevConfiguration, Clear-UevAppxPackage, Restore-UevBackup, Set-UevTemplateProfile...}                                                                                                           
Manifest   2.0.0.0    VpnClient                           {Add-VpnConnection, Set-VpnConnection, Remove-VpnConnection, Get-VpnConnection...}                                                                                                                     
Manifest   1.0.0.0    Wdac                                {Get-OdbcDriver, Set-OdbcDriver, Get-OdbcDsn, Add-OdbcDsn...}                                                                                                                                          
Manifest   1.0.0.0    WindowsDeveloperLicense             {Get-WindowsDeveloperLicense, Unregister-WindowsDeveloperLicense, Show-WindowsDeveloperLicenseRegistration}                                                                                            
Script     1.0        WindowsErrorReporting               {Enable-WindowsErrorReporting, Disable-WindowsErrorReporting, Get-WindowsErrorReporting}                                                                                                               
Manifest   1.0.0.0    WindowsSearch                       {Get-WindowsSearchSetting, Set-WindowsSearchSetting}                                                                                                                                                   
Manifest   1.0.0.0    WindowsUpdate                       Get-WindowsUpdateLog                                                                                                                                                                                   
Manifest   1.0.0.2    WindowsUpdateProvider               {Get-WUAVersion, Get-WULastInstallationDate, Get-WULastScanSuccessDate, Get-WUIsPendingReboot...}

Debug output

Error output

Could not load resource. Please ensure no mistakes in the JSON and that referenced resources exist
cormacpayne commented 5 years ago

@hvermis @zhangyd2015 would you mind taking a look at this issue?

hvermis commented 5 years ago

This is because SqlServerSource is not released in Powershell SDK yet.

hvermis commented 5 years ago

As a workaround you can use New-AzResource instead ADF resource creation cmdlets, or ARM template deployment cmdlets. Here's a sample for New-AzResource: $dataset = Get-AzResource -ResourceType "Microsoft.DataFactory/factories/datasets" -ResourceGroupName ans-data-factory -Name "ans-prod-v2/DelimitedText1" -ApiVersion "2018-06-01"

$newsonPropertiesStringFromFile = ConvertTo-Json $dataset.Properties

New-AzResource -ResourceType "Microsoft.DataFactory/factories/datasets" -ResourceGroupName ans-data-factory -Name "ans-prod-v2/DelimitedText1" -ApiVersion "2018-06-01" -Properties (ConvertFrom-Json $newsonPropertiesStringFromFile)

Doc links: https://docs.microsoft.com/en-us/powershell/module/az.resources/new-azresource?view=azps-2.2.0 https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-template-deploy

Limp-monkey commented 5 years ago

thanks, I think I got it to work for the pipelines using those datasets too. Any chance for a timeline on SqlServerSource?

Also, this brings me to another question - we're trying to use powershell to automate as much provisioning as possible, but seems we're always running into issues with versions, like:

  1. SqlServerSource is not released in Powershell SDK yet.
  2. AzureRM doesn't handle delete activity in ADF
  3. AZ module is not on the GA tasks in DevOps Pipelines

Do you have any recommendation, which API to use as the most safe one? Is it the New-AzResource?

Get-Content $pipeline_list | ForEach-Object {
$pipeline = $_

$pipeline_json = Get-Content "$path_json\pipeline\$pipeline.json" | ConvertFrom-Json

New-AzResource -ResourceType "Microsoft.DataFactory/factories/pipelines"  -ResourceGroupName $env:PIPELINE_TARGET_RESOURCE_GROUP_NAME  -Name "$env:PIPELINE_TARGET_DATA_FACTORY_NAME/$pipeline" -ApiVersion "2018-06-01" -Properties  $pipeline_json -IsFullObject -Force
}
hvermis commented 5 years ago

We are working on releasing the new types, they will either be in upcoming release or next one. AzureRM does not get updates anymore, only Az modules do. You should contact DevOps team to get you VM-s with Az modules. There is always a chance that Powershell module will be behind in features from our backend as we first develop the backend. The generic New-AzResource does not have that problem, so you could use that, however our recommended way to do CI/CD is using git integration: https://docs.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment#use-custom-parameters-with-the-resource-manager-template

Limp-monkey commented 5 years ago

thanks again! We've looked into that, but we gave up due to this unsupported feature: "You can't publish individual resources, because data factory entities depend on each other. " On development we sometimes need to be able to schedule trigger a pipeline, and the only way to do that is when a pipeline has been published. However, that doesn't means it' ready to be deployed, so we can't have it exported via git integration. Hence we needed a way of publishing particular pipelines - the best way we found was DevOps Pipelines reading from GIT master branch. Anyhow, thank you for the support and tips!