search

Azure / azure-powershell

Microsoft Azure PowerShell
Other
4.21k stars 3.81k forks source link

Set-AzDataFactoryV2Dataset - fails to set file start/end modified data properties #12332

Closed simondmorias closed 4 years ago

simondmorias commented 4 years ago

Description

Executing Set-AzDataFactoryV2Dataset with a dataset file for ADLS with start and end data on the file filters fails to apply to modified dates to the dataset. Running in debug it fails to pass them to the rest api.

When including:

"modifiedDatetimeStart": {
  "value": "@dataset().StartDate",
   "type": "Expression"
},
"modifiedDatetimeEnd": {
  "value": "@dataset().EndDate",
  "type": "Expression"
}

In the payload the values are not set in the factory. Validate through the UI. Other typeProperties such as the folderPath seem to pass correctly. In my case the ADLS is a Gen1.

Steps to reproduce

Create a file for the dataset on disk with this contents:

{
    "name": "DS_BACKUP_SOURCE",
    "properties": {
        "linkedServiceName": {
            "referenceName": "LS_ADLS",
            "type": "LinkedServiceReference"
        },
        "parameters": {
            "FilePath": {
                "type": "string",
                "defaultValue": "/xx/Sourced/"
            },
            "StartDate": {
                "type": "string",
                "defaultValue": "2020-01-31"
            },
            "EndDate": {
                "type": "string",
                "defaultValue": "2020-02-06"
            }
        },
        "annotations": [],
        "type": "AzureDataLakeStoreFile",
        "typeProperties": {
            "fileName": "*.*",
            "folderPath": {
                "value": "@concat('/xx/', dataset().FilePath)",
                "type": "Expression"
            },
            "modifiedDatetimeStart": {
                "value": "@dataset().StartDate",
                "type": "Expression"
            },
            "modifiedDatetimeEnd": {
                "value": "@dataset().EndDate",
                "type": "Expression"
            }
        }
    },
    "type": "Microsoft.DataFactory/factories/datasets"
}

Try to deploy the dataset:

 Set-AzDataFactoryV2DataSet `
            -ResourceGroupName $ResourceGroupName `
            -DataFactoryName $DataFactoryName `
            -Name "DS_BACKUP_SOURCE" `
            -DefinitionFile "DS_BACKUP_SOURCE.json"

Environment data

Name                           Value
----                           -----
PSVersion                      7.0.2
PSEdition                      Core
GitCommitId                    7.0.2
OS                             Linux 4.19.84-microsoft-standard #1 SMP Wed Nov 13 11:44:37 UTC 2019
Platform                       Unix
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Module versions

    Directory: /root/.local/share/powershell/Modules

ModuleType Version    PreRelease Name                                PSEdition ExportedCommands
---------- -------    ---------- ----                                --------- ----------------
Script     4.3.0                 Az                                  Core,Desk 
Script     4.2.0                 Az                                  Core,Desk 
Script     1.9.0                 Az.Accounts                         Core,Desk {Disable-AzDataCollection, Disable-AzContextAutosave, Enable-AzDataCollectio…
Script     1.8.1                 Az.Accounts                         Core,Desk {Disable-AzDataCollection, Disable-AzContextAutosave, Enable-AzDataCollectio…
Script     1.1.1                 Az.Advisor                          Core,Desk {Get-AzAdvisorRecommendation, Enable-AzAdvisorRecommendation, Disable-AzAdvi…
Script     1.1.2                 Az.Aks                              Core,Desk {Get-AzAks, New-AzAks, Remove-AzAks, Import-AzAksCredential…}
Script     1.1.1                 Az.Aks                              Core,Desk {Get-AzAks, New-AzAks, Remove-AzAks, Import-AzAksCredential…}
Script     1.1.3                 Az.AnalysisServices                 Core,Desk {Resume-AzAnalysisServicesServer, Suspend-AzAnalysisServicesServer, Get-AzAn…
Script     2.0.1                 Az.ApiManagement                    Core,Desk {Add-AzApiManagementApiToProduct, Add-AzApiManagementProductToGroup, Add-AzA…
Script     1.1.0                 Az.ApplicationInsights              Core,Desk {Get-AzApplicationInsights, New-AzApplicationInsights, Remove-AzApplicationI…
Script     1.3.6                 Az.Automation                       Core,Desk {Get-AzAutomationHybridWorkerGroup, Remove-AzAutomationHybridWorkerGroup, Ge…
Script     3.1.0                 Az.Batch                            Core,Desk {Remove-AzBatchAccount, Get-AzBatchAccount, Get-AzBatchAccountKey, New-AzBat…
Script     3.0.0                 Az.Batch                            Core,Desk {Remove-AzBatchAccount, Get-AzBatchAccount, Get-AzBatchAccountKey, New-AzBat…
Script     1.0.3                 Az.Billing                          Core,Desk {Get-AzBillingInvoice, Get-AzBillingPeriod, Get-AzEnrollmentAccount, Get-AzC…
Script     1.4.3                 Az.Cdn                              Core,Desk {Get-AzCdnProfile, Get-AzCdnProfileSsoUrl, New-AzCdnProfile, Remove-AzCdnPro…
Script     1.5.0                 Az.CognitiveServices                Core,Desk {Get-AzCognitiveServicesAccount, Get-AzCognitiveServicesAccountKey, Get-AzCo…
Script     1.4.0                 Az.CognitiveServices                Core,Desk {Get-AzCognitiveServicesAccount, Get-AzCognitiveServicesAccountKey, Get-AzCo…
Script     4.2.0                 Az.Compute                          Core,Desk {Remove-AzAvailabilitySet, Get-AzAvailabilitySet, New-AzAvailabilitySet, Upd…
Script     4.1.0                 Az.Compute                          Core,Desk {Remove-AzAvailabilitySet, Get-AzAvailabilitySet, New-AzAvailabilitySet, Upd…
Script     1.0.3                 Az.ContainerInstance                Core,Desk {New-AzContainerGroup, Get-AzContainerGroup, Remove-AzContainerGroup, Get-Az…
Script     1.1.1                 Az.ContainerRegistry                Core,Desk {New-AzContainerRegistry, Get-AzContainerRegistry, Update-AzContainerRegistr…
Script     1.1.0                 Az.DataBoxEdge                      Core,Desk {Get-AzDataBoxEdgeJob, Get-AzDataBoxEdgeDevice, Invoke-AzDataBoxEdgeDevice, …
Script     1.8.2                 Az.DataFactory                      Core,Desk {Set-AzDataFactoryV2, Update-AzDataFactoryV2, Get-AzDataFactoryV2, Remove-Az…
Script     1.8.1                 Az.DataFactory                      Core,Desk {Set-AzDataFactoryV2, Update-AzDataFactoryV2, Get-AzDataFactoryV2, Remove-Az…
Script     1.0.2                 Az.DataLakeAnalytics                Core,Desk {Get-AzDataLakeAnalyticsDataSource, New-AzDataLakeAnalyticsCatalogCredential…
Script     1.2.8                 Az.DataLakeStore                    Core,Desk {Get-AzDataLakeStoreTrustedIdProvider, Remove-AzDataLakeStoreTrustedIdProvid…
Script     1.0.0                 Az.DataShare                        Core,Desk {New-AzDataShareAccount, Get-AzDataShareAccount, Remove-AzDataShareAccount, …
Script     1.1.0                 Az.DeploymentManager                Core,Desk {Get-AzDeploymentManagerArtifactSource, New-AzDeploymentManagerArtifactSourc…
Script     1.0.0                 Az.DesktopVirtualization            Core,Desk {Disconnect-AzWvdUserSession, Get-AzWvdApplication, Get-AzWvdApplicationGrou…
Script     1.0.2                 Az.DevTestLabs                      Core,Desk {Get-AzDtlAllowedVMSizesPolicy, Get-AzDtlAutoShutdownPolicy, Get-AzDtlAutoSt…
Script     1.1.2                 Az.Dns                              Core,Desk {Get-AzDnsRecordSet, New-AzDnsRecordConfig, Remove-AzDnsRecordSet, Set-AzDns…
Script     1.2.3                 Az.EventGrid                        Core,Desk {New-AzEventGridTopic, Get-AzEventGridTopic, Set-AzEventGridTopic, New-AzEve…
Script     1.5.0                 Az.EventHub                         Core,Desk {New-AzEventHubNamespace, Get-AzEventHubNamespace, Set-AzEventHubNamespace, …
Script     1.4.3                 Az.EventHub                         Core,Desk {New-AzEventHubNamespace, Get-AzEventHubNamespace, Set-AzEventHubNamespace, …
Script     1.5.0                 Az.FrontDoor                        Core,Desk {New-AzFrontDoor, Get-AzFrontDoor, Set-AzFrontDoor, Remove-AzFrontDoor…}
Script     1.0.1                 Az.Functions                        Core,Desk {Get-AzFunctionApp, Get-AzFunctionAppAvailableLocation, Get-AzFunctionAppPla…
Script     1.0.0                 Az.Functions                        Core,Desk {Get-AzFunctionApp, Get-AzFunctionAppAvailableLocation, Get-AzFunctionAppPla…
Script     3.3.0                 Az.HDInsight                        Core,Desk {Get-AzHDInsightJob, New-AzHDInsightSqoopJobDefinition, Wait-AzHDInsightJob,…
Script     3.2.0                 Az.HDInsight                        Core,Desk {Get-AzHDInsightJob, New-AzHDInsightSqoopJobDefinition, Wait-AzHDInsightJob,…
Script     1.1.0                 Az.HealthcareApis                   Core,Desk {New-AzHealthcareApisService, Remove-AzHealthcareApisService, Set-AzHealthca…
Script     1.0.2                 Az.HealthcareApis                   Core,Desk {New-AzHealthcareApisService, Remove-AzHealthcareApisService, Set-AzHealthca…
Script     2.5.0                 Az.IotHub                           Core,Desk {Add-AzIotHubKey, Get-AzIotHubEventHubConsumerGroup, Get-AzIotHubConnectionS…
Script     2.0.0                 Az.KeyVault                         Core,Desk {Add-AzKeyVaultCertificate, Update-AzKeyVaultCertificate, Stop-AzKeyVaultCer…
Script     1.3.2                 Az.LogicApp                         Core,Desk {Get-AzIntegrationAccountAgreement, Get-AzIntegrationAccountAssembly, Get-Az…
Script     1.1.3                 Az.MachineLearning                  Core,Desk {Move-AzMlCommitmentAssociation, Get-AzMlCommitmentAssociation, Get-AzMlComm…
Script     1.0.0                 Az.Maintenance                      Core,Desk {Get-AzApplyUpdate, Get-AzConfigurationAssignment, Get-AzMaintenanceConfigur…
Script     1.0.2                 Az.ManagedServices                  Core,Desk {Get-AzManagedServicesAssignment, New-AzManagedServicesAssignment, Remove-Az…
Script     1.0.2                 Az.MarketplaceOrdering              Core,Desk {Get-AzMarketplaceTerms, Set-AzMarketplaceTerms}
Script     1.1.1                 Az.Media                            Core,Desk {Sync-AzMediaServiceStorageKey, Set-AzMediaServiceKey, Get-AzMediaServiceKey…
Script     2.0.1                 Az.Monitor                          Core,Desk {Get-AzMetricDefinition, Get-AzMetric, Remove-AzLogProfile, Get-AzLogProfile…
Script     2.0.0                 Az.Monitor                          Core,Desk {Get-AzMetricDefinition, Get-AzMetric, Remove-AzLogProfile, Get-AzLogProfile…
Script     3.1.0                 Az.Network                          Core,Desk {Add-AzApplicationGatewayAuthenticationCertificate, Get-AzApplicationGateway…
Script     3.0.0                 Az.Network                          Core,Desk {Add-AzApplicationGatewayAuthenticationCertificate, Get-AzApplicationGateway…
Script     1.1.1                 Az.NotificationHubs                 Core,Desk {Get-AzNotificationHub, Get-AzNotificationHubAuthorizationRule, Get-AzNotifi…
Script     2.2.0                 Az.OperationalInsights              Core,Desk {New-AzOperationalInsightsAzureActivityLogDataSource, New-AzOperationalInsig…
Script     2.1.0                 Az.OperationalInsights              Core,Desk {New-AzOperationalInsightsAzureActivityLogDataSource, New-AzOperationalInsig…
Script     1.3.1                 Az.PolicyInsights                   Core,Desk {Get-AzPolicyEvent, Get-AzPolicyState, Get-AzPolicyStateSummary, Get-AzPolic…
Script     1.1.2                 Az.PowerBIEmbedded                  Core,Desk {Remove-AzPowerBIWorkspaceCollection, Get-AzPowerBIWorkspaceCollection, Get-…
Script     1.0.3                 Az.PrivateDns                       Core,Desk {Get-AzPrivateDnsZone, Remove-AzPrivateDnsZone, Set-AzPrivateDnsZone, New-Az…
Script     2.10.0                Az.RecoveryServices                 Core,Desk {Get-AzRecoveryServicesBackupProperty, Get-AzRecoveryServicesVault, Get-AzRe…
Script     2.9.1                 Az.RecoveryServices                 Core,Desk {Get-AzRecoveryServicesBackupProperty, Get-AzRecoveryServicesVault, Get-AzRe…
Script     1.2.1                 Az.RedisCache                       Core,Desk {Remove-AzRedisCachePatchSchedule, New-AzRedisCacheScheduleEntry, Get-AzRedi…
Script     1.0.3                 Az.Relay                            Core,Desk {New-AzRelayNamespace, Get-AzRelayNamespace, Set-AzRelayNamespace, Remove-Az…
Script     2.2.0                 Az.Resources                        Core,Desk {Get-AzProviderOperation, Remove-AzRoleAssignment, Get-AzRoleAssignment, New…
Script     2.1.0                 Az.Resources                        Core,Desk {Get-AzProviderOperation, Remove-AzRoleAssignment, Get-AzRoleAssignment, New…
Script     1.4.1                 Az.ServiceBus                       Core,Desk {New-AzServiceBusNamespace, Get-AzServiceBusNamespace, Set-AzServiceBusNames…
Script     2.1.0                 Az.ServiceFabric                    Core,Desk {Add-AzServiceFabricClientCertificate, Add-AzServiceFabricClusterCertificate…
Script     1.1.1                 Az.SignalR                          Core,Desk {New-AzSignalR, Get-AzSignalR, Get-AzSignalRKey, New-AzSignalRKey…}
Script     2.8.0                 Az.Sql                              Core,Desk {Get-AzSqlDatabaseTransparentDataEncryption, Get-AzSqlDatabaseTransparentDat…
Script     2.7.0                 Az.Sql                              Core,Desk {Get-AzSqlDatabaseTransparentDataEncryption, Get-AzSqlDatabaseTransparentDat…
Script     1.1.0                 Az.SqlVirtualMachine                Core,Desk {New-AzSqlVM, Get-AzSqlVM, Update-AzSqlVM, Remove-AzSqlVM…}
Script     2.2.0                 Az.Storage                          Core,Desk {Get-AzStorageAccount, Get-AzStorageAccountKey, New-AzStorageAccount, New-Az…
Script     2.1.0                 Az.Storage                          Core,Desk {Get-AzStorageAccount, Get-AzStorageAccountKey, New-AzStorageAccount, New-Az…
Script     1.2.3                 Az.StorageSync                      Core,Desk {Invoke-AzStorageSyncCompatibilityCheck, New-AzStorageSyncService, Get-AzSto…
Script     1.0.1                 Az.StreamAnalytics                  Core,Desk {Get-AzStreamAnalyticsFunction, Get-AzStreamAnalyticsDefaultFunctionDefiniti…
Script     1.0.0                 Az.Support                          Core,Desk {Get-AzSupportService, Get-AzSupportProblemClassification, Get-AzSupportTick…
Script     1.0.4                 Az.TrafficManager                   Core,Desk {Add-AzTrafficManagerCustomHeaderToEndpoint, Remove-AzTrafficManagerCustomHe…
Script     1.10.0                Az.Websites                         Core,Desk {Get-AzAppServicePlan, Set-AzAppServicePlan, New-AzAppServicePlan, Remove-Az…
Script     1.9.0                 Az.Websites                         Core,Desk {Get-AzAppServicePlan, Set-AzAppServicePlan, New-AzAppServicePlan, Remove-Az…
Script     2.1.2829              azure.databricks.cicd.tools         Desk      {Add-DatabricksSecretScope, Export-DatabricksFolder, Import-DatabricksFolder…
Script     1.0.113               dbatools                            Desk      {Select-DbaObject, Set-DbatoolsConfig, Start-DbaMigration, Copy-DbaDatabase…}
Script     4.10.1                Pester                              Desk      {Describe, Context, It, Should…}

    Directory: /opt/microsoft/powershell/7/Modules

ModuleType Version    PreRelease Name                                PSEdition ExportedCommands
---------- -------    ---------- ----                                --------- ----------------
Manifest   1.2.5                 Microsoft.PowerShell.Archive        Desk      {Compress-Archive, Expand-Archive}
Manifest   7.0.0.0               Microsoft.PowerShell.Host           Core      {Start-Transcript, Stop-Transcript}
Manifest   7.0.0.0               Microsoft.PowerShell.Management     Core      {Add-Content, Clear-Content, Clear-ItemProperty, Join-Path…}
Manifest   7.0.0.0               Microsoft.PowerShell.Security       Core      {Get-Credential, Get-ExecutionPolicy, Set-ExecutionPolicy, ConvertFrom-Secur…
Manifest   7.0.0.0               Microsoft.PowerShell.Utility        Core      {Export-Alias, Get-Alias, Import-Alias, New-Alias…}
Script     1.4.6                 PackageManagement                   Desk      {Find-Package, Get-Package, Get-PackageProvider, Get-PackageSource…}
Script     2.2.4                 PowerShellGet                       Desk      {Find-Command, Find-DSCResource, Find-Module, Find-RoleCapability…}
Script     2.0.5                 PSDesiredStateConfiguration         Core      {Configuration, New-DscChecksum, Get-DscResource, Invoke-DscResource}
Script     2.0.0                 PSReadLine                          Desk      {Get-PSReadLineKeyHandler, Set-PSReadLineKeyHandler, Remove-PSReadLineKeyHan…
Binary     2.0.3                 ThreadJob                           Desk      Start-ThreadJob

    Directory: /root/.vscode-server/extensions/ms-vscode.powershell-2020.6.0/modules

ModuleType Version    PreRelease Name                                PSEdition ExportedCommands
---------- -------    ---------- ----                                --------- ----------------
Script     1.1.3                 Plaster                             Desk      {Invoke-Plaster, New-PlasterManifest, Get-PlasterTemplate, Test-PlasterManif…
Binary     2.1.0                 PowerShellEditorServices            Core,Desk Start-EditorServices
Binary     0.2.0                 PowerShellEditorServices.VSCode     Desk      {New-VSCodeHtmlContentView, Show-VSCodeHtmlContentView, Close-VSCodeHtmlCont…
Script     2.0.2                 PSReadLine                          Desk      {Get-PSReadLineKeyHandler, Set-PSReadLineKeyHandler, Remove-PSReadLineKeyHan…
Script     1.19.0                PSScriptAnalyzer                    Desk      {Get-ScriptAnalyzerRule, Invoke-ScriptAnalyzer, Invoke-Formatter}

Debug output

============================ HTTP REQUEST ============================

HTTP Method:
PUT

Absolute Uri:
https://management.azure.com/subscriptions/xxx/resourceGroups/xx-analytics-nonprod-rg-xx/providers/Microsoft.DataFactory/factories/xx-eun-analytics-nonprod-adf-ingest-Simon01/datasets/DS_BACKUP_SOURCE?api-version=2018-06-01

Headers:
x-ms-client-request-id        : 6e322c1e-4c33-448a-92e9-ae366a0b8425
Accept-Language               : en-US

Body:
{
  "properties": {
    "type": "AzureDataLakeStoreFile",
    "typeProperties": {
      "folderPath": "@concat('/Simon/', dataset().FilePath)",
      "fileName": "*.*"
    },
    "linkedServiceName": {
      "referenceName": "LS_ADLS",
      "type": "LinkedServiceReference"
    },
    "parameters": {
      "FilePath": {
        "type": "string",
        "defaultValue": "/xxx/Sourced/"
      },
      "StartDate": {
        "type": "string",
        "defaultValue": "2020-01-31"
      },
      "EndDate": {
        "type": "string",
        "defaultValue": "2020-02-06"
      }
    },
    "annotations": []
  }
}

============================ HTTP RESPONSE ============================

Status Code:
OK

Headers:
Cache-Control                 : no-cache
Pragma                        : no-cache
Strict-Transport-Security     : max-age=31536000; includeSubDomains
X-Content-Type-Options        : nosniff
Server                        : Microsoft-IIS/10.0
X-Powered-By                  : ASP.NET
x-ms-ratelimit-remaining-subscription-writes: 1189
x-ms-request-id               : 9c263f69-2a18-4811-9461-379f99821434
x-ms-correlation-request-id   : 9c263f69-2a18-4811-9461-379f99821434
x-ms-routing-request-id       : UKWEST:20200706T101202Z:9c263f69-2a18-4811-9461-379f99821434
Date                          : Mon, 06 Jul 2020 10:12:02 GMT

Body:
{
  "id": "/subscriptions/xxx/resourceGroups/xx-analytics-nonprod-rg-xx/providers/Microsoft.DataFactory/factories/xx-eun-analytics-nonprod-adf-ingest-Simon01/datasets/DS_BACKUP_SOURCE",
  "name": "DS_BACKUP_SOURCE",
  "type": "Microsoft.DataFactory/factories/datasets",
  "properties": {
    "type": "AzureDataLakeStoreFile",
    "typeProperties": {
      "folderPath": "@concat('/Simon/', dataset().FilePath)",
      "fileName": "*.*"
    },
    "linkedServiceName": {
      "referenceName": "LS_ADLS",
      "type": "LinkedServiceReference"
    },
    "parameters": {
      "FilePath": {
        "type": "string",
        "defaultValue": "/xx/Sourced/"
      },
      "StartDate": {
        "type": "string",
        "defaultValue": "2020-01-31"
      },
      "EndDate": {
        "type": "string",
        "defaultValue": "2020-02-06"
      }
    },
    "annotations": []
  },
  "etag": "4a004283-0000-0c00-0000-5f02f8f20000"
}

DatasetName       : DS_BACKUP_SOURCE
ResourceGroupName : xx-analytics-nonprod-rg-orion
DataFactoryName   : xx-eun-analytics-nonprod-adf-ingest-Simon01
Structure         : 
Properties        : Microsoft.Azure.Management.DataFactory.Models.AzureDataLakeStoreDataset

AzureQoSEvent: CommandName - Set-AzDataFactoryV2Dataset; IsSuccess - True; Duration - 00:00:00.4821876;
Finish sending metric.
10:12:03 - SetAzureDataFactoryDatasetCommand end processing

Error output

No error returned

ghost commented 4 years ago

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @shawnxzq, @lmy269.

dingmeng-xue commented 4 years ago

Data Factory team, please help to look into this question.

kprzysowa commented 4 years ago

I have the same issue. +1

lmy269 commented 4 years ago

@shawnxzq

simondmorias commented 4 years ago

In the reproduce steps I incorrectly used Set-AzDataFactoryV2DataFlow instead of Set-AzDataFactoryV2DataSet - I have corrected this.

shawnxzq commented 4 years ago

@simondmorias This is known issue, please upgrade you az module to the latest one, and use the new dataset for your case, it should be binary dataset. With this binary dataset, you should be able to set modifiedDatetimeStart/End in the copy activity.