dingmeng-xue
commented
4 years ago Involve Data Factory team to look into this issue.
Closed datasharkNL closed 4 years ago
dingmeng-xue
commented
4 years ago Involve Data Factory team to look into this issue.
ghost
commented
4 years ago Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @lmy269, @shawnxzq
datasharkNL
commented
4 years ago And the same happened with Set-AzureRmDataFactoryV2Pipeline:
Creating pipelines from C:\Users\jeffr\Desktop\ADF Deployment\pipelines\<pipelineName>.json
30-DeltaLakeToDataHub
Set-AzureRmDataFactoryV2Pipeline : 'Value' cannot be null.
At C:\Users\jeffr\Desktop\ADF Deployment\Create-ADF v2.ps1:112 char:5
+ Set-AzureRmDataFactoryV2Pipeline -DefinitionFile $file.FullName ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : CloseError: (:) [Set-AzureRmDataFactoryV2Pipeline], ValidationException
+ FullyQualifiedErrorId : Microsoft.Azure.Commands.DataFactoryV2.SetAzureDataFactoryPipelineCommandThe part in my script that creates the pipelines:
# Create General Pipelines from templates
$pipelineFiles = Get-ChildItem $InputFolderPath -Recurse -Include *.json -Exclude ls_*,ds_* -ErrorAction Stop
$pipelineFiles
foreach ($file in $pipelineFiles)
{
Write-Output "Creating pipelines from $($file.FullName)"
$fileName = $file.Name -replace ".{5}$"
$fileName
Set-AzureRmDataFactoryV2Pipeline -DefinitionFile $file.FullName -Name $fileName -DataFactoryName $dataFactoryName ` -ResourceGroupName $resourceGroupName -Force
#| Format-List
}And the pipelines JSON (generated by Azure Data Factory):
(Replaced all the names with <name> etc)
{
"name": "<pipelineName>",
"properties": {
"activities": [
{
"name": "<Name>",
"type": "DatabricksNotebook",
"dependsOn": [
{
"activity": "<StoredProcedure>",
"dependencyConditions": [
"Succeeded"
]
}
],
"policy": {
"timeout": "7.00:00:00",
"retry": 1,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"notebookPath": "/<dbr_path>/<notebookName>",
"baseParameters": {
"ComponentRunId": {
"value": "@pipeline().RunId",
"type": "Expression"
},
"ObjectCode": {
"value": "@pipeline().parameters.ObjectCode",
"type": "Expression"
},
"MasterRunId": {
"value": "@pipeline().parameters.MasterRunId",
"type": "Expression"
},
"SourceFilename": {
"value": "@pipeline().parameters.SourceFilename",
"type": "Expression"
}
}
},
"linkedServiceName": {
"referenceName": "<linkedServiceName>",
"type": "LinkedServiceReference"
}
},
{
"name": "<SPName>",
"type": "SqlServerStoredProcedure",
"dependsOn": [],
"policy": {
"timeout": "7.00:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"storedProcedureName": "[<Schema>].[<SPName>]",
"storedProcedureParameters": {
"ComponentName": {
"value": {
"value": "@pipeline().Pipeline",
"type": "Expression"
},
"type": "String"
},
"ComponentRunId": {
"value": {
"value": "@pipeline().RunId",
"type": "Expression"
},
"type": "String"
},
"ComponentType": {
"value": null,
"type": "String"
},
"ObjectCode": {
"value": {
"value": "@pipeline().parameters.ObjectCode",
"type": "Expression"
},
"type": "String"
},
"MasterRunId": {
"value": {
"value": "@pipeline().parameters.MasterRunId",
"type": "Expression"
},
"type": "String"
},
"LogStatus": {
"value": {
"value": "DeltaLoadStarted",
"type": "Expression"
},
"type": "String"
}
}
},
"linkedServiceName": {
"referenceName": "<linkedServiceName>",
"type": "LinkedServiceReference"
}
},
{
"name": "<SPName>",
"type": "SqlServerStoredProcedure",
"dependsOn": [
{
"activity": "<name>",
"dependencyConditions": [
"Failed"
]
}
],
"policy": {
"timeout": "7.00:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"storedProcedureName": "[<schemaname>].[<tablename>]",
"storedProcedureParameters": {
"ComponentRunId": {
"value": {
"value": "@pipeline().RunId",
"type": "Expression"
},
"type": "String"
},
"FileLocation": {
"value": null,
"type": "String"
},
"SourceFileName": {
"value": null,
"type": "String"
},
"InsertCount": {
"value": null,
"type": "Int32"
},
"LogMessage": {
"value": {
"value": "@string(activity('<activityname>').output.runPageUrl)",
"type": "Expression"
},
"type": "String"
},
"LogStatus": {
"value": "LoadFailed",
"type": "String"
},
"RowCount": {
"value": null,
"type": "Int32"
},
"DestinationFileName": {
"value": null,
"type": "String"
}
}
},
"linkedServiceName": {
"referenceName": "ls_SQL_datahub",
"type": "LinkedServiceReference"
}
}
],
"parameters": {
"ObjectCode": {
"type": "string"
},
"MasterRunId": {
"type": "string"
},
"SourceFilename": {
"type": "string"
}
},
"folder": {
"name": "<pipelinename>"
},
"annotations": []
}
}
shawnxzq
commented
4 years ago Please use the command from the new Az module instead as below, it should work, thanks!
shawnxzq
commented
4 years ago @datasharkNL You Az module is still at version 1.6, please upgrade the latest version. https://www.powershellgallery.com/packages/Az/3.5.0, then it should work.
BTW, the AzureRm module is the old one and the Az module is to replace it and always updated with new features.
kunalwadhwa
commented
4 years ago The issue still exists. I tried using Az PowerShell module 3.5.0 and Az Datafctory 1.7.0
Praxkv1
commented
1 year ago This is still an issue. Can you please help to fix this
When running Set-AzDataFactoryV2Dataset on a JSON file auto-generated by the Data Factory Portal, I receive the error Set-AzDataFactoryV2Dataset : 'TableName' cannot be null.
Description
Hi, when I use
Set-AzureRmDataFactoryV2Datasetfor setting a dataset with a variable "tableName" I still get this error: (In this issue they said it should be fixed: https://github.com/Azure/azure-powershell/issues/9261)Set-AzureRmDataFactoryV2Dataset : 'TableName' cannot be null.Still have a defaultvalue, but getting the error:
Steps to reproduce
Environment data
Module versions
Error output
Dataset properties copied from Data Factory (changed only the name for this issue on GitHub):