Azure-Player / azure.datafactory.tools

Tools for deploying Data Factory (v2) in Microsoft Azure
https://azureplayer.net/adftools
MIT License
212 stars 69 forks source link

Issue during "add" action for global parameters - config file (json) #340

Closed MartaUch closed 1 year ago

MartaUch commented 1 year ago

Hello,

I'm currently investigating usage of config file to update global parameters. We have three different environments, and for each we have different, customized values for global parameters. Currently we are not tracking global parameter change with json files, but I decided to use your feature regarding config file.

My plan would be to create three config files for each environments and maintain them e.g. extended by developer if new global parameter needs to be added to the environments. I have two issues that I'm currently struggling with:

  1. Here is my config file which adds new parameter: { "deep-dev-adf-we": [ { "name": "$.properties.globalParameters.envName.value", "value": "POC", "action": "add" } ] } After running deployment task I'm receiving the following error: image Same issue when I use csv image I thought that maybe "type" of global parameter is missing, but it didn't change anything when I added key "type":"string" into json file. In addition, when I try to update global parameter that already exists everything works fine.

  2. I'm getting error if I use action "add" on already existing parameter. { "deep-dev-adf-we": [ { "name": "$.properties.globalParameters.environment.value", "value": "POC", "action": "add" } ] } In general I understand that I could use "update", but if for example developer wants to add new parameter and config file is adjusted to do so, then during the next deployment code will fail with the following error: image The problem I see here that I would really like to avoid constant update of the config file to make it work. Is it possible to force the deployment of global parameter if it already exists and action is "add". I believe the similar situation would be with "delete" action - once it's deleted (config files updated), then the next deployment "deletion" of not existing object (deleted in the previous deployment) maybe could be handled as well to not stop the usage of config file.

Thank you for your help here in advance! Kind Regards Marta

NowinskiK commented 1 year ago

Hi Marta. I checked the issue and it works (I tested with CSV file only):

type,name,path,value
factory,BigFactorySample2,"+$.properties.globalParameters.NewGlobalParam.value",2023
factory,BigFactorySample2,"+$.properties.globalParameters.envName.value",POC

image Updated "factory" file: image

Adding new Global Param should work when it doesn't exist yet. Did I miss anything?

MartaUch commented 1 year ago

Hello Kamil, I noticed that I missed quotation marks in my csv, but when I added them it didn't change anything: before: type,name,path,value factory,deep-dev-adf-we,+$.properties.globalParameters.envName.value,POC after: type,name,path,value factory,deep-dev-adf-we,"+$.properties.globalParameters.envName.value",POC

In general I see the same information like you in your test. It is recognized that new parameter must be added and it's listed: image image But the error occurs after, when parameters suppose to be deployed. I see an issue here related to the lack of "type" for the global parameter. I did a test and didn't use config file for that, but factory json file itself: image When I tried to deploy it, it showed me exactly the same type of error: image From my perspective we are updating the factory json file correctly with name and value, but we are missing "type" which is required. It leads to en error when we really try to deploy. Could you please try to do the same test like you have done already, but with additional real deployment?

Thank you! Marta

NowinskiK commented 1 year ago

Thanks for clarification, Marta. I was able to reproduce the problem and I found the reason. Root cause: Each global parameter has two properties: type and value:

    "envName": {
      "Type": "string",
      "Value": "POC"
    }

When you update the existing one - it's fine to provide just one of them: value. However, when you create new global parameter, you must provide BOTH properties, otherwise it will try to deploy incomplete parameter, like this:

    "envName": {
      "Type": null,
      "Value": "POC"
    }

Therefore, to elude this problem you must define both properties in config file, for example:

factory,BigFactorySample2,"+$.properties.globalParameters.envName.value",POC
factory,BigFactorySample2,"+$.properties.globalParameters.envName.type",string

I hope this resolve the problem.

MartaUch commented 1 year ago

Hi Kamil,

Yes, indeed, thank you :) I checked two approaches: CSV: type,name,path,value factory,deep-dev-adf-we,"+$.properties.globalParameters.envName.value",POC factory,deep-dev-adf-we,"+$.properties.globalParameters.envName.type",string JSON: { "deep-dev-adf-we": [ { "name": "$.properties.globalParameters.envName_2.value", "value": "POC", "action": "add" }, { "name": "$.properties.globalParameters.envName_2.type", "value": "string", "action": "add" } ] } And it works. Is it possible to put these two properties: value and type, into one value? I was trying to use another approach: type,name,path,value factory,deep-dev-adf-we,"+$.properties.globalParameters.envName","{'type': 'string','value': 'POC'}" but it doesn't work so far. There is an issue to convert the value: image Is it possible to make it work this way? I was using this as an example: image That kind of format in csv or json would be more convenient to use.

Best Regard Marta