search

Azure-Player / azure.datafactory.devops

Azure DevOps Deployment Tasks for Azure Data Factory objects
MIT License
39 stars 21 forks source link

Build ADF Code Started Giving Errors #113

Closed SravanDodda closed 1 year ago

SravanDodda commented 1 year ago

Hello, We have been using Build ADF Code for quite some time and was working great until about 4 weeks ago. It started giving error - "The property 'globalParameters' cannot be found on this object. Verify that the property exists.". We tried to resolve it modifying ARM template files to remove reference to globalParameters but to no avail

Interestingly, our ADF pipelines are NOT using any global parameters. Not sure why this error and why now

Here is the log:

2022-12-01T16:25:50.7037626Z === Validating other rules ... 2022-12-01T16:26:01.3577512Z ##[debug]Leaving D:\a_tasks\BuildADFTask_8a00f62d-c46d-4019-9e11-c05d88821db8\1.14.1173\BuildADF.ps1. 2022-12-01T16:26:01.4093112Z ##[debug]Caught exception from task script. 2022-12-01T16:26:01.4159581Z ##[debug]Error record: 2022-12-01T16:26:01.4754218Z ##[debug]Test-AdfCode : The property 'globalParameters' cannot be found on this object. Verify that the property exists. 2022-12-01T16:26:01.4768873Z ##[debug]At D:\a_tasks\BuildADFTask_8a00f62d-c46d-4019-9e11-c05d88821db8\1.14.1173\BuildADF.ps1:46 char:14 2022-12-01T16:26:01.4782374Z ##[debug]+ ... $r = Test-AdfCode -RootFolder "$RootFolder" -ConfigPath $Confi ... 2022-12-01T16:26:01.4796068Z ##[debug]+ ~~~~~~~~~~~~~ 2022-12-01T16:26:01.4809370Z ##[debug] + CategoryInfo : NotSpecified: (:) [Test-AdfCode], PropertyNotFoundException 2022-12-01T16:26:01.4823061Z ##[debug] + FullyQualifiedErrorId : PropertyNotFoundStrict,Test-AdfCode 2022-12-01T16:26:01.4835915Z ##[debug] 2022-12-01T16:26:01.4857109Z ##[debug]Script stack trace: 2022-12-01T16:26:01.4894256Z ##[debug]at Test-AdfCode, D:\a_tasks\BuildADFTask_8a00f62d-c46d-4019-9e11-c05d88821db8\1.14.1173\ps_modules\azure.datafactory.tools\public\Test-AdfCode.ps1: line 96 2022-12-01T16:26:01.4907588Z ##[debug]at , D:\a_tasks\BuildADFTask_8a00f62d-c46d-4019-9e11-c05d88821db8\1.14.1173\BuildADF.ps1: line 46 2022-12-01T16:26:01.4920666Z ##[debug]at , : line 1 2022-12-01T16:26:01.4933671Z ##[debug]at , : line 22 2022-12-01T16:26:01.4946693Z ##[debug]at , : line 18 2022-12-01T16:26:01.4959914Z ##[debug]at , : line 1 2022-12-01T16:26:01.4979792Z ##[debug]Exception: 2022-12-01T16:26:01.5083785Z ##[debug]System.Management.Automation.PropertyNotFoundException: The property 'globalParameters' cannot be found on this object. Verify that the property exists. 2022-12-01T16:26:01.5097213Z ##[debug] at System.Management.Automation.ExceptionHandlingOps.CheckActionPreference(FunctionContext funcContext, Exception exception) 2022-12-01T16:26:01.5110801Z ##[debug] at System.Management.Automation.Interpreter.ActionCallInstruction2.Run(InterpretedFrame frame) 2022-12-01T16:26:01.5123815Z ##[debug] at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame) 2022-12-01T16:26:01.5137240Z ##[debug] at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame) 2022-12-01T16:26:01.5149921Z ##[debug] at System.Management.Automation.Interpreter.Interpreter.Run(InterpretedFrame frame) 2022-12-01T16:26:01.5162298Z ##[debug] at System.Management.Automation.Interpreter.LightLambda.RunVoid1[T0](T0 arg0) 2022-12-01T16:26:01.5175288Z ##[debug] at System.Management.Automation.PSScriptCmdlet.RunClause(Action1 clause, Object dollarUnderbar, Object inputToProcess) 2022-12-01T16:26:01.5188388Z ##[debug] at System.Management.Automation.PSScriptCmdlet.DoEndProcessing() 2022-12-01T16:26:01.5201562Z ##[debug] at System.Management.Automation.CommandProcessorBase.Complete() 2022-12-01T16:26:01.5263290Z ##[error]The property 'globalParameters' cannot be found on this object. Verify that the property exists. 2022-12-01T16:26:01.5265312Z ##[debug]Processed: ##vso[task.logissue type=error]The property 'globalParameters' cannot be found on this object. Verify that the property exists. 2022-12-01T16:26:01.5284130Z ##[debug]Processed: ##vso[task.complete result=Failed] 2022-12-01T16:26:01.5801971Z ##[section]Finishing: Validate ADF files

Pipeline design is attached

Could you help? CI Pipeline Screenshot 2022-12-01 132744

NowinskiK commented 1 year ago

I test the task in my test environment and everything looks good: image Can you provide ADF code in order to replicate the issue?

SravanDodda commented 1 year ago

I test the task in my test environment and everything looks good: image Can you provide ADF code in order to replicate the issue?

Azure ADF-1.zip

Thanks for your help

NowinskiK commented 1 year ago

Which one should I test? image

SravanDodda commented 1 year ago

Which one should I test? image

ADF-BIGroupFacts01-Dev

SravanDodda commented 1 year ago

I am sorry, it should be "Azure ADF". When I publish ADF, ARM templates are getting updated in "ADF-BIGroupFacts01-Dev" and pipelines are getting updated in "Azure ADF". "ADF-BIGroupFacts01-Dev" is our development ADF name. Git is configured in ADF for collaboration and deploy branch to "Dev" branch in Azure DevOps

Does it make sense?

SravanDodda commented 1 year ago

One other note, in the ADF-BIGroupFacts01-Dev repo, there were no folders other than "globalParameters" and one more (don't remember). I have copied the folders (dataflow, pipeline and so on) from "Azure ADF" to ADF-BIGroupFacts01-Dev repo to see if that works

NowinskiK commented 1 year ago

For both folders the same issue raises: image

SravanDodda commented 1 year ago

Yes, this duplication thing is new. Even before the duplication thing, "globalParameters" issue was happening.

Previous Error

NowinskiK commented 1 year ago

The duplication is because you have the same name for a pipeline as well as dataflow. But that does not cause the error.

SravanDodda commented 1 year ago

Correct. I just realized that while going through Data Flows and Pipelines

So, you are not able to reproduce the 'globalParameter" error?

NowinskiK commented 1 year ago

I reproduced the error thanks to the attached code. It will be released in PS module first (ver.0.110), and in DevOps extension afterwards (tonight).

NowinskiK commented 1 year ago

Fixed in ver.1.15.1239, just released.

SravanDodda commented 1 year ago

Fixed in ver.1.15.1239, just released.

Thank you. Appreciate your help

How do I install the new version by replacing the existing one? Could you provide some documentation of how to do that?

NowinskiK commented 1 year ago

It should be installed automatically.

SravanDodda commented 1 year ago

It should be installed automatically.

Thank you. Now I get this and failing. Could you help?

NewError

NowinskiK commented 1 year ago

You can not point ARM Template parameter file as a config. This task is not for deploying from ARM Template but from adf code directly. Read more in documentation and watch video on my YouTube channel: https://sqlplayer.net/adftools

SravanDodda commented 1 year ago

You can not point ARM Template parameter file as a config. This task is not for deploying from ARM Template but from adf code directly. Read more in documentation and watch video on my YouTube channel: https://sqlplayer.net/adftools

Got it. Thank you so much for your great help and taking care of this in no time. Really appreciate it and am a big follower on YouTube as well