search

mrpaulandrew / procfwk

A cross tenant metadata driven processing framework for Azure Data Factory and Azure Synapse Analytics achieved by coupling orchestration pipelines with a SQL database and a set of Azure Functions.
https://mrpaulandrew.com/category/azure/data-factory/adf-procfwk/
Other
185 stars 116 forks source link

ADF Pipeline Parameter value seems to be cached. When I run the pipeline its automatically taking the Previous run values even if I enable=1 new PipelineID and make Enabled=0 for the previous PipelineID #111

Closed seemaht closed 3 years ago

seemaht commented 3 years ago

Describe the error ADF Pipeline Parameter value seems to be cached. When I run the pipeline its automatically taking the Previous run values even if I enable=1 new PipelineID and make Enabled=0 for the previous PipelineID

Error message An output of the error message presented.

Affected services Which resource within the processing framework does this affect?

To Reproduce Steps to reproduce the behavior:

  1. Go to '...'
  2. Click on '....'
  3. Scroll down to '....'
  4. See error

Screenshots If applicable, add screenshots to help explain your problem.

pfiadeiro commented 3 years ago

@seemaht: This comes down to @mrpaulandrew deciding whether he wants to explore this issue further or not but you really provide no information that can allow people to help you. You don't produce any steps to reproduce the behavior of the issue you're facing nor any screenshots.

With that said, is it possible that you have a failed execution and whenever you start a new one, it's restarting the old one and thus using the old parameter values?

seemaht commented 3 years ago

@pfiadeiro, Yes I have a failed execution and its restarting the old one. I have to manually create the current execution everytime it fails.

pfiadeiro commented 3 years ago

@seemaht: You can control that behavior through the OverideRestart option in the Properties table. Take a look at the documentation so you understand how it works: https://mrpaulandrew.github.io/procfwk/frameworkrestart

It seems like you just want to execute a new run without taking into account any previous failures so just set the flag to 1.

seemaht commented 3 years ago

@pfiadeiro This is really helpful. Thank You!