search

mrpaulandrew / procfwk

A cross tenant metadata driven processing framework for Azure Data Factory and Azure Synapse Analytics achieved by coupling orchestration pipelines with a SQL database and a set of Azure Functions.
https://mrpaulandrew.com/category/azure/data-factory/adf-procfwk/
Other
185 stars 116 forks source link

Creation of new execution in overiderestart mode fails when a previous run didn't finish successfully #44

Closed pfiadeiro closed 4 years ago

pfiadeiro commented 4 years ago

Describe the bug When the OverideRestart property is set to 1 and there's a current execution where some pipelines have a status <> 'Success', the creation of a new execution will fail. This happens since the ExecutionWrapper stored procedure calls the UpdateExecutionLog stored procedure and when the OverideRestart property is set to 1, the process will only work if all pipelines in the Current Execution table have a status of 'Success' (in theory, they wouldn't even be there and moved to the final log table once framework completes), otherwise it will always throw an error.

Affected services Which resource within the processing framework does this affect?

To Reproduce Steps to reproduce the behavior:

  1. Change a pipeline parameter to force a specific pipeline to fail. For example, in a wait activity set the value 'abc'
  2. Trigger a new run in Data Factory and wait for it to finish with a failed pipeline
  3. Set the property OverideRestart to 1
  4. Trigger another new run in Data Factory
  5. Framework will fail when running the Execution Wrapper activity.

Expected behaviour The creation of a new run should be possible (by setting OverideProperty = 1) even when the previous run didn't finish successfully

mrpaulandrew commented 4 years ago

Nice catch, thanks Pedro, I'll fix this in the next minor release.

mrpaulandrew commented 4 years ago

Updated procs now available in working branch v1.8.4...

https://github.com/mrpaulandrew/ADF.procfwk/blob/v1.8.4/MetadataDB/procfwk/Stored%20Procedures/UpdateExecutionLog.sql

https://github.com/mrpaulandrew/ADF.procfwk/blob/v1.8.4/MetadataDB/procfwk/Stored%20Procedures/ExecutionWrapper.sql