search

mrpaulandrew / procfwk

A cross tenant metadata driven processing framework for Azure Data Factory and Azure Synapse Analytics achieved by coupling orchestration pipelines with a SQL database and a set of Azure Functions.
https://mrpaulandrew.com/category/azure/data-factory/adf-procfwk/
Other
185 stars 116 forks source link

Update failed pipeline error handling and restarts across execution stages. #24

Closed mrpaulandrew closed 4 years ago

mrpaulandrew commented 4 years ago
  1. Add new table [PipelineDependencies] allowing 1:many relationships for Pipelines.

| PipelineId | DependantPipelineId |

  1. Add a new Property called 'FailureHandling'. This property could have 3 possible values....

None = The failure status is updated in the Current Execution table and the framework carries on regardless.

Simple = The failure status is updated in the Current Execution table. Downstream execution stages are marked as blocked and processing is stopped after the stage where the failure occurred. This is basically what currently happens.

Informed = The failure status is updated in the Current Execution table. Downstream pipelines are marked as blocked (using the DependantPipelineId attribute in the PipelineDependencies table). However, processing is allowed to continue. When the next execution stage starts only pipelines NOT blocked will be prepared and ran.

mrpaulandrew commented 4 years ago

Related to external request: https://github.com/mrpaulandrew/ADF.procfwk/projects/2#card-40162174