Add Support for Azure Synapse

mrpaulandrew / procfwk

A cross tenant metadata driven processing framework for Azure Data Factory and Azure Synapse Analytics achieved by coupling orchestration pipelines with a SQL database and a set of Azure Functions.

https://mrpaulandrew.com/category/azure/data-factory/adf-procfwk/

Other

185 stars 116 forks source link

Add Support for Azure Synapse #21

Closed mrpaulandrew closed 3 years ago

mrpaulandrew commented 4 years ago

Following initial investigations: https://mrpaulandrew.com/2020/06/03/adf-procfwk-and-azure-synapse-orchestrate-preview-and-limitations/

Add support for Azure Synapse.

Include support for Worker and Framework pipelines in either Data Factory or Synapse.

mrpaulandrew commented 4 years ago

Severity Code Description Project File Line Suppression State Error CS1061

'SynapseManagementClient' does not contain a definition for 'PipelineRuns' and no accessible extension method 'PipelineRuns' accepting a first argument of type 'SynapseManagementClient' could be found (are you missing a using directive or an assembly reference?)

Functions ...\GitHub\ADF.procfwk\Functions\CheckPipelineStatus.cs 110 Active

mrpaulandrew commented 4 years ago

To Do:

Finish Execute Pipeline function once unblocked.
Update Get Error Details function to include orchestrator name and type.
Update Check Status function to include orchestrator name and type.
Rename 'DataFactorys' table to 'Orchestrators' and replace through metadata database.
Add orchestrator type attribute with check constraint.
CREATE SYNONYM [procfwk].[DataFactorys] FOR [procfwk].[Orchestrators];

mrpaulandrew commented 4 years ago

Synapse still not ready. Need pipeline feature parity between Data Factory and Synapse Orchestrate.

mrpaulandrew commented 3 years ago

@NowinskiK now Synapse has Git support, do you want to try deploying the framework pipelines there?

NowinskiK commented 3 years ago

I saw that yesterday. I didn't try but I believe it will not work. The reason is simple: the definition of resources is different (i.e. resource path). I will create the clone of my ADFTools and change the code accordingly and I will let you know. Then we could back to the discussion we started recently via email.

mrpaulandrew commented 3 years ago

Thanks for the Function app refactoring @richardswinbank . Using these new foundations I've just created the bare bones Synapse service.

mrpaulandrew commented 3 years ago

Development of Azure Functions App now complete using the Synapse namespace:

Azure.Analytics.Synapse.Artifacts

With version 1.0.0-beta.1 of the NuGet package.