Implement Azure Data Factory pipeline to copy data from SQL to Delta Lake - Githubissues

Azure-Samples / modern-data-warehouse-dataops

DataOps for Microsoft Data Platform technologies. https://aka.ms/dataops-repo

MIT License

588 stars 459 forks source link

Implement Azure Data Factory pipeline to copy data from SQL to Delta Lake #269

Closed NT-D closed 3 years ago

NT-D commented 3 years ago

Purpose

We need data pipeline to copy data from SQL DB to Delta Lake.

It needs support to save versioned data as Delta lake
It is parameterized pipeline, so that an end user can update pipeline without modifying source code (ex. can update SQL Table name as parameter)

Other Info: I only save ARM template in this PR, because

We need ARM template for end users to deploy solution to their infra.
We have an one source of truth (If we have source and ARM, how do make sure the JSON is equivalent to the ARM Template in the SCM ?)
ADF can regenerate source code from ARM template.

Does this introduce a breaking change?

[ ] Yes
[x] No

Author pre-publish checklist:

[ ] Added test to prove my fix is effective or new feature works
[x] No PII in logs
[ ] Made corresponding changes to the documentation