search

Azure-Samples / modern-data-warehouse-dataops

DataOps for Microsoft Data Platform technologies. https://aka.ms/dataops-repo
MIT License
588 stars 459 forks source link

Parking Sensor Synapse Overall Milestone 1 #407

Closed devlace closed 2 years ago

devlace commented 3 years ago

Description

This is parent issue to track the overall Milestone 1 of the Parking Sensor Synapse sample.

Legend: image

Overall Data Pipeline logical architecture

image

CICD Process diagram

image

Child Issues:

  1. Storage account structure, upload seed data – Not yet on backlog
  2. Spark Notebook: Cleanse & Standardize – Issue: #394
  3. Spark Notebook: Transform – Issue: #394
  4. Synapse Data Pipeline – Issue: #395
  5. SQL Serverless – Issue: #392, #393
  6. SQL Dedicated Pool – Not yet on backlog
  7. PowerBI dashboard – Not yet on backlog
  8. AppInsights – Not yet on backlog
  9. LogAnalytics – Issue: #389 BUT completed within original Parking Sensor structure.
  10. KeyVault – Not yet on backlog
  11. CI: PR validation (run python unit tests, build SQL DACPAC) and Build Pipeline to publish artifacts – Issue: #398
  12. CD: Integration Tests – Issue: #399
  13. CD: Upload python package – Issue: #390
  14. CD: Deploy DACPAC – Issue: #397
  15. CD: Deploy ARM template (Synapse workspace) – Issue: #396
  16. Full Sample Deploy Script – Not yet in backlog
devlace commented 3 years ago

The reason why certain items are not yet in backlog is because the backlog is currently built with the assumption that we are building the Synapse sample as part of the existing Parking Sensor sample as opposed to creating a completely new E2E sample. These items are already in the existing Parking Sensor sample, and does not require any further changes between Synapse version.

However, if we opt for a completely new E2E sample, these will need to be recreated (copied-over?) from the Parking Sensor sample.

See this PR with ADR: #405