Fabric E2E Sample (Milestone 1)

[devlace]

Milestone 1: end of Jan. Github Milestone Feature Branch: feat/e2e-fabric-dataops-sample

Fabric functionality considerations: Features needs to be in Public Preview by Ignite to be considered in this milestone.

E2E Sample "Business Problem"

• Business Problem: visual parking availability in the city. "Parking analytics".
  • Utilize existing Parking Sensor to keep as is similar to existing Databricks sample
• Persona: Contoso City Council
• Batch only. Land into storage, then shortcut main point of ingestion.
• Decide on the name of the sample. Propose: parking_sensor_fabric

User Stories:

• [x] #788
• [x] #789
• [ ] #790

IaC

• TF scripts utilizing the Fabric TF Provider
• Overall Deployment script ./deploy.sh
  • Dev, Test, Prod environment deployment
  • Seed data, bootstrapping
• Follow from IaC for Everything recommendations
• Document any known issues such as non-automatable tasks such as lack of SP support for some APIs

User Stories:

• [x] #759
• [ ] #798
• [ ] #799

Data Pipelines

• Medallion style data pipeline

User Stories:

• [ ] #837
• [ ] #836
• [x] #842
• [ ] #843
• [ ] #801
• [ ] #848
• [ ] #849
• [ ] #850
• [ ] #851

CI/CD

• PR validation (basic linting, unit tests)
• CI pipeline (basic build - whl package generation)
• Release pipeline
• Document any known issues such as non-automatable tasks.

Stretch:

• PR validation + Deployment to ephemeral pipeline (middle ground?)
  • This step takes substantial amount of time.
  • Requires a spike (can we go pure ephemeral or some are permanent; )

Known issues:

• Forking workspaces doesn't sync Lakehouse. Similar to shortcuts
• Data manage between dev workspaces

User Stories:

• [ ] #802
• [ ] #803
• [ ] #804
• [ ] #805
• [ ] #806
• [ ] #833

Automated Testing

• Unit Testing (Python packages, Data Pipelines)
• Integration Testing / Functional E2E testing (test/pre-prod environment - this is a permanent environment. Release is serial.)

User Stories:

• [ ] #807
• [ ] #808
• [ ] #809
• [ ] #810

Stretch:

• Notebook testing (??)

Observability

• Data Quality
• Open Telemetry standard
• Sink: AppInsights

Python Package modernization

For Package development, update to latest / best practices. Align with Databricks + ADF update team on python package modernization.

• [ ] #811
• [ ] #812

Dev Tooling, Environment

Common Dev environment, CICD, Developer onboarding guide for the sample.

• [ ] #796
• [ ] #797
• [x] #794
• [x] #795

Consideration of Fabric functionality to showcases

• TBD: Functions, Python notebook

VNext Milestone ideas:

• Add streaming scenario
• Data Governance (Purview)