Databricks+ADF E2E Sample (Milestone 1)

[ydaponte]

Milestone 1: mid-Jan.

E2E Sample "Technical Problem"

• Technical Problem: Fix the ability for the sample to be deployed e2e and modernize sample with latest features from Databricks and ADF
• Persona: Data Engineers and Developers

IaC

• Fixing sample devcontainer
• Generate data that is independent from the public REST API (it can be re-used by the other E2E sample) and allowing batch and streaming modes to ingest the data
• Fixing bicep scripts that are failing with the latest versions and newer schemas (Synapse SQL Dedicated pool)
• Fixing broken main ./deploy.sh script when configuring the Databricks workspace for dev, stg and prod
• Upgrade to the latest Databricks CLI as the previous version is no longer working for certain parts of the code
• Incorporate the deployment of a Unity Catalog
• Fixing Key Vault Soft-Delete capability in the clean-up and deployment scripts
• Act on issues identified on the ISE Checklist (Other fundamentals)
• Document all changes, including architecture schemas when applicable

CI/CD

• Incorporate Asset Bundles to the CICD Databricks process
• Act on issues identified on the ISE Checklist (Other fundamentals)
• Document all changes

Security

• Use Confidential General purpose compute SKU on the Databricks cluster (if available on the region/subscription)
• Start and act on the ISE Checklist on the Security pillar
• Document all changes

Observability

• Act on issues identified on the ISE Checklist (Observability)
• Document all changes, including architecture schemas when applicable

Testing

• Incorporate the usage of the Data Factory Testing Framework on the ADF pipelines
• Document all changes, including architecture schemas when applicable

Python Package modernization

For Package development, update to latest / best practices. Align with Fabric update team on python package modernization.

VNext Milestone ideas:

• Adding Autoloader to ingestion
• Adding Industry packages to the sample

• Add a dependency map of the versions across the sample

Ensure you've read the Contributing Guide (Microsoft Internal only.)

IaC

• [x] Fixing main issues: #817
  • [x] Fix Dev container: #644
  • [x] Fix Dev container: #646
  • [x] Fix bicep scripts: #778
  • [x] Fix main deployment script (ADB workspace): #630
  • [x] Upgrade Databricks CLI: #763
• [ ] Fix clean-up script: #819
  • [ ] Key-vault soft delete: #596
  • [ ] Delete based on deployment_id: #821
• [ ] Fix pipeline problems
  • [ ] Fix ADF pipelines - temporary solution before #764 is ready: #852
  • [ ] Fix release pipelines: #864
• [ ] Remove REST API dependency: #764
• [ ] Incorporate Unity Catalog: #765

CICD

• [ ] Incorporate Asset Bundles: #766
• [ ] ISE Fundamentals Checklist: #841

Security

• [ ] ISE Security Checklist: #826

Observability

• [ ] ISE Observability Checklist: #839

Testing

• [ ] Incorporate DFTF on ADF pipelines: #832

Next Steps

• [ ] TBD