Microsoft Data Engineering four-day instructor-led training deployment

This repo contains manual and automated deployment steps for lab environments used by the Microsoft Data Engineering four-day ILT training curriculum.

• Microsoft Data Engineering four-day instructor-led training deployment
  • Module directory
  • Lab VM for students
  • Modules that can share the same Synapse workspace
  • Modules that can share the same Azure Databricks workspace

Module directory

• Module 1 - Explore compute and storage options for data engineering workloads
  • Lab 01 setup instructions
  • Lab 02 setup instructions - step 1 of 2
  • Lab 02 setup instructions - step 2 of 2
• Module 2 - Design and Implement the serving layer
  • Lab setup instructions
• Module 3 - Data engineering considerations for source files
  • No lab setup
• Module 4 - Run interactive queries using serverless SQL pools
  • Lab setup instructions
• Module 5 - Explore, transform, and load data into the Data Warehouse using Apache Spark
  • Lab setup instructions
• Module 6 - Data exploration and transformation in Azure Databricks
  • Lab setup instructions
• Module 7 - Ingest and load data into the Data Warehouse
  • Lab setup instructions
• Module 8 - Transform data with Azure Data Factory or Azure Synapse Pipelines
  • Lab setup instructions
• Module 9 - Integrate data from notebooks with Azure Data Factory or Azure Synapse Pipelines
  • Lab setup instructions
• Module 10 - Optimize query performance with dedicated SQL pools in Azure Synapse
  • Lab setup instructions
• Module 11 - Analyze and optimize Data Warehouse storage
  • Lab setup instructions
• Module 12 - Support Hybrid Transactional Analytical Processing (HTAP) with Azure Synapse Link
  • Lab setup instructions
• Module 13 - End-to-end security with Azure Synapse Analytics
  • Lab setup instructions
• Module 14 - Real-time stream processing with Stream Analytics
  • Lab setup instructions
• Module 15 - Create a stream processing solution with Event Hubs and Azure Databricks
  • Lab setup instructions
• Module 16 - Build reports using Power BI integration with Azure Synapse Analytics
  • Lab setup instructions
• Module 17 - Perform integrated Machine Learning processes in Azure Synapse Analytics
  • Lab setup instructions - step 1 of 2
  • Lab setup instructions - step 2 of 2

Lab VM for students

There are three modules that require a lab VM. All others only require a web browser for the students. The lab VM can be used for these modules (Modules 2, 16, and 17).

This VM can be a standard Windows 10 virtual machine, and needs to be pre-loaded with the following software:

1. Azure Data Studio (for Module 2)
2. Power BI Desktop (for Modules 16 & 17)

Modules that can share the same Synapse workspace

• Synapse workspace named asagaworkspaceSUFFIX (where SUFFIX is a unique id provided during setup):
  • Module 1
  • Module 17
• Synapse workspace named asaworkspaceSUFFIX:
  • Module 4
  • Module 5
  • Module 7
  • Module 8
  • Module 9
  • Module 10
  • Module 11
  • Module 12
  • Module 13
  • Module 16
• Requires own Synapse workspace (not shared):
  • Module 2
  • Module 14

Modules that can share the same Azure Databricks workspace

• Module 1
• Module 6
• Module 15