feat: observabiltiy-as-code

[sudivate]

Type of PR

• Code changes
• Documentation changes

Purpose

• Enhanced Observability by adding below resources for Observability-as-code in Bicep modules
  • Azure Dashboard to monitor key metrics from Azure Data Factory and Azure SQL Server
  • Azure Log Analytics Workspace to collect monitoring Data
  • Enabled diagnostic settings for Azure Data Factory
  • Azure Monitor alert for Azure Data Factory Pipelines
• Demonstrated Data Quality monitoring using OSS Great Expectations in Standardize and Transformation steps of Data pipeline
• Sample Data Quality Azure Monitor workbook
• Updated Docs relevant to all above changes

Does this introduce a breaking change? If yes, details on what can break

Author pre-publish checklist

• [ ] Added test to prove my fix is effective or new feature works
• [x] No PII in logs
• [x] Made corresponding changes to the documentation

[devlace]

Updated the .NET version to 4.7 to resolve build error

[devlace]

@sudivate Running into an error in the Databricks notebook. Looks like its related to a recent breaking change on jinja's side: https://github.com/pallets/jinja/issues/1626

[sudivate]

Thank you @devlace.