Agree on the Lakehouse setup for Parking sensors data

[sreedhar-guda]

Work with the team to agree on the asset organization for Parking sensors lake house.

Env Questions:

• What will be the runtime version?

  [Sreedhar]1.3 preferred

• What will be the git location for common ENV attached scripts?

  [Sreedhar]If we are not able to upload scripts to Fabric environments, I prefer staging these scripts in lake house "Files/scripts" location. In git I used src/scripts to store otel_monitor_invoker.py.

• Are we doing .whl files

  [Sreedhar]I prefer keeping the scripts as*.py files as Fabric allows them and it's easy to work with and one less step of having a need to create the wheel file.

Configurations and data files:

[Sreedhar] What is the git location for config files (storing DDLs, process name etc.,). I provided a template which needs to be substituted with actual values before copying to Fabric. The files and target locations will be:

• reference data (dim_date.csv and dim_time.csv) --> target should be Lakehouse Files/data
• in git under config/parking_sensors; Files parking_sensors_ddls.yaml and parking_sensors.cfg --> target should be Lakehouse Files/config
• parking_sensors.cfg is derived from parking_sensors.cfg.template for each stage/env after substituting with appropriate values - such as workspace id belonging to that particular stage. Need to cross-check the substitution strings and format with Anuj and Naga.

Workspace:

Can we not have special characters in the name?

[Sreedhar]see the comment section for details

Notebook Linting:

[Sreedhar]: nit - noticed that all the imports are being moved the very first cell by the linting process. If users are using %%configure or parameters cell as the first cell and want that one as the very first one (perhaps to do conditional imports), this could have unintentional consequences.

DoD: Agree on:

• Name of the lake house
• Use of schemas (names will be same as existing sample)
• Lake house for common utilities
• All items listed above have answers

Related to #846

[sreedhar-guda]

Suggestion: Have a naming convention in place and mentioned in the README.md

Here are the assets I am looking for from setup perspective:

• [x] Fabric Environment: "env-${local.base_name}"
• [x] Workspace details
  • [x] Variable/Parameter name: workspace_name
  • [x] Variable/Parameter id: workspace_id
• [x] Lakehouse details
  • [x] Variable/Parameter name: lakehouse_name
  • [x] Variable/Parameter id: lakehouse_id
• [x] Warehouse details
  • [x] Variable/Parameter name: warehouse_name
  • [x] Variable/Parameter id: warehouse_id
• [x] Common Utilities Lakehouse: lh_common_utils(?) --> Out of Scope
• [x] Parking Sensor Lakehouse: "lh_${local.base_name_underscored}"
• [x] ADLS Gen2 shortcut name: sc-adls-main
• [x] Reference (seed) data
  • [x] /Files/sc-adls-main/reference
• [x] Config files (uploaded via deployment script)
  • [x] /Files/sc-adls-main/config
• [x] Monitoring sink: "appi-${local.base_name}"
• [x] Secrets repo: Key vault to store AppInsights connection information - "kv-${local.base_name}"
  • [x] Secret key name: app-insights-connection-string

[devlace]

@sreedhar-guda I believe most of these already are set in the current TF deployment scripts: https://github.com/Azure-Samples/modern-data-warehouse-dataops/blob/feat/e2e-fabric-dataops-sample/e2e_samples/fabric_dataops_sample/infra/terraform/locals.tf

Only thing missing is the reference data. We can just call those dim_time and dim_date.

Let's park the utilities lakehouse for now and keep it simple with just the existing lakehouse. (single lakehouse)

[sreedhar-guda]

@promisinganuj @devlace @naga-nandyala

Currently spark queries will fail if the workspace name has a special character in it when using fully qualified names( FQN) for table references. See: https://learn.microsoft.com/en-us/fabric/data-engineering/lakehouse-schemas#public-preview-limitations for more details.

We are developing notebooks with FQN - which included workspace name as well as a best practice. Our Terraform modules use "-" in the name of the workspace. This will result in errors when running queries using FQN.

Options to address this:

1. Remove special characters from workspace name
2. Update the notebooks to remove workspace name from FQN

Let me know if option 1 is something we are open to do. If not, I will update the notebook accordingly.