Create a proposal for workspace organization and common utilities

[sreedhar-guda]

What: Create a shared workspace that hosts common utilities, configurations, and data lakehouse, and serves as a source for all domain/product-specific workspaces. The default lakehouse associated with all notebooks will be the lakehouse from this workspace.   Why?

• Currently any code related assets are made part of environment resources and individual workspaces. For any change in common modules - code using these moules need to be re-built and re-deployed.
• Often users are using relative paths in their notebook code. When trying to scale the notebooks (run multiple instances of it in high concurrency mode with different parameters or cross-workspaces references) - this could cause issues or confusion. Also, every time a default Lakehouse is changed the notebook session needs to re-start.

Benefits:

• Centralized place for utilities and observability capabilities.
• Migration of central assets can follow their own cycle. Individual workspaces can choose to get the latest code at their convenience by simply updating their shortcuts. This dramatically cuts the migration times as well as maintenance burdens.
• Sets some groundwork for data mesh pillars (FG, data domain and Self-service).
• Provides a path to merge process (reference or user provided info) and execution data (telemetry data: observability DBs is typically centralized while collection is local).
• Attaching this lakehouse to all notebooks ensure that users also have access to SQL executions. Currently, no SQLs can be run (either using %%sql or spark.sql) in a notebook without having a lakehouse associated. Having this common lake house can ensure - users can run their SQL queries and also ensures they have a common place to look for shared files.
• Encourages uses to use fully qualified paths in all places.

Can we do it in this sprint?

• Yes, and here are the reasons. While the vision is long-term, the implementation:
  • Accommodates incremental development
  • Is a 2-way door decision (very easy to fall back to existing setup)
  • Effort needed now is small (for current project ~ 2 days).

How: Please refer to the proposed generic setup diagram below.

Applied to current Parking sensors sample - we have 3 options:

Is this a theory or has it been tested?

• Yes, the concept has been tested.

DoD:

• Obtain team consensus on the proposal and account for feedback.
• If approved, implement the shared workspace structure and create associated documentation.
• If not approved, document the rationale and address the team's concerns.

[sreedhar-guda]

Feedback from Lace:

My initial view is that it fits better in a Data Mesh sample. Enterprise Workspace organization isn't really in scope of the Fabric E2E sample where the focus is really CICD, testing, etc.. Having multiple, say 3 (two data products, one common), workspaces per Environment (Dev, Test, Prod) would mean the user is deploying 3x3 (9) workspaces in the sample which I think would just overcomplicate things in an already complicated sample... Not to mention the potential ephemeral workspaces we would spin up as part of the CI / PR validation process + branched out workspaces as part of dev. Its just a lot of complexity for the users of the sample....

That said, i think this would fit very well in a Data Mesh reference sample that does not have CICD demonstrated as part of it. Workspace organization also has a ton more considerations as well like best capacity allocation (since workspaces are the smallest unit a capacity can be tied to, etc) -- its a bit of slippery slop if we start adding it in scope to the Fabric E2E sample.

Decision documented in #840 - to use a single Lakehouse (Option 3).