Fix deployment issues. Replace DW by ded pool

[ydaponte]

Type of PR

• Code changes

Purpose

• fix Notebooks to correct access to the KeyVault Linked Service
• fix waiting cycle for Provisioning State
• changed Pipeline not to need KeyVault name as parameter
• changed bicep deployment to deploy just a dedicated pool, removed DW deployment
• changed Pipeline to use Dedicated pool instead of old DW
• added sql ded pool name dynamically into the json pipeline file to avoid load errors
• added missing REST Linked Service
• remove wrong reference to notebook 4
• remove linked service for DW

Does this introduce a breaking change? If yes, details on what can break

No.

Author pre-publish checklist

• [ ] Added test to prove my fix is effective or new feature works
• [X ] No PII in logs
• [ ] Made corresponding changes to the documentation

Validation steps

To validate, after the deployment you need to navigate to the synapse workspace, validate that the following artifacts are deployed:

• Spark pool
• SQL dedicated pool (no more DW is deployed in the RG)
• Notebooks
• Linked Services
• Packages in the workspace (and that they are attached to the Spark pool)
• Pipeline, including task to load now using the sql dedicated pool (for the pipeline work end-to-end before the setup notebook and the DACPAC deployment on the dedicated pool needs to run in advance)

Issues Closed or Referenced

closes #394 closes #395

[promisinganuj]

May be we can get rid of "synapse_sql_pool.bicep" if it's not required?

[promisinganuj]

Do we need to deploy the Synapse workspace in managed VNet for this sample? synapse.bicep

[ydaponte]

Replying to previous comments here as I can't do it individually - the comments are closed. > May be we can get rid of "synapse_sql_pool.bicep" if it's not required? I left the file because is still needed for a databricks deployment. If we decide to isolate the synapse piece, than this file can be deleted along with the other databricks deployment files > Do we need to deploy the Synapse workspace in managed VNet for this sample? As fas as I understood we agreed to continue with a simplified version of the deployments (with no vNets) within the Data Ops repo as we are trying to demonstrate the end-to-end concepts.

[promisinganuj]

Replying to previous comments here as I can't do it individually - the comments are closed. > May be we can get rid of "synapse_sql_pool.bicep" if it's not required? I left the file because is still needed for a databricks deployment. If we decide to isolate the synapse piece, than this file can be deleted along with the other databricks deployment files > Do we need to deploy the Synapse workspace in managed VNet for this sample? As fas as I understood we agreed to continue with a simplified version of the deployments (with no vNets) within the Data Ops repo as we are trying to demonstrate the end-to-end concepts.

That makes sense. But the latest version of "synapse.bicep" is still deploying the Synapse Workspace in a managed VNet, check here and here. I believe you would need to remove these lines to deploy Synapse without VNet.

[ydaponte]

promisinganuj, the vNet managed piece was removed. Good catch - thanks for that :-)