search

pulumi / pulumi-azure-native

Azure Native Provider
Apache License 2.0
129 stars 35 forks source link

[Synapse] Missing Data Plane Resources #2795

Open rhuanbarreto opened 1 year ago

rhuanbarreto commented 1 year ago

Hello!

Issue details

All the Data Plane resources from the Azure Synapse REST API (Link of an example) are missing. Can you add to the library?

mikhailshilkov commented 1 year ago

Hi @rhuanbarreto thank you for opening this issue.

Azure Native provider focuses on control plane resources that are available in ARM (Azure Resource Manager). We've added a couple of data plane resources for Storage, but it's always a manual implementation, so we have a fairly high bar for it.

We will use this issue to track potential addition of extra Synapse resources, but I can't promise on any timeline.

rhuanbarreto commented 1 year ago

I get the point @mikhailshilkov. So instead if we maybe add only the Workspace Library Resource that I linked above it would be very beneficial. There's a specific parameter of the BigDataPool resource (Library Info) that is directly related to this data plane resource.

Today after I add a Blob (one of the data plane resources you mentioned) using Pulumi, I need to use azure deployment scripts (in Pulumi) and a series of other pulumi resources, like user-assigned identities and specific permissions to the storage account and synapse workspace in order to run the commands manually:

az storage blob download -f ${fileName} -c ${containerName} -n '${fullName}' --account-name ${accountName} --auth-mode login --no-progress
az synapse workspace-package upload -f ${fileName} --workspace-name ${workspace.name} --no-progress || true
az synapse workspace-package list --workspace-name ${workspace.name} --query "[].{containerName: properties.containerName, path: properties.path, name: properties.name, type: properties.type} | {packages: @}" > $AZ_SCRIPTS_OUTPUT_PATH

The output of this deployment script I send to the BigDataPool resource in Pulumi. It's not optimal but it works. The downside is that if I do a pulumi refresh, then I will lose the reference to the deployment script once it gets deleted after the retention period and the script will be recreated and run again.

So this small Data Plane addition will make a huge difference for me and reduce the amount of workaround in code.