Synapse Pipelines Web Activity cannot reach sites when using Managed Virtual Network with data exfiltration protection?

[gdubya]

It appears as though the Web activity is unable to contact the management endpoint when the Synapse workspace is created using a Managed Virtual Network and data exfiltration protection. The first Web activity fails with an "unknown error".

This problem does not occur if i create the pipeline for a Synapse workspace that does not use a Managed Virtual Network with data exfiltration protection

Is there a workaround? The documentation should perhaps be updated to mention the restriction if it is not possible using a Managed Virtual Network.

---

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

• ID: 239e657e-15d8-05a3-466d-4159b178307e
• Version Independent ID: 866f6f93-0302-324b-668e-b41f67581d6d
• Content: How to pause and resume dedicated SQL pools with Synapse Pipelines - Azure Synapse Analytics
• Content Source: articles/synapse-analytics/sql/how-to-pause-resume-pipelines.md
• Service: synapse-analytics
• Sub-service: sql
• GitHub Login: @kromerm
• Microsoft Alias: makromer

[gdubya]

Asked on StackOverflow

[gdubya]

There is a section of "known limitations" under "Create a workspace with data exfiltration protection enabled" that is perhaps relevant. This says:

In data exfiltration protected workspaces, connections to outbound repositories are blocked. As a result, Python library installed from public repositories like PyPI are not supported.

I suppose this applies to pipelines too. If this is the case then I suggest adding a "known limitations" section to this "How to" page as well.

[KranthiPakala-MSFT]

Thank you for reaching out. At this time we are reviewing the ask and will provide an update as appropriate

[lucabovo]

I have a similar Github-Issue about the same doc (https://learn.microsoft.com/en-us/azure/synapse-analytics/sql/how-to-pause-resume-pipelines) and the same scenario/environment (pipeline execution in Synapse Workspace with Managed VNet and data exfiltration protection ENABLED) You can find the issue here: https://github.com/MicrosoftDocs/azure-docs/issues/99174

[gdubya]

@lucabovo my personal conclusion was that this was the root cause of the problem for us

[rebremer]

I noticed that it is possible to trigger REST APIs using a web hook in Synapse Workspace with Managed VNet and data exfiltration protection ENABLED. I assume this because in a web hook no IR has to be specified.

Is this expected behavior? I mean, I can now just use a web hook to exfiltrate data...

[Cedz]

When and how can this be addressed? Facing a similar issue . WebHook also does not work .

[gdubya]

@Cedz Use an integration runtime that is not contained within the managed vnet (e.g. a self-hosted IR).

The issue i raised here is regarding the documentation, not the functionality itself.

[IsraelOrosMsft]

@gdubya and @rebremer How can you leverage the self-hosted integration runtime (SHIR) when you are working with webhooks? I could not find a way to specify that. I created a SHIR on a private VNet and I validated I could connect to the webhook from that machine. The Synapse Workspace shows the SHIR status as running and the SHIR can communicate to the Webhook, but the pipeline calling the webhook fails as documented in this conversation.

Am I missing something? Is it really possible to access webhooks from Synapse with exfiltration turned on?

Thanks!

[rebremer]

See my GitHub what I did to solve this: https://github.com/rebremer/securely-connect-synapse-to-azure-functions

[bandersmsft]

Thanks for your dedication to our documentation. Unfortunately, at this time we have been unable to review your issue in a timely manner and we sincerely apologize for the delayed response. We are closing this issue for now, but if you feel that it's still a concern, please respond and let us know. If you determine another possible update to our documentation, please don't hesitate to reach out again. #please-close